13 Aug 2023 - Reader
“The imposing edifice of science provides a challenging view of what can be achieved by the accumulation of many small efforts in a steady objective and dedicated search for truth” - Charles H. Townes
The primary job of a researcher is to expand the frontiers of knowledge for themselves and the wider society, using the scientific method to discover new fundamental truths about the universe that were previously unknown. In this process of continuous expansion of our common understanding of the world, researchers unlock new and more efficient ways to fight diseases, communicate or travel over long distances or to just better understand the causality of events around us.
As suggested by the American physicist Charles H. Townes, our common scientific knowledge is an imposing edifice incrementally built over time in search for truth. It is a thousand years old edifice that we are still building to this day and to continue its progression, researchers must understand the existing infrastructure and add on top. Understanding the state of the art is a prerequisite to any attempt at making a meaningful contribution to any field of research but this is an increasingly difficult task to achieve.
The reasons are multiple, the most obvious being the sheer amount of scientific knowledge accumulated by our predecessors, the less obvious reasons are due to the terrible quality of research currently dumped into the scientific literature and incentivised by a broken model for research funding. Imagine completing a building with no plans and with load bearing walls obfuscated by hundreds of faux-pillars made with cheap plastic. That is how it feels being a researcher today.
In this article we will expose the dynamics that many researchers will recognise after spending some time working in academia and attempt to provide clues as to why these dynamics appear and how to navigate the keeping-up-with-literature side of being a researcher. We will also attempt to imagine a different structure for research in a way that although one still needs to put in the effort to understand the current state of the art, the faux-pillars are naturally expelled from the infrastructure in the way the scientific method was supposed to allow.
Publish or perish, this is the mantra of the early career researcher. If living beings are good at one thing, it is optimising to survive for as long as possible. Researchers are living beings. A simple directive given to millions of researchers all over the world has made the scientific literature for the most part rubbish. Some scientists talk about a reproducibility crisis in experiment-rich fields such as biology or psychology, where 70% of scientists (from a pool of 1500) admit failing to reproduce results from the literature. This survey reveals an alarming problem with the current state of the scientific literature. Let’s all remember that the scientific literature should be written to improve the pool of knowledge of society as a whole. Instead, a lay person must now be trained in every possible field of science to discern for themselves what is correct from what is pure nonsense. Of course, faced with this challenge, the lay person adopts the most rational stance, complete skepticisms accompanied with an unhealthy dose of conspiracy theories. Is global warming even real? What about 5G and Covid? Are they tracking us with the vaccines? Before becoming an expert in any given field, one starts as a lay person. Most scientists start their career in research with a limited knowledge of their field but also with the ambition to make a lasting and meaningful contribution to science. But where do you start when the literature is so vast one cannot realistically ingest it all let alone keep track of the latest developments? In an ideal world, a PhD student would receive guidance from their supervisors on the state of the art so they can at least know where the challenges are and where to begin. In reality, their supervisors have long stopped doing any meaningful research due to overwhelming teaching and managerial responsibilities which mostly consist in finding additional funding to keep the team alive. The funding comes by showing a long track record of publications and writing research proposals containing a mix match of the current trendy key-words “AI powered blockchain for quantum based COVID drug discovery”. This proposal is what the young researcher will have as a starting point accompanied with the directive to publish. Add to this the scarcity of academic positions for young researchers and what you get is a machine to produce questionable “scientific” articles by design. But of course there is the peer review mechanism that is supposed to take care of this ! I refer the reader to our article on the peer review system.
It is difficult to classify any piece of work as useless unless said work has already been done or is clearly scientifically incorrect. Even tweaking the parameters of a machine learning model to produce different results could be deemed publishable if unexpected results are produced. And sharing of ideas is the way we collectively discover new concepts and incrementally build our understanding of new science. The problem might then originate from the subjectivity of what is deemed useful, interesting, incremental or useless. This is obvious to any researcher that has ever submitted an article for publication in a journal. Often reviewers will come to completely opposite conclusions as to the significance or validity of the work proposed. Is it a problem if a scientific article is published with errors? Or if the content of the article is not interesting to every reader? We would argue that the short answer to both questions is no, but of course it is not as simple. The full answer depends on the intentions of the authors when they wrote the article and those of the reviewers that let it get published. Did they genuinely think that the article had some value besides increasing their publication count and notoriety? Were they genuinely trying to solve a real problem besides justifying their funding? Were they helping a friend as they accepted the publication? Publication count, notoriety and funding are the reality that we have to deal with but does it have to take precedence over the quality of the science we produce? Of course it shouldn’t but in reality this is what you do to survive, and it is easy to justify that it is just a means to an end. If you do not publish then you won’t secure enough funding to last you until your breakthrough, but a breakthrough often takes a long and hard focus that you cannot afford as you are chasing the next funding. And imagine the danger of focusing on a hard problem for years and not reaching an answer, that would definitely kill your career. A logical conclusion to this is that researchers are paid (incentivised) to publish articles and not necessarily to advance our understanding of the world. This conclusion is arguably supported by the state of scientific literature today.
It is interesting to compare what is happening in academia to the equivalent context in industry. Unlike academic research, industrial research is focused on solving specific problems which will bring value to the company in a relatively shorter time frame compared to academic research. This shift in incentives means that researchers in industry are focused on a specific goal that will ultimately be checked against the rigid constraints of the real world. In this environment, assessing the validity of the work is less of a subjective exercise, one only has to ask whether the original problem was solved. In this context also, writing an article without clear results does not make sense as the time spent writing the article would be better valued working on a solution to the problem. Of course the drawbacks here are that (1) there is an incentive to hide key elements of the work due to market competition (2) the research topics must often have a clear commercialisation path. In a sense, academic research has become encumbered with short term incentives - by imposing a publish or perish mentality - whilst asking academic researchers to maintain a longer term vision. Those are two incompatible incentives that hinder scientific progress in the ways we have described.
Most researchers start their career in academia knowing that their pay will not be as high as in industry but the drive to push the boundaries of science and have a shot at tackling the most challenging problems to mankind is too strong for them to resist and weighs higher in the balance. The reality they face is that quickly a tension begins to settle between normal life aspirations such as building a community, starting a family or providing stability to your family versus the precarity of academic employment. Most early career researchers are on temporary contracts and can remain in this situation for over a decade, it is expected that we change institutions, cities, even countries if we want to establish a career. At the same time faculties are reluctant to offer a permanent position to a young researcher as it is an expensive long term commitment for which they would have to take the risk of hiring an underperforming academic not able to help them achieve their target research output (mostly publications) and funding. Again, this is a logical behaviour from these faculties given their incentives to publish more and attract more funding. Nothing is said of the quality of life of their employees. This leads to researchers building strategies to improve their careers based on the number of published articles and funding secured with little emphasis on the quality of the work. The only way to increase those numbers is through short-term thinking incompatible with the scientific process.
Why is this a deeper societal issue ? Look around you and pay attention to the evolution of tastes over the course of your lifetime. Everything trends towards shorter timeframes and faster production and consumption with little consideration to our needs as humans. Often what gives is the quality of what is produced and consumed (fast food being the perfect example of this phenomenon). The irony is that researchers and engineers dedicate their lives to understanding the world and using that knowledge to build tools that will better our standard of living and free our time which in theory allows us in turn to worry less about survival and spend more time doing what we each deem more valuable to us. Instead the focus of our society is largely on economic growth at an exponential rate at the expense of quality of life, pushing prices up and leading everybody to optimise artificially their quantity of output at the expense of their quality. This is especially true for creative tasks that cannot yet be automated. Interestingly the problem we have described in the scientific literature can be seen from the angle of a deeper societal problem: The increasing cost of living.
Given it will take an unknowable amount of time to change the fabric of our society to really address the problem at the core, one must find ways to cope in the meantime.
Choose an extreme niche If you are in a position where you can choose your research project, make the best of this opportunity and pick something so niche that only a handful of people are working on the same problem, better yet create your own niche. This will make it much easier to track the progress of your area of expertise and searching through the literature will mostly mean looking for tools to help you solve a well defined problem. This will also give you focus as you develop your field and your expertise, slowly branching into new fields with your unique angle. The difficulty of this approach is that initially you might avoid the trendy topics that attract funding easily but if your chosen subject is of any importance your results will have value and you will find ways to branch into other more trendy fields - because everything in science is connected to something else if you look deeply enough. Of course you still need to identify a niche, which involves some understanding of the literature for things that people are not doing. The trap here is taking the shortcut of putting together a combination of hot topics in order to create a new discipline. Although this can lead to valuable results sometimes, one must really ask themselves what the end goal is. Is it curiosity driven or problem driven? Hopefully the goal is not just an additional paper. Of course choosing a niche subject is a greater risk to take but this should arguably be the purpose of academic research; taking risks that are not commercially viable.
Ground yourself
You found yourself working on the latest trend that promises to turn any metal into gold, what do you do? Do you admit defeat faced with the tsunami of papers claiming to achieve just this?
This is often the case for young researchers that usually start their careers working on someone else’s project. Often, the principal investigator that proposed the project and obtained funding for it, has chosen a topic aligned with the thematic of big funding bodies. These thematics are where all researchers will flock to. The pro of this situation is that it creates an incentive for people to work on specific scientific issues but unfortunately also incentivises publishing of non-results from researchers with no real interest in the problem to solve.
If you find yourself in this case and happen to care about the topic of research, one thing you can do is build a prototype or experiment to represent the problem you are trying to solve. This is because in order to identify the work in the literature that will help you move forward you will need an anchor to reality. It is easy to get distracted by articles that pretend to solve your issue but starting from very wild assumptions. An experiment grounded in reality will serve as a filter to identify ideas that help you build forward.
You found yourself working on the latest trend that promises to turn any metal into gold, but can’t build a prototype or experiment because it is too costly. If you are not in a position to build a prototype or experiment then you are probably in the worst of situations. This is usually the case where articles are written and published everyday with questionable and implicit assumptions, where many articles are written to solve trivial problems, results are applied without considering the conditions of validity of those results and there is no trivial way to know whether an article is worth reading without having read most of it. In this case a working strategy is to restrict the scope of your literature search to specialised and reputable journals or to research groups known for their good work. Such strategy reduces the variety of ideas you are exposed to and increases the likelihood of groupthink but it is arguably better than being exposed to an infinite pool of false or inconsequential results and having to sift through, especially as you are still building your expertise in the field.
Of course these three techniques are not mutually exclusive.
Researchers are increasingly building tools and curation systems to help them deal with the tsunami of articles coming their way, some of which can be found in these articles Nature and Science Magazine. Ironically, the curation systems are also getting flooded, leading to more curation systems which in turn makes it difficult to navigate them… Back to square 1.
Hopefully this article has helped you understand the dynamics responsible for the degrading state of the scientific literature and has given you some tools to navigate it and grow as a researcher. If you have enjoyed this article or not, please leave us some feedback.