~10min read
Open Science! Chances are, you’ve already stumbled upon this term. But isn’t science always open by design because we publish our results, you might wonder. Unfortunately, it is not as open as it should be to allow for quality control measures. So let’s first have a look at the scientific process itself to see where it is open by default and where not.
Is openness an inherent part of the scientific process?
Scientist or not; everyone uses the scientific method to solve everyday problems. Why does your toast land on the buttered side when accidentally thrown off the table? Which apple do you pick from the pile in the supermarket? In such cases, you’ve made an observation or asked a question and most likely also formulated a hypothesis. For example, you may pick an apple that is firm, red and without any brown spots, because you act on the hypothesis that these characteristics signal good taste.
Formally, the scientific method comprises the formulation of a hypothesis based on an observation or question: Does the absence of brown spots on apples signal good taste? This hypothesis can be tested in an experiment. For example, you could now gather apples with and without brown spots and let people taste them. The gathered data is then analyzed (is the taste rating of your participants higher for apples without brown spots). The results may then modify the hypothesis for subsequent testing and start a new cycle of the iterative scientific process.
Now, to ensure that everyone gets to know the results, the process of the experiment is summarized and published in a journal. Great, so we got our study on how brown spots on apples signal bad taste published and we’d love everyone to read it. Whereas our work is now publicly available, this does not necessarily mean it’s “open”. Openness can be hindered at different stages of the scientific process. For example, the publication does not ensure that you as the reader have access to the scripts the analyses were based on, the underlying data is often not shared or methods are described with insufficient detail. Also, your publication may end up behind a paywall, preventing many people without institutional affiliations from reading it. Therefore, you can envision Open Science as a collection of principles which should restore openness at these different stages. So let’s have a closer look at these stages.
Methods ambiguity
When we translate our research question into an experiment, there can be ambiguity in the way the study materialized. The methods section of a scientific publication describes how the experiment was set up and which analyses were conducted. This should indicate limitations regarding the interpretation of the results. For example, it may make a big difference whether your subjects tasted the apples blindly or not. Moreover, the word limit imposed by many journals encourages short descriptions lacking detail. Now imagine you would like to implement the same experiment to test if you get the same results. The only instruction you have for your implementation is this short methods section which doesn’t even tell you if the apples were tasted blindly. If you now assume they were, you interpreted the ambiguous information according to your own knowledge resulting in a different experiment. A similar problem can arise if your method already has a name (brown spots taste test) and you only refer to that name in your methods section. Chance are, someone else has a slightly different understanding of the brown spots taste test and carries it out differently (jingle fallacy).
Solutions tackling these problems can be registered descriptions of a method that can be referenced in a publication or the publication of supplemental material (for example scripts) that detail each step of your experiment.
Code undercover
Let’s consider our main method to study the brain: fMRI. Here, acquired data undergoes many preprocessing steps before the main analyses are carried out. These steps are handled using scripts, but they are not routinely published along with the paper. Not only may these scripts contain honest mistakes; the description of the pipeline might be wrong. Neuroscience offers a fair share of processing options. Researchers can choose a package that best serves their needs. However, this may lead to false assumptions of specific processing steps, especially if the processing pipeline is merged with custom code. Let’s say you use a well-known processing package, but write your own code in between to facilitate some of the steps. It may be that your custom code contains errors or influences the functions from the processing package. These are changes a reader of your subsequent publication is not aware of, although they might have dramatically altered the results. Consequently, especially in cases where an analysis relies on custom code, the disclosure of this code is tremendously important for evaluation of the results.
Data – the crown jewels of empirical science
When experimental data is not made available, it becomes increasingly difficult to verify in how far claims made in a publication hold true for the bigger picture. Coming back to the brown spots taste test, it is hard to say if your sample of people rating the apples’ taste is really representative. So did you find a general rule of how the taste of apples is perceived given their outward appearance or does the perceived taste differ between individuals. Maybe you’ve just found an overly critical group of people worshipping the perfectly looking apple. If researchers share their data, groups interested in the same phenomenon can team up to combine their datasets. Apple enthusiasts with an interest in the effect of brown spots on taste could now work with dramatically increased sample sizes which elevates the chance to find meaningful and reliable effects.
Especially in cases where the acquisition of data is expensive, as in our favorite neuroimaging method fMRI, reported results often rely on small samples. This may lead to spurious results as indicated by meta-analyses failing to converge across many small studies. Sharing of data can help to construct reasonably sized datasets cost-efficiently.
Admittedly, not every dataset can be made directly available, e.g. particularly data from medical studies where interests of the patient have to be protected. But even in such cases researchers have options such as de-identifying imaging data by removing e.g. facial information from the scans or sharing data within a clinical consortium in which every member is held accountable.
Hence, as long as the written approval of the subjects is obtained and de-anonymizing features of the data are removed, open data can greatly improve the replicability of analyses.
Publish ≠ public
The last three principles concern the publication of results directly. Usually, publications are reviewed by anonymous reviewers who evaluate the contribution of a paper. This process could be made more open. For example, reviewers may sign reviews or actively seek to perform a review for a submission. Preprint servers (like bioRxiv) openly hosting papers before formal publication in a journal provide such venues now.
After successful publication in a “classic” journal, most papers are only accessible to subscribers. Most troubling, this affects many papers published in prestigious journals such as Science or Nature. Since every researcher expects to have access to those top journals, university libraries pay enormous subscription fees. This not only constitutes a financial burden for the institutions and ultimately the tax payers, the (in most cases) once publicly funded research results are also still hidden behind a paywall. Likewise, use of novel results for teaching incurs high licensing fees impeding dissemination to everyone.
Open access counteracts this restriction by encouraging green (open availability of a paper after an embargo) or gold (direct open publication) options.
Where do we go from here
Of course, talk is cheap. In our research, we are actively engaging in Open Science in many of the above-mentioned stages. Furthermore, we are openly developing an app (download it from our GitHub repository) within the Open Science Fellows Program. But how could an app help us overcome the problems we’re facing in science?
Mainly it targets the problems surrounding the implementation of an experimental task solely from the information that is provided in the methods of a publication, as described above. Remember, if you’d try to implement the brown spots taste test and could not find any statement on if the apples were tested blindly, you would have to guess and therefore very likely create a slightly different adaptation of the test.
Now, imagine an open-source app running on any platform, be it desktop or mobile. This free app includes well-tested and fully described tasks (for example the brown spots taste test). In publications, you can refer to this description or even the source code. Replications become much more straightforward with this setup now. Everybody can download the set of tasks and run the experiment in the exact same way as before. This reduces undesired sources of noise in the experimental data collection. On top of that, differences between the description and the implementation of a task are easier to verify. Checking one well-maintained battery of tasks is arguably more effective than proofreading the hundredth reinvention of the same old wheel.
Another big concern are small and heavily pre-selected samples in biomedical research. For example, in many studies on depression, patients suffering from other mental disorders are excluded. However, this is common rather than an exception. A crowdsourcing approach inherent to our app can shine in this situation. Crowdsourcing means to gather data from a large, open and not pre-specified group of people. Instead of inviting carefully selected participants to the lab which is often necessary due to monetary restrictions, you can roll out your tasks to the public in general and assess demographics and characteristics of the people taking part. This would be very useful for example for clinical studies. As they often compare healthy versus ill, but functional individuals, we know little about extremes of a trait or the dimensionality of symptoms in people. Therefore, the transition from “healthy” to “pathological” behavior is often unclear. Distributing an app among the general population facilitates a more diverse sampling and consequently leads to higher chances of also including subjects that are underrepresented in current research samples, such as e.g. people showing subclinical behavior.
In summary, separating the good from the bad apples becomes exceedingly difficult with closed research practices. Evaluating scientific results on different stages of the scientific process rests on the availability of information on how an experiment was carried out, analyzed and interpreted. While publications should summarize and deliver all these information, in reality the informative value of studies is often impaired by a lack of routine, publication of validated paradigms, analysis scripts, and verified datasets. Open methodology, open source, open data and open access allow to tackle these problems and as demonstrated with our app help research scientists in building more appropriate models of individual behavior.
App repository with download options: VTeckentrup – Mind Mosaic
Contact: Vanessa Teckentrup (vanessa.teckentrup@uni-tuebingen.de)
