In psychology, biomedical sciences, and neurosciences, given the variability that exists in biological systems, including humans (e.g., differences between individuals in brain structure and function, in response to stress, etc.), expecting to obtain exactly the same results from one study to another is unrealistic. Nevertheless, the most salient results and the outline of the conclusion should be replicable (Begley & Ioannidis, 2015).

Several factors have been identified as potential causes of difficulties in reproducing or replicating study results (Baker, 2016; Begley & Ioannidis , 2015; Button et al., 2013). On the methodological level, these include a lack of result verification and quality control within the laboratory of origin (which notably reduce the risk of undetected errors affecting the results), a lack of transparency regarding the procedures used, shortcomings in terms of statistical analyses, as well as low statistical power, often linked to small sample sizes. This last point is an issue in neuroscience and neuroimaging studies, in particular because of the significant costs associated with data collection (e.g., for carrying out MRI examinations; Button et al., 2013) and in studies carried out with vulnerable populations, which are often more difficult to recruit (e.g., people suffering from neurodegenerative diseases). Even when sample sizes are large enough, an incomplete description of participant characteristics can hinder replicability. Indeed, if two studies have a supposedly similar population when, in reality, this is not the case (for example, one group is more educated or more active than the other), their results may differ. Finally, the fact that some specialized methodologies may lead to technical difficulties may also decrease reproducibility and replicability.

Organizational factors are also involved, such as the pressure to publish (the famous “publish or perish”), which is exacerbated by the strong competition to obtain research grants and positions in academia (Baker, 2016). In fact, valuing productivity (e.g., in terms of the number of scientific articles published) rather than quality and methodological rigor is detrimental to the reproducibility of published studies since it does not encourage rigor. Valuing innovative findings that generate excitement at the expense of establishing solid scientific fact through rigorous and incremental studies can eventually undermine scientific rigor (Begley & Ioannidis , 2015).

On a psychological level, confirmation bias can impair replicability (Begley & Ioannidis, 2015), especially inferential replicability. Confirmation bias consists in being too favorable to observations that support our hypotheses, to the detriment of those that contradict them. This bias can be at the origin of different conclusions made by two research teams based on similar results (but this can sometimes also be explained by other factors). Selective reporting, which is associated with confirmation bias and consists in presenting only a part of the results obtained (e.g., those which support our hypotheses), is a bad scientific practice which also harms the replicability of the results (Baker, 2016).

As the factors at the origin of issues with reproducibility and replicability are multiple, there is no single solution to remedy them. Fortunately, several possible solutions have been proposed and have been increasingly employed. These solutions involve several agents, including the various members of research teams, funding agencies, academic institutions, and scientific journals.

Solutions related to the training and supervision of research personnel

A first solution proposed by researchers (Begley & Ioannidis, 2015) is to adequately train students and research staff in methodology (e.g., experimental design, data collection and analysis, interpretation of results, etc.), metrology (i.e., all the methods and techniques used to obtain the greatest precision in measurements), and responsible conduct of research (on this subject, an online sensibilisation tool on the best practices in terms of responsible conduct of research is available on the website of the Fonds de recherche du Québec). Adequate methodological training may sometimes require collaborating with colleagues from other departments or universities with complementary expertise, who can provide mentorship to the student in training. Continuous training for researchers is also necessary given the constant methodological developments (Begley & Ioannidis, 2015). During the development of a project, quality control measurements must be put in place, including the verification of data and analyses, which constitute a safety net to detect potential errors and improve training as needed. The use of standardized protocols by team members also minimizes variability in the administration of procedures, thereby promoting replicability of the method and results within the team and beyond (Baker, 2016).

Solutions related to transparency and data sharing 

For a study’s methodology to be replicable, enough detail about the procedures must be provided, which requires rigorous reporting of information (Goodman et al., 2016). It has been suggested that sharing videos to visualize the procedures could be a way to improve the replicability of the method (Pulverer, 2015). Additionally, scientific journals and institutions can encourage researchers to share raw data, analysis scripts, and materials used (such as visual or audio stimuli presented to participants in an experimental test; Gilmore et al., 2017). This information may be shared publicly on data repositories (e.g., Boréalis, OpenfMRI, NeuroVault). In the field of neuroimaging, the Organization for Human Brain Mapping (OHBM) published in 2016 a report on the best practices in analyzing and sharing data when using magnetic resonance imaging (Nichols et al., 2016).

When the data is shared publicly, it can be used in meta-analyses with a large amount of data (platforms such as BrainMap and NeuroSynth  allow the generation of meta-analyses from MRI data). Meta-analyses measure the variability of results between studies, as well as the quality of the studies. Thus, the conclusions of meta-analyses are generally more robust than those of individual experimental studies.

Finally, a way to prevent the selective reporting of results is the pre-registration of scientific studies (Botvinik-Nezer & Wagner, 2022; Baker, 2016; Poldrack, 2018). Pre-registration consists in publishing your research protocol, including the hypotheses and analysis plans, even before the study is carried out. As the main and secondary measures to be analyzed are already specified by the team, it is expected that all the results concerning these measures will be published once the study has been carried out.

Solutions related to a paradigm shift valuing quality over quantity 

As part of the process of awarding research grants and scholarships, not only the quality of the studies but also the efforts of the researchers in terms of transparency and reproducibility should be taken into account (Begley & Ioannidis, 2015). For example, the adoption of randomization measures, blinding procedures (when possible and appropriate), and good statistical practices should be valued. Greater weight should also be given to research questions and methods, rather than to the results obtained (Poldrack et al., 2018).

Scientific journals could also issue guidelines for good scientific practices and checklists for authors when submitting articles (Pulverer, 2015; Baker, 2016). Moreover, academic institutions can reward the completion of replicable and robust studies and provide continuing education opportunities for researchers and students. Such actions are likely to improve the quality of scientific publications and their replicability (Begley & Ioannidis, 2015; Baker, 2016).

Researchers are of the opinion that publishers of scientific journals should also encourage the implementation of confirmatory studies and the publication of negative results, i.e., results that do not support the initial hypotheses and do not replicate the expected results (Wagenmakers & Forstmann, 2014; Ioannidis, 2006). If multiple research teams unsuccessfully attempt to replicate one research team’s results, but their negative results are not published, this fuels the reproducibility crisis. Indeed, if negative results were published more often, teams could decide to look into other research questions rather than persisting in reproducing or replicating the same studies without success. Moreover, a certain skepticism about new results, which are often published in the most reputable journals, is desirable (Begley & Ioannidis, 2015). Indeed, any new result must be reproduced several times before it can be considered “true”.

It is therefore joint actions, at several levels and involving different agents of research, that are necessary to promote the reproducibility and replicability of scientific studies. As several of the actions that can be carried out at the level of the research teams (e.g., better training and supervision of personnel, establishment of systematic verification procedures within the team, etc.) are costly in terms of human and financial resources, the support from granting agencies, academic institutions and publishers of scientific journals is essential!