Meta-analysis
The results of a scientific study indicate this, those of another study rather suggest that, a third study supports the opposite… scientific literature can sometimes seem contradictory. What’s more, not all studies are of the same quality! How do you navigate all of this?!
As a general rule, to get a clear picture of a topic, one should read and analyze the quality of all the articles that have been published about this topic. This isn’t always possible. One solution is to check if a systematic review or meta-analysis on the topic is available. A meta-analysis is a type of study that allows one to have a big-picture view of the scientific literature related to a specific research question.
As illustrated in figure 1, systematic reviews and meta-analyses are at the top of the “pyramid of scientific evidences”, which means that their reliability is higher than that of other types of scientific studies. Scientific evidences’ strength is the confidence we can place in the results of a study. Moreover, the risk of bias is lower for systematic reviews and meta-analyses, in part because their conclusions are based on the scientific analysis of all the available information.
A meta-analysis has a high confidence level due to the scientific protocol it uses to find and evaluate the available evidence. When it is done well, all the steps have been performed by more than one person so as to increase the confidence level one can place in the results. Here is an overview of the steps that go into carrying out a meta-analysis:
1) Formulating a precise research question. In this step, the research team must define a precise topic which will serve to guide the search for evidence.
This step is often carried out with the help of the PICO method (Caldwell et al., 2012), which consists in identifying the Population or problem studied, the targeted Intervention or exposition, the Comparison element, and the Outcomes of interest.
One example of a research question based on the PICO method could be: Do elderly folks (P), when they are in a noisy environment (I), have a harder time perceiving speech (O) than young adults (C)?
2) Planning the methodology
During this step, the research team plans the methodology that will be used to perform the meta-analysis. For example, the team specifies the specific objectives of the study with respect to the question, the way the literature search will be carried out, the data that will be extracted from each article for statistical analyses, the analyses that will be carried out, etc. All of these elements are written into a protocol, which is sometimes published before the start of the meta-analysis to increase the transparency of the process (“pre-publication” process).
3) Determining eligibility criteria
This step is essential as it will guide the selection of articles. The criteria for inclusion of a study in the meta-analysis are determined at this step. The language of the articles to be chosen must be determined, as well as the types of studies, where the studies have been published, the characteristics sought in the participants, etc. Usually, the meta-analysis focuses on peer-reviewed articles published in recognized scientific journals. A fully comprehensive search should include articles in all languages, but in practice, this is often not possible as it depends on the languages spoken by the research team.
In the example cited in point 1, the study eligibility criteria would include, among other things, the need to have included a group of elderly participants as well as a group of young adults, which would be essential to answer the research question.
4) Elaboration of a research strategy
The research team chooses the scientific databases (ex. Pubmed, PsycInfo, etc.) in which the scientific articles will be searched. Team members also carefully select the keywords and keyword combinations that will be used for the search, so as to identify all existing studies relevant to their research question.
5) Search for studies in the scientific literature
The search for studies is carried out in the different databases chosen, and duplicates are eliminated. The eligibility of each scientific article obtained is then independently verified by at least two members of the team, in order to keep only the studies that meet the pre-established criteria. All the articles are then carefully read by at least two team members.
6) Extraction and compilation of the data
This step requires a lot of patience and meticulousness. A large amount of data is extracted from each selected article, such as participant characteristics (e.g., age, sex), tests used, and procedures and results (ex. scores and response times to tests). This data is usually extracted independently by at least two members of the team, to minimize the risk of errors. Data from all studies are then compiled.
7) Evaluation of the sources of bias for the included studies
Two team members (or more) evaluate the risk of bias (e.g., participant selection bias, selective reporting of results) and the quality of the studies (e.g., quality of the sample sizes, quality of the statistical analyses). This assessment allows the team to determine the quality of each study – their strengths and weaknesses – as well as the specifics of the field more generally.
Figure 2. Example of a figure (taken from Zumbansen & Tremblay, 2019) illustrating the percentage of studies with a low, uncertain, or high risk of exhibiting different types of bias. Note that in this example, the studies are generally of high quality in terms of the way they report the data, but few studies provide enough information about the randomization process and few studies correctly treat or report “the confidentiality of the randomization method“.
8) Data analysis
It is at this step that the results of different studies included in the meta-analysis are combined into a single statistical analysis. This analysis makes it possible to investigate the presence or absence of an effect (e.g., the effect of age on the ability to perceive speech in a noisy environment, the effectiveness of a vaccine, side effects of a treatment) and the strength of this effect (ex. weak, moderate, or strong).The results of a meta-analysis are generally illustrated in a graph called a forest plot.
Figure 3. Example of a forest plot in a meta-analysis (adapted form an article in preparation by Maillard et al.)
In this type of graph, each study is identified on the left and its result illustrated by a square. The size of the square reflects the weight assigned to each study, which depends on the sample size and the quality of the measurement reported. The bars on both sides of the square represent the confidence interval (i.e., the range in which the true value of the effect is likely to lie). The larger the confidence interval, the greater the margin of error, and the lower the level of confidence in this result (which may mean that there are too few participants or studies to draw a conclusion). If the effect size is very close to 0 (shown by the dotted line), this suggests the absence of an effect. The farther the square is from the dotted line, the greater the effect. The effect can be positive (e.g., reduction in deaths following intervention) or negative (increase in deaths following intervention).
At the bottom of the graph, we find the estimated average effect (circled in blue on the graph), which represents all the results. In the example above, the mean effect size is 0.56, which corresponds to a significant positive effect of moderate size. The actual effect is between 0.28 and 0.84, as indicated by the confidence interval. Since this interval does not contain the value 0, the effect is considered significant.
9) Interpretation of the results
It is at this step that the team members draw conclusions from the meta-analysis (e.g., When we consider all of the literature on the topic, is there an effect of age on the ability to perceive speech in a noisy environment or not? What is the level of confidence that can be placed in these results?) In addition to providing a systematic portrait of the literature, meta-analyses generally identify avenues to be explored in future studies to increase knowledge surrounding the research question. They can also help identify factors explaining the presence of diverging results from one study to another.
In short, meta-analyses are an excellent tool to gain a general idea of the state of current knowledge on a specific subject. Quick little trick to recognize one: the word meta-analysis usually appears in the title or abstract of the scientific article. This is the case, for example, for a meta-analysis recently published by members of our laboratory, the results of which are summarized here. Furthermore, when science journalists present results that come from a meta-analysis in their articles, they usually specify so. Finally, meta-analyses can be found in a variety of databases, including the Cochrane Library, Corpus UL and the HAL Open Archive.
As with all studies, the methodological quality of the meta-analysis may vary. It is important to ensure that that the meta-analysis has been published in a serious, peer-reviewed scientific journal by a team of people that have scientific knowledge and experience. To learn more about the different types of scientific sources or to discover other tips to better distinguish what is true from what is false among all the information you receive, see our blog article on scientific sources or the video-series On veut des faits!, available on Télé-Québec’s website (in French only).
Happy reading!
References:
Caldwell, P. H., Bennett, T., & Mellis, C. (2012). Easy guide to searching for evidence for the busy clinician. Journal of Paediatrics and Child Health, 48(12), 1095–1100.
Maillard, E., Joyal, M., Murray, M., Tremblay, P. (In preparation) Do musical activities enhance the perception of speech in noise? A systematic review and meta-analysis.
Zumbansen, A., & Tremblay, P. (2019). Music-based interventions for aphasia could act through a motor-speech mechanism: A systematic review and case–control analysis of published individual participant data. Aphasiology, 33(4), 466–497.
