To map the different Covid-19 misinformation interventions, we used, to the extent possible, pre-existing definitions of interventions found in the misinformation literature which best fit the interventions discussed in the 50 studies. In some cases, we had to accommodate interventions that weren’t explicitly detailed in the extant literature. “Monetary incentives” and “priming” are particular examples of interventions we identified in this review that had previously received little attention in the misinformation literature. As such, the process was both deductive — we borrowed terms and definitions from existing intervention taxonomies — and inductive — we identified and classified interventions that had previously received little to no attention. Current taxonomies of misinformation interventions are not perfect; definitions could be tightened and refined. In some instances, interventions may not  fall perfectly into one intervention type or another. Such definitional ambiguities make strict classifications difficult and we acknowledge the limitations of adopting a rigid, bucketed approach. However, we hope that bringing these definitional imperfections to light may inspire more conversation about and action around concretizing terminology about misinformation interventions. 

In order to evaluate the extent to which different Covid-19 misinformation interventions were effective, we first had to map all of the outcome variables identified in the 50 studies. Outcome measures were included if they were clearly defined in the Methods sections of each respective study and if they were established as direct outcomes of the intervention. Examples of outcome measures include: belief in misinformation, intent to share misinformation, and intent to practice Covid-19 preventative behaviors among others. Some studies included mediation models to better understand the relationships of different variables. Where a study showed an outcome measure to be an indirect effect of the intervention through mediation models, the details of these measures were detailed under the column relevant_intervention_details; these measures were not coded independently. We initially identified 65 different outcome measures across the 50 articles; these included long-term measures, which we coded separately. For example, belief in misinformation and long-term belief in misinformation were coded as two separate outcome measures, despite them sharing the same scales. Through a process of iterative grouping, where we closely inspected the scales used to assess each outcome measure (e.g. perceived credibility was measured using a Likert scale from 1 (not at all credible) to 5 (very credible)), and inductive organizing (e.g. outcome measures closely resembling each other such as perceived accuracy, perceived credibility and perceived veridicality were merged into the same outcome measure) we ended up with 47 separate outcome measures. 

Borrowing from Guay et al. (2022), we coded the direct outcomes of the interventions as either helpful, no_effect or harmful. Whether or not an outcome was coded as helpful or harmful depended on how the intervention impacted that particular outcome across the entire sample population and only if the effect was statistically significant at p <0.05. We did not segment the outcome measures by different demographic slices or confounding variables.. For example, if a debunk had a positive impact on reducing belief in Covid-19 misinformation across the entire population and its effect was significant we coded that as helpful. If subsequent analyses revealed that among conservatives with low levels of conscientiousness the debunk had no effect, the outcome variable belief_in_misinformation would still be coded as helpful. However, within the column relevant_effectiveness_details those mediating factors (conservatives with low levels of conscientiousness) would be detailed. Interventions that had non-significant  outcomes  were coded as no_effect. A meta-analysis of all the interventions was deemed infeasible due to the wide variety of interventions studied, the different misinformation stimuli tested in studies, and the extensive use of different outcome measures. 

“Covid-19 misinformation” can mean many different things. It may refer to vaccine misinformation, false treatments, misleading statistics, or 5g conspiracy theories, among many other examples. To  assess the extent to which the interventions were tested against different topics of misinformation, we first extracted all the Covid-19 misinformation claims (referred to as misinformation stimuli) shown to participants in the studies. Rarely did articles present the misinformation stimuli in the body of the article. Most often the misinformation examples were tucked away in supplementary material sections that were attached to the article, or hosted on OSF. Thus the identification of misinformation stimuli presented to participants in these studies required the research team to sift through droves of supplementary material and appendices to identify stimuli. Given this process, it’s possible that we missed certain examples of misinformation in the data collection process. Nonetheless, to the best of our knowledge, this is the first study to systematically extract and explore the misinformation stimuli from Covid-19 misinformation intervention studies. 

After extracting the Covid-19 misinformation examples, we then had to decide how to categorize the stimuli. Categorizing the misinformation stimuli under specific topics would more clearly show the extent to which the interventions had been tested on different examples of Covid-19 misinformation and where possible gaps exist. To identify relevant topics we revisited 12 studies conducted at different points of the pandemic that aimed to classify different topics of misinformation. These studies were segmented into two groups: 1) studies that classified general topics of Covid-19 misinformation and 2) studies that classified Covid-19 vaccine misinformation.

Topics overlapped across the studies. Through an iterative process, different topics from these studies were selected based on their relevance to the Covid-19 misinformation found within the 50 papers. The process of deductively categorizing content under specific topics comes with limitations. Even the best-trained human coders and machine learning classifiers get it wrong. There are always certain pieces of content that don’t fit squarely into one category or another; they may contain elements of several categories. Recognizing these limitations and in an attempt to minimize human error, topics were chosen not only for their relevance to the data we extracted from the studies but also for their mutual exclusivity. In order to ensure the accuracy and reliability of the results of the two coders, we used the Cohen’s Kappa inter-rater reliability test, which ranges from 0-1 with higher numbers equating to higher levels of reliability. Both coders independently coded 90 instances of Covid-19 misinformation. The Kappa test resulted in a score of 0.9, suggesting a high level of reliability. Both researchers then coded the remaining pieces of Covid-19 misinformation from the studies. Results were compared between the two coders and where discrepancies existed, coders discussed the piece of content until a final topic was decided on by both researchers. In instances where there was still disagreement, a third researcher was consulted.