Content Analysis of 400 Fact-Check Articles Published by 6 Major UK and US Fact-Checking Organisations, 2024

We selected six fact-checkers to include within our corpus, three from the UK and three from the US. From the US we chose PolitiFact, Factcheck.org and the Washington Post’s Fact Checker. In the UK we selected FullFact, BBC Verify, and Channel 4 FactCheck. 2024 saw both a UK general election and a US presidential election. We intentionally selected a time period in which the content produced by the fact-checkers would not be dominated by these major political events. We therefore selected a time period of January – February 2024.

We collected all of the English-language fact-check articles published by these six fact-checking organisations within our time period. We decided to focus specifically on fact-check articles. We therefore exclude other types of content published by fact-checkers. The approach detailed in this subsection produced a corpus of 400 fact-check articles, which were downloaded and then imported into NVivo 14 for analysis.

We pursued the following research questions:
RQ1 - What types of claims to the fact-checkers check?
RQ2 – What types of issues do the fact-checkers identify with claims?
RQ3 – What types of arguments do fact-checkers advance about claims?

In order to answer these questions we conducted an interpretative, inductive, quali-quantitative content analysis.

Our analytic procedure began with both authors closely reading and independently coding a sub-set of the sample (30 fact-checks, 7.5% of the corpus). We then met to discuss, refine and integrate our initial coding into a provisional coding framework. The purpose here was not to develop a framework that would then be applied deductively, but to collaboratively produce a starting point that would be refined as coding progressed. The remainder of the corpus was then split between both authors, and we each coded our allotted fact-checks in NVivo 14. We met regularly to discuss progress, issues, borderline cases, and changes to the coding framework. In line with our inductive, interpretative approach, we sought to utilise our two perspectives as coders productively by discussing and reflecting on coding decisions and coding framework changes.

Once this initial round of coding was complete, the lead author then organised the codes into broader categories, grouping similar codes together. This process of organisation, nesting and refinement involved splitting some codes and combining others, relegating some codes to sub-categories and promoting some to top-level categories, and recoding some extracts to new or different categories. This process aimed to ensure ‘internal homogeneity’ within and ‘external heterogeneity’ between categories and sub-categories (Braun and Clarke 2006). After meeting to discuss this refined coding framework, both authors embarked upon a final process of refinement, splitting all coded material between us, and each reading through our allocation to ensure that all coding fit within the refined coding framework. This required some further recoding and adjustments to the framework, and again, the authors met regularly to discuss re-coding decisions and framework alterations.

In accordance with our research questions above, we coded the claim types checked in each article (RQ1), and the issues that fact-checkers identified with claims (RQ2). In order to answer RQ3, we coded what we term the ‘tasks of justification’ undertaken by the fact-checkers. The notion of a ‘task of justification’ refers to all instances of a fact-checker presenting arguments and evidence in support of their assessment of a claim, categorised by what the fact-checker is seeking to argue and justify. So whilst the notion of an issue refers to what is identified as epistemically problematic with a claim, the notion of ‘tasks of justification’ refers to the argumentative and evidentiary work undertaken to substantiate that critique.

We also assigned each article with the verdict it reached. Where fact-checking organisations used a rating system we recorded the rating they assigned to the claim (e.g. three Pinocchios or mostly false). For the fact-checking organisations that offer ‘narrative verdicts’ the lead author developed a set of verdict-types based closely on the language used by the fact-checker in their summary or conclusion statement (e.g. misleading, mislabelled, lack of context, etc.). These verdict types were then grouped into six overarching verdict types (true, neither true nor false, false, no evidence, no clear verdict, and multiple verdicts). The false category includes the ‘false’ and ‘pants on fire’ ratings from PolitiFact and the ‘4 Pinocchio’ rating from the WaPo’s Fact Checker, as well as content found to be fake or mislabelled. The ‘neither true nor false’ category includes verdicts that find at least some fault with a claim while falling short of finding the claim false, and includes PolitiFact’s ‘mostly true’, ‘half true’, and ‘mostly false’ ratings, and the Fact Checker’s 1-3 Pinocchio ratings.

Finally, we also coded the sources cited or quoted by the fact-checkers to justify their assessment.

The Data Collection is available to any user without the requirement for registration for download/access.