Bridging Social Media and Science: The ClimateCheck Dataset for Fact-Checking Climate Claims

How do you fact-check a viral tweet about climate change? Until now, there was no dataset connecting the informal language of social media to the formal language of scientific research. ClimateCheck changes that.

Published at the Fifth Workshop on Scholarly Document Processing (SDP 2025), the ClimateCheck dataset provides 435 unique climate-related social media claims linked to 3,048 scientific abstracts—all manually annotated by climate science experts. Unlike existing datasets that rely on Wikipedia or focus on other domains, ClimateCheck directly bridges public discourse with peer-reviewed research.

How It Was Built

The team collected 1,325 claims from five sources—Twitter, Reddit, news articles, and fact-checking sites—then filtered and processed them to match real social media language. Claims from formal sources were deliberately rephrased into tweet-style language using LLMs, achieving 99.87% confidence in matching social media style while preserving scientific content.

The publication corpus drew from S2ORC and OpenAlex, totaling nearly 400,000 climate science papers after quality filtering (minimum 10 citations). To link claims with relevant abstracts, the researchers used a hybrid retrieval pipeline: BM25 sparse retrieval followed by neural reranking and a pooling approach where six different AI models voted on which abstracts provided supporting or refuting evidence.

Five graduate students in climate sciences then manually annotated 3,048 claim-abstract pairs, labeling each as “Supports,” “Refutes,” or “Not Enough Information.” The final dataset includes 259 training claims (1,144 pairs) and 176 test claims (1,904 pairs), with inter-annotator agreement of 0.69 (Cohen’s κ)—indicating substantial agreement despite the task’s inherent subjectivity.

What Makes It Unique

The dataset spans 16 climate topics—from renewable energy to extreme weather to biodiversity. But its real innovation lies in the linguistic style: these are claims written in the informal, colloquial language of social media, complete with slang and unconventional grammar. This matters because fact-checking models need to understand how people actually talk about climate change online, not just formal statements from news articles.

The label distribution reflects the reality of fact-checking: 44% “Not Enough Information,” 39% “Supports,” and only 17% “Refutes.” When the researchers tested different AI models on the dataset, instruction-tuned LLMs (like Yi-1.5-9B) performed much better than standard classification models, achieving 0.61 F1 compared to 0.32-0.33 for BERT-based classifiers—suggesting that climate fact-checking requires more sophisticated reasoning than general-purpose NLI models provide.

Access & Impact

The dataset is freely available on HuggingFace along with the publications corpus and code. The paper by Raia Abu Ahmad, Aida Usmanova, and Georg Rehm appears in the ACL Anthology (pages 42-56, SDP 2025).

Since its release, ClimateCheck has powered the ClimateCheck@SDP 2025 Shared Task, where multiple teams developed systems for abstract retrieval and claim verification. The dataset provides a foundation for training AI models to verify climate claims against scientific evidence—a crucial capability as climate misinformation continues to spread on social media.

Related Resources:

• LLM vs BERT Comparison Research
• ClimateCheck 2025 Competition
• Climate Fact-Checker Collaboration Project