· tools · 5 min read

Climate+Tech FactChecker: Fighting Climate Misinformation with Open Source AI

Climate+Tech launches an open source AI fact-checking system that combines multiple AI models to evaluate climate claims against scientific sources, achieving 85% agreement with expert consensus.

Climate+Tech launches an open source AI fact-checking system that combines multiple AI models to evaluate climate claims against scientific sources, achieving 85% agreement with expert consensus.

Climate+Tech FactChecker: Fighting Climate Misinformation with Open Source AI

The spread of climate misinformation poses a significant challenge in our digital age. Recent research highlights the scale of this problem:

  • A 2023 study found that over 60% of social media users report encountering climate misinformation at least weekly1
  • Research from MIT shows that false information on Twitter spreads significantly faster than true information, reaching more people and spreading deeper into social networks2
  • The IPCC has identified coordinated disinformation campaigns as a key barrier to climate action and public understanding3

To address this challenge, we’re excited to announce the launch of Climate+Tech FactChecker, an open source AI-powered system designed to fact-check climate claims at scale while maintaining scientific rigor and transparency.

Related Solution: This tool is part of our broader initiative on AI tools for society. Learn more about our complete ecosystem of fact-checking and democratic discourse tools in “AI Tools for Society: Fact-Checking, Democracy & Climate Communication”.

The Advocate-Mediator Architecture

Our system’s unique approach mirrors how expert panels evaluate scientific claims. Multiple AI “advocates” examine evidence independently from different scientific sources, provide structured reasoning with citations, and a “mediator” AI reconciles their findings into a final verdict.

The system components include:

  1. Source Indexing:

    • Chunks IPCC reports and scientific papers into searchable segments
    • Maintains source provenance and context
    • Enables rapid evidence retrieval

  2. Evidence Retrieval:

    • Uses semantic search to find relevant passages
    • Ranks evidence by relevance and authority
    • Preserves source context for verification

  3. Claim Analysis:

    • Multiple advocates examine evidence independently
    • Each provides structured reasoning and citations
    • Verdicts are categorized as correct/incorrect/misleading/unsupported

Implementation Example

The advocate-mediator pattern is straightforward to implement and benchmark. Here’s a simplified example:

from factchecker.strategies.advocate_mediator import AdvocateMediatorStrategy

# Configure how advocates access and use sources
indexer_options_list = [{
    'source_directory': 'data',
    'index_name': 'advocate1_index'
}]

retriever_options_list = [{
    'similarity_top_k': 8,
    'indexer_options': indexer_options_list[0]
}]

# Configure advocate behavior
advocate_options = {
    'max_evidences': 10,
    'top_k': 8,
    'min_score': 0.75
}

# Initialize the strategy
strategy = AdvocateMediatorStrategy(
    indexer_options_list,     # Different scientific source configurations
    retriever_options_list,   # How each advocate searches for evidence
    advocate_options,         # Individual advocate behavior settings
    mediator_options         # How conflicting views are reconciled
)

# Evaluate claims and measure performance
claims_df = pd.read_csv('datasets/expert_verified_claims.csv')
true_labels = []
predicted_results = []

for claim in claims_df['claims']:
    # Get verdict and reasoning from multiple advocates
    final_verdict, verdicts, reasonings = strategy.evaluate_claim(claim)
    
    # Store results for performance analysis
    true_labels.append(claim['expert_label'])
    predicted_results.append(final_verdict)

# Generate performance report
report = classification_report(true_labels, predicted_results)

This code demonstrates how to:

  1. Configure source access and evidence retrieval
  2. Set up advocate behavior parameters
  3. Process multiple claims in parallel
  4. Measure performance against expert-verified data

Flexible Deployment Options

Climate+Tech FactChecker supports both commercial and open-source AI models through simple configuration:

  • Cloud deployment with OpenAI
  • Self-hosted deployment with Ollama
  • Hybrid setups for different components

This flexibility allows organizations to:

  • Control costs by choosing between commercial APIs and open-source alternatives
  • Keep sensitive data on-premises
  • Customize model parameters for their needs

Real-World Performance

In our initial validation testing using Climate Feedback’s expert-verified dataset:

  • Over 80% agreement with expert consensus on clear factual claims
  • Approximately 70% accuracy on complex, nuanced claims requiring detailed context
  • Demonstrated ability to identify common misinformation patterns including:
    • Cherry-picked data
    • Out-of-context statistics
    • Oversimplified cause-effect relationships

These results are based on testing against a dataset of 170 expert-reviewed claims from Climate Feedback, aligning with the methodology described in recent research on automated climate fact-checking4. Read our detailed analysis of this research in our article “Automated Fact-Checking of Climate Change Claims with Large Language Models”.

Key Features

  1. Transparency: Every verdict includes:

    • Specific evidence passages
    • Reasoning chains
    • Source citations

  2. Scalability: The system can:

    • Process thousands of claims daily
    • Continuously incorporate new scientific sources
    • Adapt to emerging climate topics

  3. Scientific Rigor: Built on:

    • IPCC assessment reports
    • Peer-reviewed climate science
    • Expert-verified fact checks

Future Development

We’re actively working on:

  1. Enhanced Source Integration

    • Real-time scientific paper integration
    • Automated source credibility assessment
    • Multi-language source support

  2. Advanced Analysis

    • Image claim verification
    • Multi-modal content analysis
    • Temporal claim tracking

  3. Community Features

    • Expert feedback integration
    • Collaborative source verification
    • Public API access

Project Roadmap

Our development roadmap focuses on making the system more accessible, powerful, and community-driven:

  1. Open Source Models

    • Integration with Llama, Mistral, and other open source LLMs
    • Fine-tuning on climate science datasets
    • Reduced dependency on commercial APIs

  2. Web Interface

    • User-friendly claim submission portal
    • Interactive verdict exploration
    • Source visualization and citation tracking

  3. Optimization Tools

    • Performance benchmarking suite
    • Model comparison framework
    • Resource usage optimization

  4. Pattern Experimentation

    • Testing new advocate-mediator configurations
    • Exploring alternative reasoning frameworks
    • Implementing novel verification strategies

  5. Collaborative Dataset

    • Community-driven claim collection
    • Expert verification workflow
    • Continuous dataset expansion

We invite the community to participate in any of these initiatives. Check our GitHub repository’s issues and projects for specific tasks and milestones.

Join the Project

Climate change is the defining challenge of our time, and accurate information is crucial for informed action. Whether you’re a climate scientist, ML engineer, software developer, or fact-checking professional, your expertise can help improve this vital tool.

The project is open source and available on GitHub. We welcome contributions from:

  • Climate scientists
  • ML engineers
  • Software developers
  • Fact-checking professionals

Visit our documentation to:

  1. Set up your development environment
  2. Understand the architecture
  3. Make your first contribution

Together, we can build a more reliable information ecosystem for climate science.

Connect With Us

Join our community through various channels:

Licensed under AGPL for research and academic use. Commercial licensing available upon request.

Footnotes

  1. Pew Research Center. (2023). “Social Media and Climate Change Information” ↩

  2. Vosoughi, S., Roy, D., & Aral, S. (2018). “The spread of true and false news online.” Science, 359(6380), 1146-1151. ↩

  3. IPCC. (2022). Climate Change 2022: Impacts, Adaptation and Vulnerability. ↩

  4. Leippold, M., et al. (2024). “Automated Fact-Checking of Climate Change Claims with Large Language Models.” arXiv:2401.12566. ↩

Share: