CHATREPORT: Democratizing Sustainability Disclosure Analysis through LLM-based Tools

In the realm of corporate sustainability, the accurate analysis of sustainability reports is paramount. The study titled “CHATREPORT: Democratizing Sustainability Disclosure Analysis through LLM-based Tools” introduces an innovative approach to this challenge by leveraging large language models (LLMs). This article provides a comprehensive summary of the study, highlighting its key findings and implications for the future of sustainability reporting.

Authors

The CHATREPORT study is authored by:

Jingwei Ni (University of Zurich, ETH Zurich)
Julia Bingler (University of Oxford, Council on Economic Policies)
Chiara Colesanti-Senni (University of Zurich)
Mathias Kraus (FAU Erlangen-Nürnberg)
Glen Gostlow (University of Zurich)
Tobias Schimanski (University of Zurich)
Dominik Stammbach (ETH Zurich)
Saeid Ashraf Vaghefi (University of Zurich, Eawag)
Qian Wang (University of Zurich)
Nicolas Webersinke (FAU Erlangen-Nürnberg)
Tobias Wekhof (University of Zurich, ETH Zurich)
Tingyu Yu (University of Zurich)
Markus Leippold (University of Zurich, Swiss Finance Institute)

For more details, you can read the full paper here.

The Need for Advanced Analysis Tools

Corporate sustainability reports are essential for evaluating a company’s environmental and social impact. However, traditional analysis methods are labor-intensive and prone to human error. The CHATREPORT study introduces an LLM-based system to automate this process, enhancing both accuracy and efficiency. This system is designed to democratize the analysis of sustainability reports, making it accessible to a broader range of stakeholders, including smaller organizations that may not have the resources to conduct such analyses manually.

Key Features of the CHATREPORT System

The CHATREPORT system addresses several challenges in sustainability report analysis:

1. Mitigating LLM Hallucinations

LLMs often generate plausible but incorrect information, known as hallucinations. CHATREPORT incorporates mechanisms to minimize these, ensuring reliable and trustworthy analysis. The system grounds its analytical prompts with retrieved information from the target report and makes the answers traceable, helping users identify and verify the sources of information.

2. Reducing Expert Involvement

Typically, domain experts validate sustainability report analyses. CHATREPORT automates much of this validation, saving time and resources. The system includes an automatic prompt engineering algorithm that transfers experts’ feedback on specific outputs to general analysis guidelines, which are then used to improve future analyses.

3. Enhancing Data Accuracy

Using advanced AI methodologies, CHATREPORT improves the accuracy of data extracted from sustainability reports, crucial for informed decision-making. The system evaluates the reports’ conformity to TCFD guidelines, providing a conformity score that benchmarks disclosure quality.

Methodology

CHATREPORT employs a novel LLM-based approach to analyze corporate sustainability reports. The LLM is trained on a diverse dataset of reports, enabling accurate information extraction. The system’s performance is then evaluated on a separate set of reports. The pipeline includes modules for report embedding, summarization, TCFD conformity assessment, and customized question answering.

Report Embedding

The report is split into text chunks, which are transformed into a vector space representation for future reference and semantic searching. This allows the system to handle the limited context window of LLMs effectively.

Report Summarization

The system summarizes the report based on TCFD’s eleven recommended aspects, retrieving relevant parts from the report and prompting the LLM to summarize the disclosure.

TCFD Conformity Assessment

The system evaluates the reports’ conformity to TCFD guidelines, generating an analysis paragraph and a conformity score for each recommendation.

Customized Question Answering

Users can conduct personalized analyses by posing customized questions. The system retrieves relevant contexts and answers the questions using the LLM.

Findings and Implications

The study shows that LLM-based tools can significantly improve the efficiency and accuracy of sustainability report analysis, democratizing the process for a broader range of stakeholders, including smaller organizations.

Key findings include:

Improved Efficiency: The system reduces the time required to analyze sustainability reports.
Enhanced Accuracy: Advanced AI methodologies ensure higher accuracy in data extraction and analysis.
Broader Accessibility: Reduced need for expert involvement makes the analysis more accessible to smaller organizations.

Future Directions

The study suggests several future research and development avenues:

Integration with Existing Systems: Exploring how CHATREPORT can be integrated with existing sustainability reporting frameworks.
Expanding the Dataset: Increasing the diversity and size of the training dataset to improve accuracy and reliability.
User-Friendly Interfaces: Developing interfaces to make CHATREPORT more accessible to non-experts.
Mitigating Hallucinations: Further research into reducing hallucinations and improving the interpretability of the system’s outputs.

Call for Feedback and Collaboration

We invite users and stakeholders to provide feedback to improve CHATREPORT. Your insights are invaluable for optimizing the tool’s structure and usage. Visit our GitHub repository at https://github.com/climateandtech/chatreport to contribute feedback and collaborate on this ongoing project.

Conclusion

CHATREPORT represents a significant advancement in sustainability report analysis. By leveraging LLM-based tools, it addresses challenges such as hallucinations and the need for expert validation, making the process more efficient and accurate. The study’s insights will be crucial for developing more advanced and accessible tools for sustainability reporting.

For more details, you can read the full paper here.