RESEARCH PAPERS

A relevant passage is not necessarily useful for a decision. UsefulBench tests whether information retrieval systems can tell the difference in sustainability reports.

UsefulBench: Can Information Retrieval Find What Is Useful, Not Just Relevant?

· 4 min read

UsefulBench: Can Information Retrieval Find What Is Useful, Not Just Relevant?

Search systems are usually rewarded for finding text that is relevant to a query. But relevance can be a weak target when someone needs evidence for a decision.

Consider the question: “What are a company’s CO2 emissions?” A sustainability report may contain many passages about emissions targets, reporting methods, and climate strategy. Those passages are relevant. Only a smaller set contains the figures needed to answer the question.

The working paper “UsefulBench: Towards Decision-Useful Information as a Target for Information Retrieval” introduces a benchmark for this distinction. It asks whether retrieval systems can identify information that contributes directly to an answer, rather than text that is merely similar to the query.

What UsefulBench Contains

Three professional sustainability analysts reviewed corporate sustainability reports using real-world questions and detailed query descriptions. They assessed passages on two separate scales:

  • Relevance: whether a passage is connected to the question.
  • Usefulness: whether it provides practical information for answering the question or supporting a decision.

The paper describes two dataset variants:

  • UsefulBench-gold: 1,061 annotated query-passage pairs across 15 sustainability reports and 64 queries.
  • UsefulBench-full: more than 53,000 report-query-passage combinations for report-level retrieval experiments.

The analysts first identified passages independently and then resolved labels through consensus discussions. This process matters because usefulness can depend on domain knowledge that is not fully expressed in keywords or short query descriptions.

What the Experiments Show

The authors evaluated conventional retrieval methods, embedding systems, and models from the GPT-4.1 family.

Classic similarity-based retrieval aligned more strongly with relevance than usefulness. LLM-based systems were better able to respond to the usefulness definition, but their gains reached a plateau. Increasing model size did not remove the need for domain expertise.

The researchers also reviewed 50 disagreements between model predictions and human labels. Most were assessed as model errors, while a further group involved ambiguity or expert knowledge that was not sufficiently captured by the written descriptions. Human annotation errors were less common.

Additional experiments with prompting, examples, specialized descriptions, and fine-tuning produced partial improvements. Fine-tuning improved usefulness classification in particular, but introduced a trade-off in calibration. The paper therefore does not present decision-useful retrieval as a solved problem.

Why This Matters for Sustainability AI

Retrieval is a critical part of many sustainability analysis systems. If a pipeline retrieves passages that share the right terminology but do not contain the evidence needed for a decision, a fluent answer can still be incomplete or misleading.

UsefulBench provides a way to test that failure mode. It also supports a more precise discussion about human review. Domain experts are not only checking final answers. Their knowledge helps define what counts as useful evidence in the first place.

The paper focuses on sustainability reports, so its findings should not automatically be generalized to every domain. The authors also note the cost and selection effects of expert annotation and the limited size of the core dataset.

Case Study: Improving Score4More’s AI Retrieval Pipeline

Score4More wanted to improve how its AI pipeline selected evidence from sustainability reports. The practical problem was not only whether a retrieved passage discussed the right topic. The team needed to measure whether that passage contained information useful for answering an analyst’s question.

Climate+Tech helped set up an applied research collaboration between Score4More and University of Zurich researchers, including Tobias Schimanski, around this text-selection problem. Professional Score4More analysts contributed real questions, searched the reports, and labelled relevance and usefulness. The research and engineering contributors turned that expert workflow into the UsefulBench dataset and evaluation study.

UsefulBench is therefore one outcome of the wider collaboration around the Open Sustainability Analyst. It shows a complete research-to-practice path: start with a real pipeline limitation, define the quality question with domain experts, build a benchmark dataset, evaluate current methods, and publish both the findings and limitations.

The authors are Tobias Schimanski, Stefanie Lewandowski, Christian Woerle, Nicola Reichenau, Yauheni Huryn, and Markus Leippold. Their affiliations span the University of Zurich, ETH Zurich, the Swiss Finance Institute, Score4More, and Climate+Tech.

This is the type of applied research structure we aim to support: domain stakeholders help define and annotate the problem, researchers develop and test methods, and product and engineering contributors connect the findings to reproducible evaluation and usable tools. The result is not simply academic access or software delivery. It is a shared research process with explicit evidence, limitations, and practical relevance.

Read the Research

Share: