Climate Risk Companies Don’t Always Agree: CarbonPlan’s Vendor Comparison

About the Source

CarbonPlan is a US nonprofit (501(c)(3) public benefit corporation) focused on climate science and policy analysis. This piece, published in August 2024 by Oriana Chegwidden, Maggie Koerth, and Jeremy Freeman, is independent nonprofit research rather than peer-reviewed academic work or a coalition-produced industry benchmark. CarbonPlan’s own disclosure states it received no financial support for the work, and it’s released under a CC-BY 4.0 license with the underlying data request, data, and analysis code public on GitHub.

Introduction

If you look up a home on Realtor.com, you can see a fire, flood, or heat risk score supplied by the company First Street. If you’re a bank, insurer, or city planner, you might buy similar scores from any of a dozen other climate analytics companies. CarbonPlan’s starting question was simple: if you asked several of these companies for a risk assessment of the exact same buildings, would you get the same answer? To find out, they did what any consumer could, in principle, do — they asked.

Nine Companies, Two Answers

CarbonPlan requested historical and future risk scores for 342 real addresses: fire risk for 128 US Postal Service locations in California, and flood risk for 214 post office and public school locations in New York. This was a modest, narrowly scoped, non-commercial request — small enough to fit in an email, framed transparently as research in the public interest, with no request for payment.

Of the nine companies contacted, only two agreed to participate: Jupiter Intelligence and XDI (part of The Climate Risk Group). Four explicitly declined, including two prominent, well-known vendors — Verisk and First Street, the same company that supplies Realtor.com’s public-facing risk scores. First Street pointed CarbonPlan to its own public website instead, but that site’s terms of service appeared to prohibit the kind of reuse CarbonPlan’s comparison required, and a follow-up request for explicit permission went unanswered. Three other companies never responded at all.

CarbonPlan’s first, and arguably most important, finding has nothing to do with climate science: an industry that sells risk transparency to its customers was largely unwilling to be transparent about its own methods and outputs, even for a narrow, good-faith research request from a nonprofit.

Where Jupiter and XDI Agreed — and Where They Didn’t

For the two companies that did participate, CarbonPlan used Kendall’s Tau, a statistic that measures how consistently two sets of rankings agree with each other, independent of whether one company’s scores run systematically higher or lower than the other’s. A value of 0 means no more agreement than chance; a value of 1 means the rankings are identical.

California wildfire risk (128 locations): Tau = 0.25 (historical), 0.22 (2100 projection) — statistically significant, but weak. Both companies agreed on the coarse regional pattern (higher risk in the wooded central Sierras) and that climate change would increase fire risk over time — but they agreed on where that increase would occur in only 12% of locations (restricting to properties not already at each vendor’s maximum risk level narrows this further: Jupiter projected increases at 63% of locations, XDI at 38%, a large gap even in direction of change). The two vendors also use different underlying hazard models entirely: Jupiter bases historical fire risk on the US Forest Service’s Wildfire Risk to Communities dataset and CMIP6 climate projections, while XDI uses a hot-dry-windy index combined with an empirical ignition probability model and CMIP5 data downscaled via CORDEX.
NYC coastal flooding (90 locations): Tau = 0.55 in the 2100 projection — the strongest agreement CarbonPlan found, though still far from the 1.0 that would indicate full consistency. In the historical period, so few locations were flagged as at-risk by either vendor that a meaningful rank comparison wasn’t possible at all.
New York State riverine and surface-water flooding (124 locations): Tau as low as 0.19, and in most categories too few properties were flagged as at-risk by either vendor to support a robust comparison at all.

Across all three case studies, a consistent pattern emerged: broad regional agreement (Sierra Nevada forests being fire-prone in general; the coastline being more flood-exposed than inland areas) can coexist with substantial disagreement at the level of the individual address — which is exactly the level of granularity that a bank underwriting a specific mortgage, or a school district deciding where to invest in flood defenses, actually needs.

Why This Matters Beyond Two Vendors

CarbonPlan is explicit that this is a narrow study — two vendors, a few hundred addresses, four hazard types — and that it can’t say whether the seven non-participating companies would have shown more or less agreement. But the piece argues the industry’s reluctance to be compared is itself informative, and that this has real consequences: if an insurer or a corporation can access several different climate risk assessments for the same asset, nothing stops them from selectively reporting whichever number is most favorable to their argument — a higher score to justify a premium increase, a lower score to reassure investors — all while presenting the figure as an objective fact rather than one of several plausible model outputs.

A Recurring Proposal: Treat This Like Climate Model Intercomparison

Like the ILN’s later “Why Vendors Disagree” study, CarbonPlan lands on essentially the same structural recommendation: a comprehensive, mandatory, publicly documented, asset-level, multi-hazard comparison of climate risk vendors, modeled on the Coupled Model Intercomparison Project (CMIP) that climate science already uses to compare and understand disagreement between global climate models. CarbonPlan goes a step further than most industry-produced studies on this point, arguing that if vendors won’t submit to this voluntarily — and their own experience getting seven out of nine companies to decline or ignore a modest, free data request suggests they won’t — then regulation may be the only path to it. In the meantime, they argue for public-sector alternatives (building on tools like the US National Risk Index) as a benchmark and a check against purely private, unaccountable risk scoring.

Conclusion

CarbonPlan’s study is small by design, and it says so itself: two vendors, a few hundred addresses, four hazards, no attempt to adjudicate which vendor’s number is “right.” What it adds to the broader picture of climate risk data quality is a distinctly different kind of evidence than the ILN or GARP benchmarks: those studies got vendor cooperation and measured the resulting disagreement in some detail; CarbonPlan’s core finding is about the lack of cooperation itself — seven of nine companies asked for research-grade transparency by a credible, non-adversarial nonprofit simply declined or ignored the request. Read together, the ILN, GARP, and CarbonPlan studies triangulate on the same conclusion from three different angles: the market for climate risk data is opaque enough, and disagreement large enough, that buyers, regulators, and the public would all benefit from independent, standardized, and ideally mandatory comparison — something none of the three studies believe the market will produce on its own.

Access the Source Material

Chegwidden, O., Koerth, M., & Freeman, J. (2024, August 9). Climate risk companies don’t always agree. CarbonPlan. (CC-BY 4.0; data and code on GitHub)