Researched by BEAST Library Curator | Verified by Harper | Quality: 8.4/10
Every AI-energy-management vendor pitch has the same shape: a case study, a percentage, and an implicit promise that your building will look like the case study. New York State's energy agency just spent seven years and $60.5 million finding out how often that promise holds — across 654 real sites, 327 projects, and 57 vendors. The answer, published in a November 2025 impact evaluation, is the most useful single data point a facility GM will find this year for deciding which AI-HVAC vendor category to trust with a contract: verified savings landed at 48% of what vendors originally claimed — and two entire vendor categories verified essentially zero.
This report is about that evaluation, what it says about vendor type as a predictor of real savings, and the one M&V limitation that tripped up even the state's own auditors. For general information only; this is not professional advice, and your results may vary by jurisdiction, vendor, and portfolio.
The study: NYSERDA's Real-Time Energy Management (RTEM) impact evaluation
NYSERDA's RTEM program funds automated energy-management technology — software overlays, fault detection and diagnostics (FDD), automated system optimization (ASO), and OEM building-automation controls — across commercial, industrial, and small-to-medium-business buildings in New York State. The November 2025 RTEM Commercial & Industrial impact evaluation (NYSERDA, published by the program's independent evaluation contractor) covers installations from January 2017 through Q2 2024: 654 sites, 327 applications, 57 unique vendors, $60.5M in NYSERDA incentives, and 222.65 million square feet of impacted floor area — verified via engineering review of vendor service reports plus a sampled 240-site savings review (per Memoori's AI in Smart Commercial Buildings 2026 report, which first flagged this dataset as the field's clearest vendor-claim-vs-verified benchmark).
The headline metric is the Verified Gross Savings Realization Rate (VGSRR) — the ratio of independently evaluated savings to what the program originally estimated (largely from vendor-submitted service reports). From program inception through Q2 2024, the overall VGSRR was 48% for electricity and 47% for gas. In plain terms: for every 100 kWh a vendor's report claimed as saved, an independent engineering review could verify roughly 48.
| Evaluation period | Electric VGSRR | Gas VGSRR |
|---|---|---|
| Q1 2017 – Q4 2020 | 32% | 33% |
| Q4 2020 – Q4 2021 | 61% | 34% |
| Q1 2022 – Q2 2024 (current) | 62% | 66% |
| Overall, program inception | 48% | 47% |
Two things jump out. First, realization rates nearly doubled over seven years — 32% to 62% for electricity — which is genuinely good news: M&V discipline, metering, and vendor accountability in this program are improving. Second, even at that improved current-period rate, verified savings are still just 62 cents on the dollar of what was originally projected. That is the honest floor a facility GM should underwrite to, not the vendor's opening number.
The number that should change how you screen a vendor: realization rate by category
The evaluation didn't stop at a portfolio average. Per the NYSERDA study, Table 2-25 breaks the VGSRR down by vendor category, weighted, with confidence intervals — worth pasting into your next vendor RFP scorecard:
| Vendor category | Electric VGSRR | 90% CI | Verified savings, % of baseline | Sample (n) |
|---|---|---|---|---|
| ASO Reporting Focus Only | 122% | ±69% | 12.62% | 8 |
| Software Overlay Only | 105% | ±44% | 10.21% | 13 |
| ASO + FDD Reporting Focus | 44% | ±38% | 4.33% | 19 |
| Software Overlay + Controls Install | 5% | ±4% | 0.47% | 15 |
| FDD Reporting Focus Only | 0% | ±0% | 0.00% | 9 |
| OEM Controls Only | 0% | ±0% | 0.00% | 8 |
| Overall (weighted) | 62% | ±30% | 6.08% | 38 |
Source: NYSERDA RTEM Commercial & Industrial Impact Evaluation, November 2025, Table 2-25 (direct electric benefits realization rates by vendor type, weighted, Q1 2022–Q2 2024 evaluation period).
Read this table carefully before you read the vendor's slide deck. Vendors whose service focus was FDD-reporting-only or bare OEM controls verified zero measurable savings against baseline in this evaluation. Vendors whose focus was ASO reporting or a straight software overlay verified realization rates at or above 100% — meaning, on the sites reviewed, they actually delivered more than the program's original estimate. The evaluators' own explanation lines up with practitioner intuition: sites using FDD-focused services tended to chase cost reductions and maintenance fixes that don't necessarily reduce energy use, and historically didn't convert into the capital projects that actually move the needle. ASO and software-overlay vendors, by contrast, disproportionately recommended and implemented HVAC control measures — the intervention type the evaluation found most reliably tied to real savings.
Read the confidence intervals honestly. The FDD-only and OEM-controls-only 0% figures rest on samples of 9 and 8 sites respectively — small, but a hard 0% with a ±0% interval is not statistical noise; every reviewed site in those categories verified no savings. The 105%–122% figures for software-overlay and ASO vendors carry wide intervals (±44% and ±69%, on n=13 and n=8) — genuinely strong performance, but don't treat "122%" as a number you can bank a pro forma on. Ask your own vendor for their category-specific realization rate and sample size, not just their headline case study.
Even NYSERDA's evaluators couldn't finish IPMVP Option C
The most sobering line in the report has nothing to do with vendor performance — it's a methodology footnote. The evaluation's stated intent was to run IPMVP Option C: Whole Building Verification against building-level consumption data for every reviewed site, as the gold-standard check on measure-level vendor claims. The result: "In all cases, data gaps prevented full Option C analysis; however, available building-level consumption data were used as a reasonableness check on measure-level reported savings, rather than to confirm the presence or absence of additional savings."
Read that again. This is a $60.5M, seven-year, state-funded evaluation with subpoena-level access to utility billing data and vendor cooperation — and it still could not complete full whole-building M&V verification on a single one of the reviewed sites. The evaluators fell back to engineering review of vendor-reported, measure-level savings (closer to IPMVP Option A/B logic) with building consumption data only as a sanity check. If your own portfolio's M&V process claims Option C rigor on a spreadsheet built from monthly utility bills, this is the reference point that should make you ask harder questions about what "Option C" actually meant in your contract.
The APAC read: same discipline, no local dataset to lean on yet
New York is not Taipei or Singapore, and no equivalent public, multi-year, cross-vendor realization-rate study exists yet for an APAC AI-HVAC market. That absence is itself the actionable point. A Taiwan facility team weighing a TSMC-adjacent AI-HVAC pilot, or a Singapore team evaluating a BCA Green Mark energy-management upgrade, cannot pull a local RTEM-style dataset to sanity-check a vendor's savings claim — which means the burden of proof sits entirely on the contract terms you negotiate, not on ambient market data doing the work for you. The vendor-category lens above is portable regardless of jurisdiction: ask which of the six RTEM categories your prospective vendor most resembles (pure FDD/alerting vs. active ASO/controls implementation), because that classification predicted realization rate more reliably than any single vendor's marketing claim in the largest independent dataset available.
Here's what I'd do if this were my building
- Classify your vendor before you sign, not after. Ask directly: are you FDD/alerting-only, or do you actively implement ASO/HVAC control measures? The NYSERDA data says this single distinction predicts realization rate better than any vendor's own case study.
- Underwrite to 48–62%, not 100%. Build your ROI model off the program-wide realization rate range, and treat the vendor's original projection as a ceiling, not a baseline expectation.
- Ask for sample size, not just a percentage. A quoted realization rate on n=8 sites with a ±69% confidence interval is a very different claim than the same number on n=100. Insist on both.
- Don't assume your "Option C" M&V clause is actually Option C. If your contract's M&V process runs on monthly bill regression without full interval-level whole-building data, you likely have an engineering-review-grade verification, not a true Option C — and that's fine, as long as everyone signing the contract knows which one they're getting.
- Push for HVAC control implementation, not just diagnostics. The categories that verified real savings were the ones that acted on the building (controls installs, ASO), not the ones that only reported findings (FDD-only, alerting).
The uncomfortable takeaway for 2026: the single best predictor of whether your AI-energy-management vendor will deliver isn't the AI. It's whether their service model ends in an implemented control change or ends in a PDF report. NYSERDA's 654-site audit put a number on that difference — 0% for report-only categories, over 100% for controls-implementing ones — and it is the most concrete M&V-standards data point a facility GM has had to negotiate from all year.
For more on the verification stack behind AI-HVAC and M&V claims, see our M&V standards library and the full AISB Library. This report is provided for general information, is not professional advice, and your results may vary.
Have a question about this topic? Ask our CRE AI Agent →