"AI fails 81% of finance questions": what the FinanceBench study actually measured

In November 2023, Patronus AI, with researchers from Contextual AI and Stanford, released FinanceBench, an open-book test of whether language models can answer questions about public companies from their filings. The abstract carried one line that went viral: GPT-4-Turbo used with a retrieval system "incorrectly answered or refused to answer 81% of questions." nContext: n=150 questions per configuration (2,400 graded answers) · models as of late 2023 · Islam et al., arXiv:2311.11944, Table 2 · margin: not disclosed

The line is accurate. It is also routinely stripped of its context. Here is what the paper measured.

The test

FinanceBench contains 10,231 questions about 40 U.S. public companies, built from 361 filings released between 2015 and 2023: 10-Ks, 10-Qs, 8-Ks and earnings reports. The authors describe the questions as clear-cut and straightforward, a minimum performance standard. A typical example: what is Boeing's FY2022 cost of goods sold?

The models were not graded on all 10,231 questions. The team built a sample of 150 cases and had experts manually review every answer from 16 model configurations, 2,400 answers in all. The 150 graded cases are open source.

The full results

The key is that each model was tested in different setups, from no documents at all to being handed the exact page with the answer.

The famous 81% is the third row: 13% incorrect plus 68% refusals, for GPT-4-Turbo searching one shared database holding all the filings. Most of that failure was the model declining to answer, not inventing one. nContext: n=150 questions per configuration (2,400 graded answers) · models as of late 2023 · Islam et al., arXiv:2311.11944, Table 2 · margin: not disclosed

Three things the table shows

1. Retrieval was the bottleneck. The same GPT-4-Turbo went from 19% correct with a shared vector store to 50% with a separate store per filing, 79% with the whole filing in its context window, and 85% when handed the exact evidence page. The authors' conclusion: access to the right information and correct retrieval are critical. nContext: n=150 questions per configuration (2,400 graded answers) · models as of late 2023 · Islam et al., arXiv:2311.11944, Table 2 · margin: not disclosed

2. Refusing is better than inventing. GPT-4-Turbo with retrieval mostly refused when it could not find the answer. Llama 2 in the same setup answered wrong 70% of the time. The authors argue a refusal is preferable to a wrong answer because it creates less risk of misplaced trust. nContext: n=150 questions per configuration (2,400 graded answers) · models as of late 2023 · Islam et al., arXiv:2311.11944, Table 2 · margin: not disclosed

3. Even the best setups invent answers. With the full filing in context, Claude 2 was wrong 21% of the time and GPT-4-Turbo 17%, but rarely refused. The paper describes answers that were superficially coherent, sometimes with extensive calculations, and still wrong, which it calls hallucinations, and notes these are harder to catch. nContext: n=150 questions per configuration (2,400 graded answers) · models as of late 2023 · Islam et al., arXiv:2311.11944, Table 2 · margin: not disclosed

Even the best result came with a caveat. The authors called long-context setups unrealistic for enterprise use at the time, because of latency and because some filings did not fit: they truncated inputs at 95,000 tokens when needed.

Why that headline figure is dated

These were late-2023 models with late-2023 retrieval. Since then, benchmark builders have moved on to harder tests. The August 2026 FrontierFinance paper describes financial data extraction as "a narrow slice that current models have largely saturated." The Vals AI Finance Agent v2 benchmark now asks models to do multi-step analyst work with tools, and the best score there is about two thirds.

So quoting the 81% in 2026 as the state of AI is misleading. But the lessons inside the table have aged well. nContext: n=150 questions per configuration (2,400 graded answers) · models as of late 2023 · Islam et al., arXiv:2311.11944, Table 2 · margin: not disclosed

What still applies today

• Ask how the tool finds the page. The gap between 19% and 79% for the same model was entirely about retrieval. That is still where AI research tools differ most, as FrontierFinance found when the same model more than doubled its score with better tools. nContext: n=150 questions per configuration (2,400 graded answers) · models as of late 2023 · Islam et al., arXiv:2311.11944, Table 2 · margin: not disclosed
• Prefer a tool that says "not found." A refusal costs you a minute. A confident wrong number in a model can cost much more.
• Check the math, not just the quote. The worst errors in the study were plausible calculations built on the right documents.
• Small samples, big claims. The viral figure rests on 150 graded questions per setup. It was a careful study, but a narrow one.