Can AI look up a stock price correctly? FinSearchComp ranked 21 AI products against 50 analysts
Before an AI can analyze a company, it has to find the right number. FinSearchComp, from ByteDance Seed and Columbia Business School, tests exactly that: fetching live prices, historical figures and multi-step data, the way an analyst does. Web search changes everything, and fresh data is still the weak spot.
Most AI finance tests hand the model a document and ask a question about it. Real research starts earlier: find the latest close, the right fiscal quarter, the figure that was later restated. FinSearchComp, posted to arXiv on September 16, 2025 by researchers at ByteDance Seed and Columbia Business School, is built around that first step.
The test
The benchmark has 635 questions written by 70 professional financial experts, split into a Global subset and a Greater China subset, and three task types:
- T1, time-sensitive data fetching: for example, IBM's latest closing price. The right answer changes daily.
- T2, simple historical lookup: for example, Starbucks' total assets as of September 27, 2020, with rounding allowed.
- T3, complex historical investigation: multi-step questions across several periods and sources.
The team tested 21 systems from August 1 to 20, 2025: 12 consumer web products with search turned on and 9 raw APIs. Live price questions were scored after the local market close on each day. Answers were graded by an AI judge with tolerance bands; on a human-checked sample of roughly 400 answers per dataset, the judge agreed with human labels 95% of the time. Context: n=roughly 400 answers per dataset · Aug 2025 evaluation · FinSearchComp, arXiv:2509.13160, Table 3 · margin: not disclosed
To set a human bar, 50 financial experts who did not help build the dataset took the test with search tools. Their average is the baseline.
The scoreboard
| System | T1 live data | T2 simple lookup | T3 complex | Global average |
|---|---|---|---|---|
| Human experts (50) | 100.0 | 73.3 | 51.4 | 75.0 |
| Grok 4 (web) | 87.3 | 68.1 | 51.2 | 68.9 |
| GPT-5-Thinking (web) | 76.9 | 67.2 | 47.6 | 63.9 |
| Gemini 2.5 Pro (web) | 56.0 | 44.5 | 27.4 | 42.6 |
| DouBao (web) | 61.2 | 33.6 | 22.6 | 39.1 |
| Qwen3-235B (web) | 60.2 | 37.8 | 14.3 | 37.4 |
| Kimi k2 (web) | 30.6 | 47.1 | 10.7 | 29.5 |
| DeepSeek-R1 (web) | 17.9 | 19.3 | 14.3 | 17.2 |
| Gemini 2.5 Pro (API, no search) | 0.0 | 22.7 | 13.1 | 11.9 |
On the Global subset, Grok 4 with web search led at 68.9, 5.0 points ahead of GPT-5-Thinking and 6.1 points behind the human experts. On complex investigations, Grok 4 (51.2) essentially matched the experts (51.4).
The Greater China subset tells a different story
On questions about Chinese markets, Chinese products did markedly better. DouBao (web) led with 54.2, followed by YuanBao with DeepSeek-R1 (52.5) and Grok 4 (51.9). Human experts scored 88.3, so every system stayed more than 34 points behind. The authors attribute the regional split to differences in training data, language and retrieval infrastructure.
One disclosure matters here: DouBao is a ByteDance product, and ByteDance Seed built the benchmark. The data and evaluation harness are public, which lets others check the result, but the leader of one subset and the publisher share a parent company.
What drives the scores
Search is the difference between useless and usable. Without search, models cannot answer live-price questions at all: every raw API without search scored 0.0 on T1 in the Global subset. Across systems, turning search on added an average of 40.8 points on T1, 29.0 on T2 and 8.1 on T3 (system counts per comparison not restated in that passage; no interval disclosed).
A finance data plugin helps on live prices. In a DeepSeek-R1 case study, adding a financial data plugin improved live-data accuracy by 31.9 points over standard web search (single model; no interval disclosed). The authors explain that ordinary web search can surface stale pages and lead the model to report a previous day's price.
Memory produces outdated numbers. When models answered from their training data, they surfaced approximate figures that were often outdated or misaligned with the right reporting period, with higher error rates.
Calendars trip models up. The annotation guide shows why: questions must specify fiscal versus calendar years, because companies such as Nvidia do not align the two. Getting the period wrong is a classic source of confident but wrong answers.
Limits
These results date from August 2025, and the tested products have since been updated. Live-price questions depend on the day they were asked. The human baseline is an average of 50 experts using search tools, not a professional data terminal. And the benchmark measures retrieval accuracy, not investment judgment.
For how retrieval errors turn into confident wrong answers, see our roundup on AI hallucinations on financial numbers.
What it shows / What it doesn't
It shows
- How well consumer AI products retrieve live and historical financial data compared with 50 human experts using search.
- That search access is decisive: models without it scored zero on live-price questions.
- That rankings differ sharply by market, with Chinese products stronger on Greater China questions.
It doesn't show
- How today's versions perform. Tests ran in August 2025.
- An independent ranking for the Greater China subset. The publisher's parent company makes the leading product there.
- Whether an AI's analysis is any good. The test checks whether the right number was found.
- Confidence intervals. None are reported, so small gaps between products may not be meaningful.
Why it matters
Every AI research workflow depends on the first number being right. FinSearchComp shows that with web search, the best assistants find financial data nearly as well as experts on global stocks, and that without live data access they cannot answer price questions at all. When an AI quotes a price or a quarterly figure, the date and period are the first things to check.
Primary sources
- VerifiedFinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning (arXiv:2509.13160) (arXiv / ByteDance Seed, Columbia Business School)Primary source. Task design, evaluation dates, human baseline, Table 3 judge agreement, Table 5 scores and search-ablation figures checked in the HTML version on October 14, 2026.
- LinkedFinSearchComp project page (FinSearchComp team)Linked from the paper; not re-checked.
In our directories
How we used AI: AI tools helped research and draft this article. Every figure was then checked against the primary sources listed above before publication.
Not financial advice. For information and education only. AI model outputs and research results can be wrong and do not predict future returns. Nothing here is a recommendation to buy or sell any asset. See our disclaimer and methodology.