Methodology
Measuring AI answer accuracy means asking AI engines the questions real investors ask, then checking each answer against the company’s own SEC filing. MS House does this in the Investor Answer Audit across ChatGPT, Google AI Overviews, Google AI Mode and Microsoft Copilot. For earnings, the source of truth is the latest Form 8-K filed under Item 2.02 on SEC EDGAR. Every answer gets 1 of 6 verdicts.
Where do the correct figures come from?
The correct figures come from the company’s filings on SEC EDGAR. For each release, MS House finds the latest Form 8-K filed under Item 2.02 (Results of Operations and Financial Condition) and reads the press release exhibit. Each fact we use is stored with a word-for-word quote from that exhibit. We do not use news articles or data sites as the source of truth.
Which AI engines do you check?
MS House checks ChatGPT, Google AI Overviews, Google AI Mode and Microsoft Copilot. Answers are collected through an AI search analytics platform, from a United States location. We only list an engine on this site when we actively collect its answers.
Which questions do you ask?
The audit asks the questions real investors are likely to ask, in groups: results, guidance, capital return, leadership, segments, outlook, and where to find information. Questions use the company’s name the way an investor would type it.
MS House does not use a fixed question list. Our prediction model generates a synthetic set of the questions real investors are likely to ask about your company. It is built from question patterns on Google, Bing, DuckDuckGo, ChatGPT, Reddit and Quora, and from academic studies of how investors search. The set is made for each company and updated every quarter. Questions about stable facts, such as who the CEO is, mostly stay the same, so those answers stay comparable quarter to quarter.
What are the 6 verdicts?
| Verdict | Meaning |
|---|---|
| Correct | The answer matches the filing. |
| Outdated | The answer gives a figure from an earlier period. |
| Wrong | The answer contradicts the filing. |
| Partly wrong | Some parts are right, others are wrong or missing. |
| No answer | The engine did not answer the question. |
| Not checkable | The question needs facts the filing does not contain. |
Wrong, outdated and partly wrong answers count as errors. Severity is high for financial figures and guidance, medium for leadership and partial answers, and low for the rest.
How do you find the source of a wrong answer?
Each engine answer comes with the websites it cited. For each error, the chief agent names the cited site most likely behind it. When an engine cites no source, the audit says so. This is a best judgment, not proof, and the audit labels it as “likely source”.
How do you measure AI answer accuracy over time?
MS House measures accuracy over time by comparing verdicts from one audit to the next. Each Investor Answer Audit is a dated sample of what ChatGPT, Google AI Overviews, Google AI Mode and Microsoft Copilot told investors, checked against the company’s SEC filing.
The question set is made for each company and updated every quarter. Questions about stable facts, such as who the CEO is, mostly stay the same, and these repeat questions show whether an error is closing or still being repeated. In the Annual Program, MS House re-checks each fix after 14 days, so the team can see whether the engines now give the filed figure.
What are the limits of this method?
- AI answers can change daily, so each answer is a dated sample.
- Answers depend on location and wording. We use one location (United States) and fixed wording.
- The verdict is made by an AI model with the filing quote in front of it. We show the quote next to every verdict so your team can check it.
See how this looks in practice on the sample audit, or read what an investor answer audit is.