Same Question, Different Answer

An advisor at a mid-size RIA runs a portfolio analysis on Monday using a popular AI tool. She gets a Sharpe ratio of 1.19 for a client’s equity portfolio. On Thursday, preparing for a client review, she runs the same analysis. The answer comes back 0.87. She uses the Monday number, presents it to the client, and moves on.

She has no way of knowing which answer, if either, was correct.

A Different Kind of Failure

Our first white paper showed that general-purpose AI tools fail approximately 85% of investment tasks advisors perform daily. The reaction we heard most was some version of the same question:

“But when they do answer, are the answers at least consistent?”

It’s a reasonable question. We designed a study to find out.

Between April and May 2026, we asked six AI systems ten identical financial calculation questions, each in a fresh session, five times each. We measured two things: how consistent the answers were across runs, and how accurate those answers were against independently calculated ground truth values.

The results reveal something more concerning than simple failure.

We Put It to the Test

The six platforms we tested:

• DeepVest AgentLab: purpose-built investment AI for financial advisors
• ChatGPT 5.3 (OpenAI)
• Claude Opus 4.7 (Anthropic)
• Perplexity (Sonar model)
• Gemini 3.1 Pro (Google)
• SuperGrok (Grok 3 Expert, xAI)

The ten questions covered real advisor workflows: volatility and drawdown calculations, Sharpe ratios, portfolio construction, correlations, and minimum-variance optimization. Each question was asked five times in a fresh session to eliminate memory effects. We measured consistency using the Coefficient of Variation (CV%), a standard statistical measure. A CV of 0% means identical results every time.

Three Categories of AI Behavior

The data revealed something more nuanced than pass or fail.

Category 1: Deterministic and Accurate

Only one model achieved both perfect consistency and accuracy. DeepVest AgentLab returned identical answers across all five runs on all ten questions (CV of 0% across the board) and matched independently calculated ground truth across all 10 questions within methodology parameters (8 with zero deviation, 2 with small positive deviations reflecting realistic execution modeling rather than theoretical assumptions).

Category 2: Deterministic but Inaccurate (The Most Dangerous Pattern)

One model showed perfect consistency but material accuracy problems: SuperGrok (CV = 0% on all ten questions).

An advisor using this tool regularly would see the same answer every time and reasonably conclude the tool is reliable. But it diverged materially from ground truth on several questions. SuperGrok’s portfolio value was off by 6.1%, its annualized return off by 26.6%.

This is the category that should worry advisors most. Inconsistency is visible: advisors notice when a tool gives different answers on different days. Consistent inaccuracy is invisible until a client’s accountant catches it, or an SEC examiner asks you to explain your methodology.

Category 3: Unreliable (Inconsistent and/or Incomplete)

Four models exhibited patterns that disqualify them from production financial workflows:

Claude Opus 4.7 showed strong consistency on simple calculations but critical reliability failures: it refused to answer 32% of all attempts (16 of 50), with complete failure on two moderate-complexity questions (portfolio tracking and rebalancing calculations). On questions it did answer, it showed high inconsistency on Sharpe ratio calculations (CV 10.1%) and material accuracy deviations on several metrics.

ChatGPT 5.3’s correlation calculation varied by 43.7% across five identical runs. It refused to answer 3 of 50 attempts and showed accuracy deviations averaging 20.7% from ground truth.

Perplexity showed the widest variation: Sharpe ratio varied by 93.7%, correlation by 68.1%. It refused 8 of 50 attempts (16%) and averaged 26.6% deviation from ground truth when it did answer.

Gemini 3.1 Pro showed meaningful variability on complex calculations, with CV reaching 11.5% on Sharpe ratios and 11.4% on correlation. It refused 2 of 50 attempts and averaged 22.1% deviation from ground truth.

The Consistency Breakthrough: Why It’s Not Enough

Two models (DeepVest and SuperGrok) achieved perfect consistency (CV = 0% on all questions). Frontier models are beginning to solve the reproducibility problem.

But consistency and accuracy are not the same thing. Of the two perfectly consistent models, only one matched ground truth across all questions. The other produced confident, identical answers that were materially wrong on complex calculations, creating exactly the false confidence that makes compliance officers nervous.

Meanwhile, models that appeared consistent on simple questions broke down on moderate-complexity workflows. Claude Opus 4.7, for instance, achieved CV = 0% on seven of ten questions, yet refused to complete 32% of all calculation attempts. Consistency on simple tasks does not predict reliability on real advisory workflows.

What This Means for Your Practice

The regulatory context makes this more than an accuracy question. The SEC’s 2026 examination priorities focus on whether firms can explain and document AI-assisted decisions. An AI tool that produces different answers on different days cannot produce a defensible audit trail. An AI tool that is consistently wrong generates documentation that is confidently incorrect. An AI tool that intermittently refuses to answer creates workflow failures at the exact moment an advisor needs a reliable result.

For advisors evaluating AI tools, this study suggests three questions worth asking before use with a client:

• Does this tool give the same answer twice?
• If it does, can you verify that the answer is correct?
• Can it complete the calculation 100% of the time, or will it intermittently refuse?

The architecture that answers yes to all three questions exists. It is not a future state; it is a design choice.

Download the Full White Paper

The complete report includes full consistency and accuracy analysis, ground truth validation methodology, and a detailed breakdown of all ten questions.

About This Research

This study was conducted by DeepVest between April and May 2026 to evaluate both the consistency and accuracy of AI tools for professional investment management workflows. All testing was performed using the latest available versions of each platform, with each question asked five times in fresh sessions to measure reproducibility.

DeepVest is a purpose-built portfolio intelligence platform designed specifically for financial advisors. Unlike general-purpose AI tools, DeepVest provides institutional-grade data, complete audit trails, and deterministic results built for fiduciary-level work.

For questions about this research: contact [email protected].