# Building an AI Analyst That Can Defend Its Numbers

An AI analyst can generate a polished executive deck in minutes. Which decision does it support, and where did its numbers come from?

I built an AI Analyst System to make those questions part of the workflow, not an afterthought. It uses an LLM agent runtime (tested on Claude Code and Google Antigravity; any runtime with file and shell access works) for problem framing and executive storytelling, Python for calculations, and file-based contracts to connect the two. The consulting workflow was tested on: 32,122 e-wallet transactions and 2,008 e-commerce order lines.

## Start with the decision

An ambiguous request such as “Why did wallet success decline?” is insufficient input for a diagnostic report. We need the KPI definition, comparison periods, stakeholder, decision to be made, business boundaries, and testable explanations.

The system applies BAD—Business, Analytical, Data—a framework adapted from my analytical-thinking coursework. The Business stage establishes the decision and metric. The Analytical stage structures an issue tree, builds competing hypotheses, and prioritizes tests using ICE. The Data stage maps those questions to available columns and computational methods. A framing validator checks required fields, plausible hypotheses, and references to actual dataset columns before analysis proceeds.

An agent needs the question before producing the chart.

## Separate reasoning from computation

The system has two deliberate LLM checkpoints. An agent writes `framing.json` after reviewing the data profile and cleaning assessment. Once the framing gate passes, Python computes KPIs, segment comparisons, hypothesis tests, and rate-versus-mix decompositions. Results are stored as identified facts in `facts.json`. The agent then uses those facts to build the executive storyline in `storyline.json`.

These handoffs live under `runs/{run_id}/`, alongside the report specification, status artifacts, and generated outputs. Downstream renderers consume the analysis contracts without recomputing the source dataset. This makes the calculation boundary inspectable and lets a failed report be revised without inventing new numbers inside a prompt.

## Put the evidence gate before publication

The report validator checks the storyline structure, slide claims, analytical chart choices, recommendation ownership, and references to known fact IDs. It also scans numerical expressions in report prose against values represented in `facts.json`, with a rounding allowance of approximately 0.1. If the checks fail, the reporting stage stops instead of approving the draft.

This is a useful safeguard, but it is not semantic proof of every claim. The current numerical matcher checks a value against a pool of facts; it does not always establish that the correct fact was used for the correct period, segment, or causal interpretation. A second scan detects Vietnamese diacritics in English-language outputs. Both gates are deliberately bounded and can be strengthened.

The report design follows selected IBCS principles: meaningful action titles, restrained chart types, consistent comparison notation, and a clear separation of findings, implications, recommendations, and limitations. The current consulting renderer produces a self-contained HTML report with embedded chart images and an expandable fact appendix, plus a PowerPoint deck with editable text and image-based exhibits. It is not yet a fully interactive D3 dashboard or an IBCS-certified implementation.

## Test on decisions that matter

On the e-wallet dataset, the transaction success rate declined from 91.7% in July 2023 to 86.9% in August, a 4.7-percentage-point decrease. The engine attributed 64.5% of the aggregate decline to Billing, where transaction volume increased by 50.0%. Authentication accounted for 54.0% of the additional Billing failures, although other error categories also increased. The evidence supports prioritizing Billing investigation; it cannot, by itself, prove a server-capacity root cause.

On TechWorld's first-quarter 2022 sample, profit margin dropped from 29.6% in January to 14.3% in February and recovered to 30.0% in March. Electronics represented 82.3% of the aggregate drop because of its size, yet its own margin decline was smaller than the decline in other categories. That distinction matters: contribution is not the same as relative deterioration. Orders delivered after more than seven days had a 37.2% return rate, compared with 10.0% for faster deliveries; this is a diagnostic association, not a causal experiment.

For both case studies, I reran the repository's defect-injection checks: 18 of 18 passed per case, including rejection of fabricated headline numbers, unsupported chart types, missing recommendation owners, and incomplete framing.

## What I learned

The most useful boundary in an AI analytics workflow is the one between interpretation and measurement. Models are valuable for translating an unclear request into a structured investigation and communicating what the evidence suggests. Calculations, contracts, and quality checks make that investigation reviewable.

The next engineering priorities are context-aware fact references, stronger causal-claim validation, broader language detection, and interactive exhibits. The current HTML reports and executive decks demonstrate the workflow today; the remaining gaps define the next iteration.