# Engineering the Harness Behind an AI Analyst

A model can write an executive summary in seconds. That tells us very little about whether the result is worth trusting. The difficult engineering work starts where the model meets data, tools, state, and validation.

I built an AI Analyst System around that boundary. It was built on Claude Code and also run end to end on Google Antigravity, but the analysis does not depend on it: the repository defines its own domain-specific harness: commands that route work, specialized agent instructions, reusable skills, Python tools, persistent file contracts, human checkpoints, and deterministic quality gates. The purpose is to turn an ambiguous business question into a report whose calculations can be inspected independently of the model that writes the narrative.

## Model versus harness

The host runtime (Claude Code in the tested build) supplies the agent loop and tool execution. My repository supplies the analytics operating model. `CLAUDE.md` describes the non-negotiable rules, `.claude/commands/` defines entry points, 27 agent definitions specialize reporting and ML tasks, and eight `SKILL.md` files package analytical methods and executable scripts.

This distinction matters: the project is a custom analytics harness built on an existing agent runtime. It does not implement a new general-purpose LLM runtime, and it does not need to claim one.

## Context and state are engineering decisions

A recurring failure mode in analytical agents is passing large datasets, intermediate results, and chart payloads through chat history. In this system, durable state stays under `runs/{run_id}/`. A run has a manifest, step-status files, machine-readable work products, logs, and final deliverables. The agent writes `framing.json` and later `storyline.json`; Python computes the authoritative `facts.json` and drafts the report specification.

That file boundary reduces dependence on the conversation window and makes handoffs reviewable. It also creates a clear revision surface. A failed storyline can be corrected without rewriting the underlying calculations.

## Orchestrate reasoning, execute calculations

The `/consult` route begins by profiling the input data and assessing cleaning requirements. The agent then applies a Business–Analytical–Data (BAD) framing process: define the decision and KPI, construct a MECE issue tree, identify hypotheses, rank them, and state what the dataset cannot answer. A validation script must approve that framing before downstream analysis.

Python handles KPI calculations, segment comparisons, hypothesis tests, and rate/mix decomposition. Once numerical facts have been written, the agent constructs a consulting storyline using SCQ and Minto Pyramid principles. The report specification passes another quality gate before HTML and PowerPoint rendering.

The repository also contains separate forecasting, regression, and classification commands with model-specific agents and scripts. In these ML workflows, multiple model runners are specified to work in parallel before winner selection and reporting.

## Make failures visible

The harness uses three targeted checks. The framing validator rejects incomplete business context or invalid field references. A report validator checks structure, recommendation ownership, chart rules, and numerical expressions against the fact store with a rounding tolerance. The final language scan flags Vietnamese diacritics in English-language deliverables.

These checks are useful precisely because they have defined scope. Number matching is not yet full semantic provenance: a number may exist in the facts store but be used for the wrong segment or period. Nor does a language regex prove perfect Business English. Those limitations are part of the engineering backlog.

I reran the consulting gate tests for two cases; each passed 18 of 18 checks, including injected invalid claims and structural defects. That is regression evidence for the gates, not a benchmark of overall agent reliability.

## What the system produced

On 32,122 e-wallet transactions, the analysis found the success rate falling from 91.7% to 86.9%; Billing contributed 64.5% of the aggregate deterioration. On 2,008 TechWorld order lines, it investigated a February profit-margin decline of 15.3 percentage points and separated contribution by category size from deterioration within categories. Both runs produced HTML reports and executive slide decks, with 14 slides for the wallet study and 13 for TechWorld.

These are diagnostic findings on sample datasets. They are not proof that a proposed intervention increased business revenue or reduced cost after deployment.

## The engineering lesson

An analytics agent is only as dependable as its surrounding execution system. For this project, that means specialized tools, constrained handoffs, auditable run state, explicit approvals, and tests that make incorrect output fail visibly.

The next iteration will deepen semantic provenance, add richer run tracing and model-behavior evaluations, and strengthen isolation around execution. Those are the components that would move this specialized harness closer to a production-grade agent platform.