A domain-specific agent harness that any agent runtime with file and shell access can drive. It orchestrates specialized agents, invokes deterministic Python tools, persists structured state, enforces quality gates, and delivers evidence-grounded executive reports.
The project is best described as a custom analytics agent harness that is not tied to one runtime. That is a precise engineering claim: the model and runtime are external and replaceable; the business-specific operating system is authored in this repository.
Underlying LLM reasoning, tool-invocation loop, agent execution environment and foundational permissions come from the host runtime. The tested build uses Claude Code; no Python file calls a model SDK.
Task routing, role definitions, reusable skills, context rules, Python tools, file-based state, approval controls, QA gates and regression tests.
CLAUDE.md · .claude/ · runtime/ · scripts/Everything that produces or checks a number lives in Python and JSON. The runtime only reads Markdown, runs shell commands and writes files, so moving to Codex CLI, Gemini CLI or a similar tool means changing how the workflow is started, not how it works.
runs/{run_id}/scripts/, runtime/, skills/*/scripts/config/schemas/SKILL.md methods and agent role files (plain Markdown)AGENTS.md entry point, read by most coding agents.claude/ and CLAUDE.md/consult: elsewhere, the agent reads .claude/commands/consult.md and follows it/forecast: elsewhere they run one after another with the same output.claude/settings.jsonTested end to end on Claude Code and Google Antigravity. The runbook for other runtimes is in AGENTS.md → Running on Other Agent Runtimes.
Each layer has a concrete implementation and an observable role in system reliability.
Defined workflow routing and stages
Maps task intent into checkpoints, scripts and error exits. Agent execution itself comes from the host runtime.
.claude/commands/consult.mdscripts/run_consulting.pyRole-scoped execution and delegation
27 agent definitions across ML and reporting, with parallel model runners specified for ML workflows.
.claude/agents/forecasting/.claude/agents/regression/Keep analytical payloads out of chat
Central operating rules plus task-specific skills; working data and calculated facts move through files.
CLAUDE.md.claude/skills/*/SKILL.mdUse code for operations with exact outputs
Python tools handle profiling, calculation, schema checks and HTML/PPTX rendering.
runtime/.claude/skills/*/scripts/Persist per-run handoff artifacts
Work files, status envelopes, manifests and report outputs isolate each run and facilitate review.
runs/{run_id}/config/schemas/Block invalid reports before release
Framing, report numerical checks, English-language scan and fault-injection tests.
validate_framing.pyqa_spec.pytest_consulting_gates.pyThe /consult route alternates agent reasoning with deterministic computation and hard validation boundaries. The source directs the agent to correct framing/storyline files and rerun the relevant stages when a gate rejects them.
framing.json and rerun analysis. QA fails → revise storyline.json and rerun reporting. Follow-up hypotheses can return to the analysis stage.runs/{run_id}/ → work/framing.json → work/facts.json → work/storyline.json → report/*Switch between verified source-code paths. These are concise maps of the repository, not simulated agent execution logs.
1. run_consulting.py init → profiling 2. AGENT writes framing.json 3. run_consulting.py analyze → Framing Gate 4. Python computes facts.json 5. AGENT writes storyline.json 6. run_consulting.py report → QA + render # Failure → fix the contract and rerun the gate
Task routing and adaptive analytical checkpoints are encoded in a slash command; Python coordinates deterministic stages.
forecasting/ ├── prophet-agent.md ├── sarima-agent.md ├── lgbm-agent.md ├── baseline-agent.md └── finalize-agent.md ml-core/ ├── model-selector.md ├── model-monitor.md └── ml-report-builder.md
Role definitions are plain Markdown that any runtime can load, while model-specific training and comparison use Python scripts.
analytical-thinking/SKILL.md consulting-analytical-report/SKILL.md data-cleaning/SKILL.md descriptive-diagnostics-analysis/SKILL.md ml-toolkit/SKILL.md ml-report/SKILL.md analysis-report-html/SKILL.md slide-deck-builder/SKILL.md
Skills package guidance and executable scripts. Skills are not separate deployed model services.
runs/{run_id}/
├── manifest.json
├── work/
│ ├── framing.json # agent-generated
│ ├── facts.json # Python-computed
│ ├── storyline.json # agent-generated
│ └── report_spec.json
├── _status/{step}.json
└── report/{report.html,slidedeck.pptx}The system passes analytical artifacts through persisted files, avoiding conversational data transfer and reducing context pressure.
Framing Gate required business context + hypotheses dataset column references Report QA Gate slide structure + fact references numeric value matching (±0.1 rounding) Language Scan Vietnamese-character check on English output
The gates verify specific mechanical constraints; the numeric matcher does not prove semantic correctness for every claim.
E-Wallet 18 / 18 checks passed TechWorld 18 / 18 checks passed Examples of rejected defects: • fabricated headline number • unsupported chart type • incomplete business framing • missing recommendation owner
Deterministic gate tests passed in both sample runs. No general end-to-end agent benchmark is claimed.
All source paths in this section were checked against ai-analyst-system.zip.
Each term is mapped to the code that implements it. Status labels separate what is implemented, what is partial and what is not present.
| Keyword | Implementation | Code-level evidence |
|---|---|---|
| Agent Harness Engineering | Implemented | CLAUDE.md + commands + scripts + gates |
| AI Agent Orchestration | Implemented | /consult + ML workflow commands (Markdown) |
| Multi-Agent Architecture | Implemented | 27 .claude/agents definitions |
| Agent Skills | Implemented | 8 filesystem-based SKILL.md folders |
| Context Engineering | Implemented | CLAUDE.md + skills + file handoffs |
| Tool-Augmented Reasoning | Implemented | Python scripts invoked by agent workflow |
| Human-in-the-Loop | Implemented | --approve-cleaning gate |
| Persistent Agent State | Implemented | runs/{run_id} + manifests |
| Structured Handoffs | Implemented | JSON contracts |
| Agentic Workflow | Implemented | Sequential consulting; parallel ML runners |
| Keyword | Implementation | Code-level evidence |
|---|---|---|
| Deterministic Guardrails | Implemented | Framing/QA/Language validators |
| Fail-Fast Validation | Implemented | exit on invalid framing/report |
| Evaluation-Driven Development | Partial | defect injection; no behavioral eval suite |
| Evidence Grounding | Implemented | facts.json used as report input |
| Numerical Traceability | Partial | value matching, not full semantic lineage |
| Agent Observability | Partial | step status and run logs, no spans |
| Human Approval Checkpoint | Implemented | data cleaning approval |
| Schema Validation | Implemented | config/schemas + runtime/schema_guard.py |
| Keyword | Implementation | Code-level evidence |
|---|---|---|
| LLM–Python Separation | Implemented | reasoning vs deterministic compute |
| Hypothesis-Driven Analytics | Implemented | BAD / MECE / ICE framing |
| Analytical Decomposition | Implemented | KPI and rate/mix |
| Executive Storytelling | Implemented | SCQ / Minto Pyramid |
| Automated Report Generation | Implemented | HTML + PPTX |
| IBCS-Inspired Reporting | Partial | selected conventions, no certification |
| Forecasting / Classification / Regression | Implemented | separate ML workflows |
| MCP / RAG / Vector Database | Not present | not in current repository |
Two sample-data investigations demonstrate consulting output and quality-gate behavior. These diagnostic findings are not measured post-deployment business improvements.
A case study for AI Engineer, Agent Engineer and Analytics Engineer readers. It leads with runtime boundaries, execution and reliability, then the consulting outcomes.
A model can write an executive summary in seconds. That tells us very little about whether the result is worth trusting. The difficult engineering work starts where the model meets data, tools, state, and validation.
I built an AI Analyst System around that boundary. It was built on Claude Code and also run end to end on Google Antigravity, but the analysis does not depend on it: the repository defines its own domain-specific harness: commands that route work, specialized agent instructions, reusable skills, Python tools, persistent file contracts, human checkpoints, and deterministic quality gates. The purpose is to turn an ambiguous business question into a report whose calculations can be inspected independently of the model that writes the narrative.
The host runtime (Claude Code in the tested build) supplies the agent loop and tool execution. My repository supplies the analytics operating model. CLAUDE.md describes the non-negotiable rules, .claude/commands/ defines entry points, 27 agent definitions specialize reporting and ML tasks, and eight SKILL.md files package analytical methods and executable scripts.
This distinction matters: the project is a custom analytics harness built on an existing agent runtime. It does not implement a new general-purpose LLM runtime, and it does not need to claim one.
A recurring failure mode in analytical agents is passing large datasets, intermediate results, and chart payloads through chat history. In this system, durable state stays under runs/{run_id}/. A run has a manifest, step-status files, machine-readable work products, logs, and final deliverables. The agent writes framing.json and later storyline.json; Python computes the authoritative facts.json and drafts the report specification.
That file boundary reduces dependence on the conversation window and makes handoffs reviewable. It also creates a clear revision surface. A failed storyline can be corrected without rewriting the underlying calculations.
The /consult route begins by profiling the input data and assessing cleaning requirements. The agent then applies a Business–Analytical–Data (BAD) framing process: define the decision and KPI, construct a MECE issue tree, identify hypotheses, rank them, and state what the dataset cannot answer. A validation script must approve that framing before downstream analysis.
Python handles KPI calculations, segment comparisons, hypothesis tests, and rate/mix decomposition. Once numerical facts have been written, the agent constructs a consulting storyline using SCQ and Minto Pyramid principles. The report specification passes another quality gate before HTML and PowerPoint rendering.
The repository also contains separate forecasting, regression, and classification commands with model-specific agents and scripts. In these ML workflows, multiple model runners are specified to work in parallel before winner selection and reporting.
The harness uses three targeted checks. The framing validator rejects incomplete business context or invalid field references. A report validator checks structure, recommendation ownership, chart rules, and numerical expressions against the fact store with a rounding tolerance. The final language scan flags Vietnamese diacritics in English-language deliverables.
These checks are useful precisely because they have defined scope. Number matching is not yet full semantic provenance: a number may exist in the facts store but be used for the wrong segment or period. Nor does a language regex prove perfect Business English. Those limitations are part of the engineering backlog.
I reran the consulting gate tests for two cases; each passed 18 of 18 checks, including injected invalid claims and structural defects. That is regression evidence for the gates, not a benchmark of overall agent reliability.
On 32,122 e-wallet transactions, the analysis found the success rate falling from 91.7% to 86.9%; Billing contributed 64.5% of the aggregate deterioration. On 2,008 TechWorld order lines, it investigated a February profit-margin decline of 15.3 percentage points and separated contribution by category size from deterioration within categories. Both runs produced HTML reports and executive slide decks, with 14 slides for the wallet study and 13 for TechWorld.
These are diagnostic findings on sample datasets. They are not proof that a proposed intervention increased business revenue or reduced cost after deployment.
An analytics agent is only as dependable as its surrounding execution system. For this project, that means specialized tools, constrained handoffs, auditable run state, explicit approvals, and tests that make incorrect output fail visibly.
The next iteration will deepen semantic provenance, add richer run tracing and model-behavior evaluations, and strengthen isolation around execution. Those are the components that would move this specialized harness closer to a production-grade agent platform.
What the repository implements today, and what is still missing.
The terminology is aligned to current documentation from LangChain, Anthropic and OpenAI, plus the official IBCS standard. Sources define the patterns; source-code inspection determines what this project actually implements.