Jonathon's AI Wiki

Build a Custom AI Harness on the Claude Agent SDK (Claire Vo)

10 min read

Source: raw/How_to_build_a_custom_AI_harness_with_Claude_SDK.md — YouTube, How I AI (Claire Vo), video id ofS-4RRw9zw

Creator: Claire Vo (How I AI) URL: https://www.youtube.com/watch?v=ofS-4RRw9zw Duration: data not available Platform: YouTube

Claire Vo demystifies the word everyone uses but few define — the harness — by building one live on the show. Her definition is deliberately plain: “a harness is some code around an AI agent that makes it more effective.” She builds a working Sentry-debugging harness on the Claude Agent SDK, then reverse-engineers the reusable recipe so you can build your own for any repeatable workflow. This article is that recipe: the definition, the decision of when to build one, the four load-bearing architecture choices, and the 7-step build sequence.

Key Takeaways

  • Definition, stripped of mystery. A harness is just code wrapped around an AI agent to make it more effective for a specific job. Its three parts are specific context + specific actions + specific outcomes. Putting AI inside the harness is optional — “can that code have AI in it? Sure. Does it have to? Not necessarily.”
  • The thesis: “a wrapper is just a harness.” Cursor is “a really complex harness”; Claude Code and Codex are “very complex coding harnesses.” They are all code that wraps AI calls to make them better at a specific job. The broad end (general coding agents) and the narrow end (a one-workflow harness) are the same shape at different scope.
  • Build one when the same workflow needs the same setup and the same outcomes — and the process mixes deterministic and non-deterministic steps. A harness lets you “micromanage a little bit… be more prescriptive about how that job gets done.”
  • Worked build: a Sentry bug-triage harness. Core runs on the Claude Agent SDK; model is Claude Sonnet 4.6 (“the right model for the job”); interface is a TUI built with the Ink library. Vo stresses the surface is a choice, not a requirement — “your harness does not have to be a TUI… it could be a web app.”
  • Four load-bearing architecture choices (below) make it reusable: opinionated tool adapters, a filesystem artifact store, tool-policy flags, and custom prompts encoded as workflow steps.
  • The payoff reframes the work: instead of the open chat field where “if I just type in, the agent will do good work,” you hand an intelligent agent a constrained job and let a general-purpose agent orchestrate it. Constraining the work is how you “get specific jobs done really efficiently.”

When to Build a Harness

Build a harness when a workflow recurs with the same setup and the same required outcomes, and the steps are a mix of deterministic and non-deterministic work. This is close to when you’d build an agent — Vo notes “harness” and “agent” are sometimes interchangeable — but the trigger is a specific, slightly-complex job to be done that follows a standard process and uses specific tools.

Examples she names:

  • Coding (the reason coding harnesses are so common — coding is a job with specific tools and a standard workflow)
  • Managing production incidents
  • Getting PRs ready for release
  • Handling support escalations
  • Managing migrations
  • Non-technical: doing research in a very specific way, or consolidating docs in a very specific way

Why not just use a general coding tool directly? With a direct tool you re-explain intent every time (“Dear agent, please fix this bug, here’s the link”). With the harness she pastes in a Sentry link and “the agent already knows my intent, already knows the job to be done.” You also gain prescriptive tool control (an investigate-only harness that never writes code) and repeatable outcomes encoded once (every fixed bug documented in Linear, with a specific report, every time).

Architecture: The Four Load-Bearing Choices

These are the decisions that turn “a script” into a reusable harness. ^[inferred: grouping/labeling is a synthesis of Vo’s walkthrough; each item below is stated in the source.]

1. Opinionated tool adapters — not raw MCP. Rather than handing the agent a generic MCP and letting the “coding agent wander through all these traces,” Vo wrote adapters that call each external API in a job-specific way: “I’m just very precise about exactly what I think you need to pull from a bug report.” Separate scoped adapters for Sentry, Vercel, Linear, and GitHub — not how you can use these tools generally, but specifically how you’d use them when hunting a bug.

2. A filesystem artifact store. The harness saves the evidence from every run to the file system for the agent to reuse later (a pattern she notes appears in other harnesses like OpenClaw). Each run’s artifact bundle contains:

  • the task run (all the messages)
  • the reports (the Sentry issue + a brief on what was discovered)
  • any logs judged relevant
  • what the Claude worker ended up doing
  • a summary of the output
  • a generated HTML file rendering what happened, plus a worker report

3. Tool-policy flags. The harness carries explicit flags controlling what the agent may touch — edit source, modify inputs, or even message customers — and destructive/outbound actions happen “only if I flag and approve it.” Her demo runs in investigate mode, not fix: “the investigation should not touch and modify files.” This replaces prompting “I only want you to investigate, do not ship a fix” on every run.

4. Custom prompts encoded as workflow steps, not skills. Instead of a generic “you are Claude Code, make no mistakes” prompt, her prompt says the agent is working inside a company-specific engineering harness, that it is not an open-ended coding system, that the run artifacts are the source of truth, and exactly what to return (X, Y, Z). Crucially she encodes this as a specific step in the harness “so the model follows it every time” — versus a skill, which “you have to babysit” and hope gets invoked correctly.

The 7-Step Recipe

Vo’s own summary of how to build your own harness:

  1. Identify a specific workflow. Write it down — “proverbial paper, HTML, or markdown.”
  2. Define the run and its task input. A “run” executes a “task”; each task takes a specific input (for her, a Sentry issue).
  3. Make opinionated calls to tools/data sources. Build adapters that hit external APIs in a job-specific way rather than dropping in a generic MCP.
  4. Define the structured artifacts the workflow should output.
  5. Decide rules and permissions — which tools the harness may call and which it may not.
  6. Pick the runtime — Claude Code, Codex, or a model router; run most of it through an agent SDK (Claude’s or OpenAI’s).
  7. Build a surface to interact with the agent — a TUI, a CLI, or a web app.

Then plug the whole spec into Claude Code or Codex and have it build the harness, and test against real data.

Implementation

Tool/Service: Claude Agent SDK (harness core + agentic planning, using Claude Code primitives like reading and writing files); Claude Sonnet 4.6 (model); Ink (TUI library); Sentry, Vercel, Linear, GitHub (scoped tool adapters).

Setup: A small codebase — a high-level index that reaches the TUI, plus “about eight files” of specific capabilities: a bug-hunter workflow file, one adapter per external tool, an artifacts file that emits the run bundle, and entry files for the TUI and CLI. The repo ships instructions for where to place API keys. The harness exposes both a TUI (for a human to navigate runs) and CLI commands (to run quickly against a specific issue with specific tool-use flags).

Cost: data not available.

Integration notes:

  • The TUI reflects the harness structure — it shows past runs, errors, and the three-phase process: gather evidence → stream in activities → build artifacts.
  • Each investigation kicks off a Claude Agent SDK session that gathers evidence and forms a root-cause hypothesis.
  • Deeper point: the human-facing TUI is optional polish. The real value is “a structured way to give agents the job of running these investigations on a simpler basis” — you can hand a capable agent a constrained harness and let a general-purpose agent orchestrate it.

Worked demo output (Sentry error “edit document operations dropped,” run bug hunter C7) — the harness returned a structured investigation brief:

  • Confirmed evidence: a real Sentry warning (not an error), ~150 users impacted, still occurring hourly; Vercel logs were unavailable so that data was skipped.
  • Priority-ranked root causes: invalid original range, or overlapping original range; plus a flagged blind spot in the function.
  • Location: exactly where in the product surface the issue lives.
  • Verification step: fetch a raw Sentry event to confirm the hypotheses.
  • Recommendations: yes, open a Linear issue and assign it; no, do not turn on patch mode / attempt a fix yet — “I need a little bit more information.” (This priority-ranked, evidence-first, explicit-next-step shape is exactly the outcome she encoded into the harness step.)

Meta note — how she built it. Vo ran dueling Claude Code and Codex sessions (“help me build a harness… I want to use the Claude Agent SDK”). It was not a one-shot: both GPT-5.5 and Opus “really wanted to build something super deterministic” and resisted putting any AI in the harness, so she had to prompt very specifically for what she wanted. Her advice: be very specific about the workflow, the tools, and where custom prompts make sense, and run most of it through an agent SDK. Notably, “Codex did the best job of building the agent, but it used the Claude Agent SDK to actually implement it” — spanning models and coding agents.

Try It

  • Pick one recurring workflow from your week that always needs the same setup and produces the same outcomes (incident triage, PR-to-release, support escalation, a prescriptive research pass). Write the process down first.
  • List the tools it touches, then write opinionated adapters — decide exactly which fields each API call should pull for this job, instead of wiring a generic MCP.
  • Decide the artifact bundle: what evidence, reports, logs, and summary should every run persist to disk for reuse.
  • Set tool-policy flags up front — default to investigate-only; gate any file-write or outbound (customer-facing) action behind explicit approval.
  • Encode the job prompt as a workflow step, not a skill, so the model follows it every run rather than needing to be babysat.
  • Hand the full spec to Claude Code or Codex and have it build the harness on the Claude Agent SDK; test against real inputs. See Claude Agent SDK and Agent SDK — How the Agent Loop Works for the runtime it plugs into.

Open Questions

  • Vo’s harness runs on Claude Sonnet 4.6 for this workflow but mentions multi-model routing is possible; the source does not detail how routing would be wired inside the harness.
  • The artifact store is described as feeding “the agent to use in the future,” but the mechanism for how prior run bundles are retrieved and reused on later runs is not specified.
  • No cost figures (API spend per run, build time) are given in the source.
  • The harness is scoped to Vo’s own company codebase; the source does not cover how much of the adapter/prompt layer is reusable across different repos versus rewritten per project.

Graph View

Backlinks