Published on

Engineering the Agentic Harness

View .md

A practical field guide to the systems that wrap an LLM and turn it into a reliable agent — with patterns, worked examples, real deployments, and where the field is heading (mid-2026).


How to read this

This guide is organized so you can skim the decision aids first and drill into any layer you need.

  • Part I — Foundations: the vocabulary and the core "Agent = Model + Harness" idea.
  • Part II — Loop engineering: the run loop, the menu of agentic patterns, concurrency, durable execution, and loop control (retries/timeouts/gates).
  • Part III — Harness engineering: the stack, guides & sensors, context engineering, eval-driven development, and the newest techniques.
  • Part IV — Applications & use cases: what real production agents actually look like.
  • Part V — Observability & reliability.
  • Part VI — Recent trends + open problems.

Two decision aids to start

1. Do you even need an agent loop? Most teams reach for a loop too early.

Reality check (2026 production data): a survey of 306 practitioners across 26 domains found ~68% of production agents run 10 steps or fewer before a human steps in. The systems that work are small and supervised, not autonomous swarms. Agents trade latency and cost for capability — only make that trade when the task actually needs it.

2. The harness in one picture. Everything that is not the model is the harness.


Part I — Foundations

1. Three nested disciplines

These terms get conflated; they actually nest, each wider than the last.

DisciplineScopeQuestion it answersEra it dominated
Prompt engineeringOne inferenceWhat do I say right now?2022–2024
Context engineeringWhat's in the window over timeWhat should the model see — and not see?2025
Harness engineeringThe whole running systemWhat should the system enable, constrain, measure, and repair while the agent runs unsupervised?2026

Andrej Karpathy popularized the move from "prompt" to "context" engineering in mid-2025, arguing the former understates the work. Harness engineering (named in early 2026) goes one level wider: prompt and context shape what the model perceives; the harness also governs what it can do, what happens after it acts, and what the system does when things break over hours of operation. "Vibe coding" — prompt-and-pray with no surrounding system — is what you get when you skip the harness.

A useful frame from Martin Fowler / Birgitta Böckeler: the harness is a cybernetic governor, combining feedforward (steer before acting) and feedback (correct after acting) to regulate the system toward a desired state. The horse metaphor in the name is apt: a harness — reins, blinders, saddle — directs raw power toward useful work.

2. Agent = Model + Harness

A raw model takes text (plus images/audio/video) and emits text. Out of the box it cannot maintain durable state, execute code, access post-cutoff knowledge, or set up an environment. Those are all harness-level features. Even basic "chat" is a harness: a while loop tracking prior messages.

Where the term came from (early 2026): Mitchell Hashimoto (HashiCorp, Terraform, Ghostty) named it in a blog post: every time an agent makes a mistake, engineer a permanent fix into its environment so that mistake becomes structurally impossible. Within weeks, OpenAI published Harness engineering: leveraging Codex in an agent-first world, Martin Fowler / Birgitta Böckeler published their guides/sensors analysis, LangChain formalized the formula, and Anthropic had already shipped the foundational reliability work (Effective harnesses for long-running agents, Nov 2025).

The core cultural stance — internalize this one: treat every agent failure as an engineering problem to fix permanently, not a prompt to retry. First update the instruction file with a rule preventing the known failure; better, change the environment so it can't recur (a hook that forces the linter to run, not a doc line that asks the agent to). The difference is "almost every time" vs. "every time without exception."

Harness vs. framework vs. orchestrator vs. platform:

TermWhat it is
FrameworkProvides components (LangChain, LlamaIndex, Vercel AI SDK)
OrchestratorDecides the sequence of model calls
HarnessAssembles components into a running system + provides capabilities (tools, memory, context)
PlatformRuns many harnesses across many teams over time

Part II — Loop Engineering

3. The base run loop (ReAct)

The dominant pattern is ReAct: the model reasons, takes an action via a tool call, observes the result, repeats — inside a while loop the harness controls. Modern harnesses use native tool calling: the model returns structured tool_call objects, not free text to parse.

The same loop viewed as a message exchange between the parts of the system:

# The canonical loop — annotated with the control points that matter
def run(goal, tools, max_steps=12, budget):
    messages = [system_prompt(), user(goal)]
    for step in range(max_steps):              # <- hard iteration cap
        resp = model(messages, tools=tools)    # <- prompt-cache the prefix
        if not resp.tool_calls:
            return resp                         # <- natural completion
        results = execute(resp.tool_calls)      # <- retries/timeouts live here
        messages += results
        if budget.exceeded() or no_progress(messages):
            break                               # <- budget / no-progress gates
        messages = maybe_compact(messages)      # <- context management
    return finalize_or_escalate(messages)       # <- explicit fallback

The loop's per-turn question is binary: tool calls present → execute and loop; none → final answer. Everything interesting in loop engineering is the annotations above: where caps, gates, retries, and compaction attach.

4. The menu of agentic patterns (loop topologies)

This is the part most teams under-engineer. "The loop" isn't one shape — it's a menu, largely codified by Anthropic's Building Effective Agents (Dec 2024) and now the industry's default vocabulary. Real systems compose several.

The decision logic for picking one:

PatternShapeUse whenClassic failure modeReal example
Prompt chainingLinear pipeline; output → inputSteps decompose cleanly and are fixedBrittle to inputs that don't fit the fixed shapeDoc → outline → draft → polish
RoutingClassify, then dispatchDistinct input categories need different handlingMis-routing; needs a good classifierSupport triage: billing vs. bug vs. sales
ParallelizationFan-out / fan-in (sectioning or voting)Independent subtasks, or repeat-for-consensusAggregation logic, cost multipliesMap over 200 files; N-way vote on a risky call
Orchestrator–workersLead delegates dynamic subtasks, synthesizesSubtasks emerge from the input, not known aheadCoordination overhead; token blow-upCoding agent deciding which files to change; deep research
Evaluator–optimizerGenerate → critique → revise, loopOutput quality is the bottleneck and you can judge itAdds a "second optimist" if the check isn't objectiveCode that must pass tests; translation refinement
Reflection / ReflexionSingle agent critiques its own outputSelf-contained quality taskUncritically trusts its own critiqueSelf-debugging a failing function

Pseudocode sketches

# Routing
category = classify(input)                 # cheap model is fine here
return PIPELINES[category](input)

# Parallelization (sectioning + fan-in)
chunks = split(task)
results = await gather(*[worker(c) for c in chunks])  # concurrent
return aggregate(results)

# Orchestrator-workers (dynamic decomposition)
plan = orchestrator.plan(task)             # subtasks NOT hardcoded
outs  = await gather(*[worker(s) for s in plan.subtasks])
return orchestrator.synthesize(outs)

# Evaluator-optimizer (the quality loop)
draft = generator(task)
for _ in range(max_rounds):
    verdict = evaluator(draft, criteria)   # MUST be an objective gate
    if verdict.passed: break
    draft = generator(task, feedback=verdict.critique)
return draft

Anthropic's own guidance, worth repeating: if a task can be solved with prompt chaining or a single well-prompted call, do that. Reach for agentic loops only when the task genuinely requires adaptation.

5. The objective-gate principle (promote this to a rule)

An evaluator/reflection loop only works if the gate fails objectively — a failing test, a type error, a non-zero build, a schema-validation error. A second LLM told to "review this" with no hard signal is just a second optimist: it rubber-stamps, the loop "converges," and you ship a confident wrong answer while burning tokens.

This is the failure behind the "Ralph Wiggum loop," documented by Geoffrey Huntley: an agent meant to emit a completion token only when finished emits it early, and the loop exits on a half-done job. Without a hard gate, loops fail quietly and keep spending.

Worked example — code generation:

for attempt in range(6):
    code = generate(spec, feedback=last_error)
    result = run_tests(code)          # objective: pytest exit code + failures
    if result.passed:
        return code
    last_error = result.failures      # exact tracebacks, not "looks wrong"
raise Escalate("6 attempts, tests still red")  # hand to a human, don't loop forever

Maker–checker, generator–verifier, critic loops, reflection loops — all names for the same shape. The non-negotiable is: clear acceptance criteria + an iteration cap + a defined fallback when the cap is hit (escalate, or return best-effort with a warning).

6. Event loop vs. run loop, and parallelism

Two "loops" get confused:

  • The event loop is the runtime concurrency mechanism (e.g. Python asyncio) that lets the harness await many I/O-bound operations — model calls, tool HTTP requests, sandbox commands — concurrently. Infrastructure, not agent logic.
  • The agent run loop is the semantic reason→act→observe loop. It rides on top of the event loop.

The practical payoff is parallelism, an underused lever:

  • Parallel tool calls within one turn — when the model emits several independent tool calls, execute them concurrently rather than serially. Big latency win.
  • Parallel workers / subagents — fan out independent subtasks. Anthropic's multi-agent researcher reported isolated-context subagents outperforming a single agent on breadth tasks — at up to ~15× the tokens of a chat. Parallelism buys quality and speed; it costs money.

7. Graph-based & durable execution

A more structured alternative models the agent as a graph of discrete steps (e.g. LangGraph):

  • Nodes = units of work (call model; run tools; deterministic logic).
  • Edges = control flow, incl. conditional branches.
  • State = a typed object flowing through nodes, with reducers merging updates.
  • Checkpointing at step boundaries (a "superstep") enables resume-after-interruption and time-travel debugging.

Why this matters for long or high-stakes work:

  • Durable execution / resumability: a task running for hours that hits an unrecoverable error shouldn't restart from zero. Checkpoints let you resume from the last good step (the Temporal-style "durable workflow" idea applied to agents).
  • Human-in-the-loop as a first-class primitive: the graph can pause at an approval gate, surface the proposed action, and wait for a human yes/no before continuing — essential when actions are consequential and hard to reverse.

8. Long-horizon loops

When work exceeds one context window, two patterns dominate.

The Ralph loop (brute force). A hook intercepts the model's exit attempt and re-injects the original goal into a fresh, clean context window, forcing continuation against a completion target. Each iteration starts clean but reads accumulated state (progress files, git history) from the filesystem.

The structured initializer/coding-agent split (Anthropic). Detailed in Part IV's coding-agent case study — an initializer sets up the environment once; a coding agent makes incremental, one-feature-at-a-time progress every session, leaving a clean, mergeable state behind.

9. Loop control: retries, timeouts, gates, compensation

The happy path is easy; the error-handling boilerplate that survives production is often longer than the business logic. Four primitives, best owned by the engine (not hand-rolled in every node):

PrimitiveHandlesKey knobsGotcha
Stopping conditionsWhen to end the loopnatural completion, goal met, max-steps, budget, no-progress, human interruptEarly-stopping ("declares victory") and non-convergence
RetriesTransient failures (5xx, resets)exponential backoff, jitter, max_attempts, retry_onDon't retry bugs (ValueError/TypeError); guard non-idempotent side effects
TimeoutsHung callsrun_timeout (wall-clock), idle_timeout (no progress) + heartbeatsA streaming call isn't "hung"; use idle, not wall-clock, for it
Error handlersRetry exhaustionruns only after retries fail; failure context attachedMake the transition atomic; no handler-for-the-handler (recursion)
# Representative per-step policy (LangGraph style)
RetryPolicy(initial_interval=0.5, backoff_factor=2.0, max_interval=128,
            max_attempts=3, jitter=True,
            retry_on=(ConnectionError, TimeoutError))   # conservative
TimeoutPolicy(run_timeout=30, idle_timeout=5, refresh_on="auto")

Compensation (SAGA) for side-effecting workflows. If reserve_seat succeeded but process_payment failed, retrying the whole thing leaves a stuck reservation. The SAGA pattern: retry each step individually; track which steps actually completed; on exhaustion, run compensation that undoes only the completed steps in reverse order — keeping the operation all-or-nothing. This matters more every month as agents take high-consequence actions (booking, payments, internal API calls). A 1% transient failure rate is a demo nuisance; across dozens of consequential steps it compounds fast.

10. When not to loop

Loop engineering includes knowing when to stop looping:

  • If prompt chaining or a single call solves it, don't build an agent.
  • Cap autonomy: ~10 supervised steps beats 100 unsupervised ones for most current tasks.
  • Prefer an objective gate + human handoff over "let it run."
  • Background/async agents (Slack-task → PR) work precisely because there's a human review gate at the end.

Part III — Harness Engineering

11. The harness stack

A useful layering (synthesized from OpenClaw's public architecture, LangChain's anatomy, and MongoDB's component model):

12. Guides & sensors (the feedforward/feedback model)

Böckeler's mental model is the cleanest way to organize a harness:

Feedforward-only → encodes rules but never learns if they worked. Feedback-only → repeats the same mistakes. You need both, plus a steering loop: when an issue recurs, harden the guides/sensors so it becomes less probable — or impossible.

Keep quality left. Distribute checks by cost and speed: PostToolUse hook (ms) → pre-commit (s) → CI (min) → human review (hrs). The faster the layer, the more effectively the agent self-corrects. Separately, continuous drift sensors run outside the change lifecycle to catch gradual decay.

13. Infrastructure primitives

PrimitiveWhy it existsUnlocks
FilesystemModels only operate on the context windowWorkspace, context offload, durable cross-session state, multi-agent collaboration surface
GitVersioning on top of the filesystemRollback, branch experiments, fast onboarding for the next session
Bash + code execCan't pre-build a tool for everythingAgent designs its own tools on the fly ("give the model a computer")
SandboxLocal execution is unsafe and doesn't scaleIsolated, on-demand envs; allow-listed commands; network isolation; fan-out scale
Memory + searchKnowledge cutoff + finite weightsAGENTS.md injected at start (continual learning); web/docs search for fresh facts

14. Context engineering (write / select / compress / isolate)

Context is a scarce resource and the window is bounded. Two failure modes drive everything: context rot (reasoning degrades as the window fills) and hard overflow (the API would error).

The four strategies in practice:

  • Write — scratchpads, plan files, AGENTS.md. Anthropic's LeadResearcher saves its plan to memory so it survives truncation.
  • Select — hybrid semantic + keyword search; Skills via progressive disclosure (only front-matter loads at start, full instructions on demand).
  • Compresscompaction is the first lever. Claude Code triggers auto-compact near ~95% of the limit, summarizing history while preserving architecture decisions and open bugs. Tool-output offloading keeps head+tail in context, writes the full blob to disk.
  • Isolate — split across subagents (own tools, instructions, window). Anthropic reported isolated-context subagents outperforming a single agent on breadth tasks — at up to ~15× the tokens.

Live debate: Anthropic (multi-agent, isolated contexts) vs. Cognition/Devin (single agent + long-context compression for stability and lower cost). Both are arguing about the same problem.

15. Eval-driven harness development (NEW)

The most underweighted harness discipline. If you don't measure, every regression is a blind debug across prompt × retrieval × model × tools. Treat evals as a first-class harness component, not an afterthought:

  • Evals in CI — write assertions like unit tests; run them on every change to catch regressions before they ship. Agent-specific metrics that matter: task completion, tool-call correctness, argument correctness, step efficiency (libraries like DeepEval expose these; LangChain shipped Rubrics for agents that evaluate and correct their own work).
  • LLM-as-judge as a gate, not décor — useful, but only with explicit criteria; pair with computational checks so the gate can fail objectively.
  • Trace-based eval — score real production traces (sampled) for quality, not just synthetic test sets.
  • Synthetic edge-case generation — auto-generate the hard cases you can't easily collect.

The loop closes: evals feed the steering loop; failing evals become new guides/sensors.

16. Newer harness techniques (NEW)

TechniqueWhat it isWhy it matters
Prompt cachingCache the stable prompt prefix (system + tools + history) across turnsMajor cost & latency reduction in any multi-turn loop — often the single biggest cost lever
Code-execution tool orchestrationModel calls tools by writing code that invokes them, rather than many direct tool callsFewer round-trips, composable tool use, less context bloat from tool schemas
Just-in-time tool/context assemblyHarness loads the right tools + context per task instead of pre-wiring everythingBeats context rot from dozens of always-loaded tool definitions
Model routing / cascadesCheap model for easy turns; escalate to a strong model for hard onesCuts cost without tanking quality; routing is itself a pattern (§4)
Middleware / hooksDeterministic interceptors around model/tool calls (compaction, lint, guardrails, redaction)Where "engineer the failure away" rules physically attach
Skills & SOPsSKILL.md (and OpenClaw's SOUL.md) encode role, constraints, and standard operating proceduresStable, repeatable behavior; the more precise the SKILL.md, the more stable the output
MCP (Model Context Protocol)Open standard (Anthropic, Nov 2024) connecting an agent to tools/data over JSON-RPCThe vertical interop layer of the harness (see §17)
A2A (Agent2Agent)Open standard (Google → Linux Foundation) for agents to discover and delegate to each otherThe horizontal interop layer for multi-agent systems (see §17)

17. Interoperability protocols: MCP & A2A

As soon as you have many agents and many tools, you hit an N×M integration problem — every agent-to-tool and agent-to-agent pair wants a bespoke connector. Two open protocols collapse that into N+M along two different axes:

  • MCP (Model Context Protocol) — the vertical axis: how one agent connects down to tools and data.
  • A2A (Agent2Agent) — the horizontal axis: how agents talk across to each other as peers.

They're complementary, not competing.

MCP — agent ↔ tools/data

Open standard from Anthropic (Nov 2024), built on JSON-RPC 2.0 — often called "the USB-C of AI applications." Architecture: host → client → servers. An MCP Host (Claude Desktop, Cursor, ChatGPT, a custom agent) runs a Client that connects to one or more Servers, each exposing Tools (callable actions), Resources (readable data), and Prompts (reusable templates). Transports: STDIO (local process) and HTTP/SSE (remote). Mid-2026: thousands of public servers, 100M+ SDK downloads.

A2A — agent ↔ agent

Announced by Google (Apr 9, 2025) and donated to the Linux Foundation (June 2025, Apache-2.0, vendor-neutral governance under the Agentic AI Foundation). By April 2026, 150+ organizations back it (Microsoft, AWS, Salesforce, SAP, ServiceNow, IBM…), with native integration in Azure AI Foundry, Amazon Bedrock AgentCore, and Google Cloud. It uses a client–remote model: a client agent discovers a suitable remote agent, delegates a Task, and gets back Artifacts plus status updates — without either side exposing its internal logic, memory, or tools. Four core objects:

  • Agent Card — a public JSON doc (at /.well-known/agent-card.json) advertising skills, endpoint, and auth.
  • Task — a unit of work with a unique ID whose status updates over multiple rounds.
  • Message — a turn in the exchange (role: user or agent).
  • Artifact — the produced output.

A2A v1.0 (early 2026) made it production-grade: signed Agent Cards (cryptographic domain verification to stop card-forgery), multi-tenancy (one endpoint hosts many agents), multi-protocol bindings (JSON-RPC and gRPC), and version negotiation; v1.2 extends signed-card verification. The AP2 extension adds an agent-payments layer.

How they compose

In a real multi-agent system the two stack: an orchestrator uses A2A to delegate across specialist agents (possibly different clouds/vendors), and each agent uses MCP to reach its own tools and data — exactly the vertical+horizontal axes in the diagram above.

MCP vs. A2A at a glance

MCPA2A
AxisVertical: agent → tools/dataHorizontal: agent ↔ agent
OriginAnthropic, Nov 2024Google, Apr 2025 → Linux Foundation
TransportJSON-RPC over STDIO / HTTP-SSEJSON-RPC / gRPC over HTTP
Core objectsTools, Resources, PromptsAgent Card, Task, Message, Artifact
Discoveryserver config / registryAgent Card at a well-known URL
Answers"what can this agent use?""who can this agent delegate to?"

Security — the open wound (mid-2026)

Both protocols ship a permissive default trust model, and it's biting. The headline attack is tool poisoning — malicious instructions hidden in a tool's description/schema that the model reads but the user can't inspect (OWASP 2026 Agentic Top 10: ASI01 — Agent Goal Hijack). Related vectors: rug-pull updates, cross-server tool shadowing, STDIO RCE (OX Security's April 2026 disclosure showed 9 of 11 MCP registries were poisonable). Auth adoption is still thin (~8.5% using OAuth in one analysis).

The practical posture: treat every server as hostile until proven otherwise. Gateway → scope tokens (OAuth 2.1 + PKCE) → sandbox the runtime → sign Agent Cards → log and diff tool descriptions between sessions. In multi-agent chains, compromise propagates along delegation edges where per-agent sensors are blind — exactly why harness-level tracing matters.

18. Harnessability, templates, Ashby's Law

Not every codebase is equally harnessable. Strong typing gives you a type-checker as a free sensor; clear module boundaries afford architecture rules; opinionated frameworks abstract away whole classes of mistakes. Ned Letcher's term "ambient affordances" names this — structural properties that make an environment legible to agents. Greenfield can bake it in; legacy faces the cruel irony that the harness is hardest to build where it's needed most.

Ashby's Law of Requisite Variety: a regulator must have at least as much variety as the system it governs, and can only regulate what it has a model of. Committing to a known topology (a CRUD service, an event processor) is a deliberate variety-reduction move that makes a comprehensive harness achievable — hence emerging harness templates (reusable bundles of guides + sensors per topology).


Part IV — Applications & Use Cases

The patterns above aren't theoretical. Here's what production agents actually look like in mid-2026.

18. Coding agents (the proving ground)

Anthropic — the claude.ai clone experiment. Out of the box, even a frontier model in a loop fails to build a production web app: it one-shots (runs out of context mid-feature) or declares victory early. The harness fix:

  • Initializer agent (first session): writes init.sh, a progress log, an initial git commit, and a 200+ feature JSON list (each "passes": false). JSON chosen deliberately — the model is less likely to overwrite it than Markdown.
  • Coding agent (every later session): work one feature at a time → commit → update progress → leave a clean state.
  • Orientation ritual: pwd → read progress + git log → pick highest-priority unfinished feature → smoke-test before new work.
  • Test as a real user: browser automation (Puppeteer MCP) for e2e verification — huge accuracy gain over unit tests alone.

Production deployments at scale (early 2026 reports):

Org / toolWhatReported numbers
Stripe "Minions"Slack task → agent writes code → passes CI → opens PR for human review, no interaction between1,000+ merged PRs/week; pre-push hooks run relevant linters by heuristic ("shift feedback left")
OpenAI Codex teamBuilt a ~1M-line internal product, 3 engineers, ~5 monthsZero hand-written lines by design; ~3.5 PRs/engineer/day, rising with team size
OpenClaw (Peter Steinberger)Personal coding harness running Claude/GPT under the hoodCreator reports shipping code he doesn't read; 6,600+ commits in a month, 5–10 agents in parallel
LangChain coding agentChanged only the harness on TerminalBench 2.052.8% / outside top-30 → rank 5, same model weights
Codex Security (Mar 2026)App-security agent that finds and fixes vulnerabilitiesA harness specialized for a verification-heavy task

The common thread: the harness, not the model, is what made these production-grade. A widely-discussed MBZUAI finding (Apr 2026) noted four competing teams independently converging on nearly the same harness design — evidence that the harness is the durable engineering surface (and arguably the moat).

19. Deep research agents

Pattern: orchestrator–workers + context isolation. A lead agent plans the investigation and spawns subagents, each exploring a different facet in parallel with its own context window, then synthesizes. This is where isolation pays for its ~15× token cost — breadth tasks genuinely benefit from parallel, focused contexts. Guides: search tools, source-quality rules. Sensors: citation-validity checks, an evaluator pass on the synthesis.

20. Customer support & ops agents

Pattern: routing → tool use → HITL gate. A router classifies the ticket (billing / bug / sales) and dispatches to a specialized pipeline; the agent uses tools (order lookup, refund API); consequential actions (issue a refund) pass through a human-in-the-loop approval gate or a maker–checker loop with clear acceptance criteria. Stopping conditions and budgets keep cost bounded; SAGA-style compensation guards multi-step actions with side effects.

21. Personal agents

OpenClaw's rapid adoption (hundreds of thousands of users, connecting to email/calendar/messaging/files) proved users need far more than a capable model — they need reliable multi-step execution, safety boundaries, and knowledge that accumulates across sessions. Its harness layers: identity/constraints (SOUL.md), memory (including a "Dreaming" consolidation step modeled on sleep), an always-on agent loop, and SKILL.md SOPs. As agents take consequential actions on a user's behalf, the harness's safety and memory layers become the product.

22. The open-source harness ecosystem

Reusable harnesses: OpenHarness (HKUDS), goose (Linux Foundation), Superpowers (skills library), cross-agent memory like Supermemory and Cognee. The category has matured from "write a pile of bash" to "the pieces ship inside the tools."


Part V — Observability & Reliability

23. Tracing with OpenTelemetry GenAI

You can't operate what you can't see, and agent telemetry is harder than microservice telemetry: prompts are large blobs, tool params vary every call, and you care about tokens, cost, and quality, not just latency.

The de-facto standard is OpenTelemetry's GenAI Semantic Conventions (GenAI SIG, formed Apr 2024; still moving fast — every release in the v1.37–v1.41 range touched GenAI):

  • Standard gen_ai.* attributes — model, token counts, cost, provider. Portable across any OTel backend.
  • A span tree mirroring the agent's structure:
  • Content capture is three-mode. Default captures only metadata (model, tokens, durations) — no prompt content or tool args. Opt-in adds full prompts/results. Best practice: store content as span events (filterable/droppable at the Collector), not attributes (always indexed, PII risk); cap content length; redact at the Collector.

Tooling: intent-side observability — LangSmith, Langfuse, Datadog LLM Observability, and OTel-native backends — increasingly ingests standard GenAI spans directly. Complementary action-side / system-level observability (e.g. eBPF tools like AgentSight) watches syscalls to see what an agent actually did, closing the intent-vs-behavior gap (matters for debugging and security). Umbrella term: AgentOps.

Track: per-step + total latency; tokens & cost per call/run; tool success/silent-failure rates; loop metrics (iterations, retries, timeout hits); quality signals (eval scores, rubric pass rates, judge sampling); the full trace tree for replay.

24. Failure-mode catalog

FailureWhere it bitesPrimary mitigation
One-shotting (runs out mid-feature)Long-horizonFeature list + one-feature-at-a-time
Premature victoryMulti-sessionHard-to-overwrite spec/feature list (JSON)
"Done" without testingVerificationBrowser/e2e self-verification; objective gate
Context rotAny long runCompaction, offloading, Skills/progressive disclosure
Hard overflowAny long runCompaction; clean reset + handoff (Ralph loop)
Transient provider/network errorsProductionRetryPolicy: backoff + jitter, conservative retry_on
Hung tool / frozen subprocessProductionrun_timeout + idle_timeout (heartbeats)
Retry exhaustion on side-effectsHigh-consequence actionsError handler + SAGA compensation
Endless loop / no convergenceAny agentMax-steps, budget caps, no-progress detection
Quiet "Ralph Wiggum" exitReflection loopsObjective gate (test/build), not "review this"
Forgets across sessionsMulti-sessionFilesystem + git + progress files
Architectural drift / entropyLiving codebasesContinuous drift sensors, "garbage-collection" agents
Repeats the same mistakeAny agentSteering loop: add a guide/sensor or change the environment
Silent tool failures, opaque latencyProductionOTel tracing, tool success metrics

  • The harness is the moat. Same model, different harness → up to ~6× performance swings (Stanford/Tsinghua); LangChain's top-30→top-5 jump; a research project hit a 76.4% benchmark pass rate by having an LLM optimize the harness infrastructure itself, beating hand-designed systems. The MBZUAI "four teams, one harness" convergence reinforces that this is where durable engineering lives.
  • Model–harness co-evolution. Coding products (Claude Code, Codex) are now post-trained with their harness in the loop, so models get natively better at the actions designers expose (filesystem ops, bash, planning, subagents). Side effect: overfitting — changing tool logic can degrade a model — and the best harness for your task isn't necessarily the one a model trained with.
  • Self-improving harnesses. An active frontier: agents that analyze their own traces to identify and fix harness-level failure modes — observability feeding directly back into harness design.
  • Async / background agents. Slack-task → autonomous work → PR-ready-for-review is now a real production pattern (Stripe, others), built around a final human gate.
  • Capability is outrunning scaffolding — but slowly. Reports put the task length a model can reliably complete as doubling roughly every ~4 months; yet most production agents still run ≤10 supervised steps. The gap between demo autonomy and trustworthy autonomy is exactly the harness.
  • Harnesses are reshaping infrastructure. Because harnesses turn transactional API calls into long, stateful, tool-heavy sessions, they're changing how teams think about inference, caching, and even CPU/GPU provisioning (per industry reporting).
  • Standardization is arriving — on two axes. MCP (Anthropic) for agent↔tools and A2A (Google → Linux Foundation, 150+ orgs) for agent↔agent are converging into a default interop stack, alongside OTel GenAI conventions for telemetry and AGENTS.md/SKILL.md as de-facto instruction/skill formats. A2A reaching production v1.0/v1.2 with signed Agent Cards is the quiet-but-significant 2026 milestone.
  • Protocol security is the new frontier. As MCP and A2A wire agents into mission-critical systems, the permissive default trust model is biting: tool poisoning (OWASP ASI01), rug-pulls, STDIO RCE (OX Security, Apr 2026), and contagion along multi-agent delegation edges. Expect gateways, signed cards, OAuth 2.1, and "treat every server as hostile" to become standard harness hygiene.
  • JIT and dynamic harnesses. Moving from pre-wired tools/context toward harnesses that assemble the right tools and context just-in-time per task.

Open problems

  • Behaviour harnesses. Sensing whether an app functionally does what the user wants is still largely faith in AI-generated tests. No general answer yet.
  • Harness coherence at scale. As guides/sensors multiply, keeping them in sync and non-contradictory is unsolved — we lack a "code coverage for harnesses." (If sensors never fire, is that quality or blindness?)
  • Single vs. multi-agent. Open question whether one general agent or a team of specialists wins long tasks, traded against multi-agent token cost.
  • Reflection's attack surface. Reflection loops uncritically trust the feedback they're given — a real safety concern as agents act in the world.
  • Orchestrating hundreds of agents on a shared codebase without collisions.

The through-line: the model contains the intelligence; the harness is the system that makes that intelligence useful, reliable, and safe. Context engineering decides what the model sees; loop engineering decides how it acts and when it stops; harness engineering is the whole governing system that wraps both — and treats every failure as something to engineer away for good.


Reference: a pragmatic build order

Start with the instruction file and grow. You don't need all of this at once.

  1. Instruction file (AGENTS.md/CLAUDE.md): structure, build/test commands, conventions, growing anti-pattern list (one rule per repeated mistake).
  2. Run loop — ReAct with native tool calling; a graph engine if you need branching, checkpointing, or HITL.
  3. Tools + a general escape hatch (bash/code exec) in a sandbox with command/network guardrails.
  4. Filesystem + git for durable state, rollback, cross-session handoff.
  5. Context strategy — compaction near the limit, tool-output offloading, Skills, subagents where they pay off.
  6. Loop control — explicit stop conditions, per-step retries (backoff + jitter, conservative predicate), run + idle timeouts, error handlers, SAGA for side effects.
  7. Sensors — fast computational checks shifted left, inferential review where semantic judgment is needed, continuous drift sensors.
  8. Eval harness — agent metrics (completion, tool correctness, step efficiency) in CI; rubrics/judge as objective gates.
  9. Long-horizon scaffolding — initializer vs. coding-agent split, structured feature list, progress file, init.sh, orientation ritual, e2e self-verification.
  10. Observability — OTel GenAI instrumentation (standard attributes, content as governed span events), token/cost/latency/quality metrics, full trace trees.
  11. Cost levers — prompt caching, model routing/cascades, JIT tool loading.
  12. The steering loop — review failures; harden guides/sensors; where possible, change the environment so the failure is structurally impossible.

Sources

Primary and notable sources (2024–2026):

  • Mitchell Hashimoto — My AI Adoption Journey (origin of "engineer the harness"), mitchellh.com
  • OpenAI (Ryan Lopopolo) — Harness engineering: leveraging Codex in an agent-first world, openai.com
  • Anthropic — Building Effective Agents (the workflow-pattern taxonomy), anthropic.com/engineering
  • Anthropic — Effective harnesses for long-running agents (Nov 2025); Effective context engineering for AI agents, anthropic.com/engineering
  • Martin Fowler / Birgitta Böckeler — Harness engineering for coding agent users, martinfowler.com
  • LangChain (Vivek Trivedy) — The Anatomy of an Agent Harness; (Long/Runkle) — Fault Tolerance in LangGraph; Context Engineering for Agents; Rubrics for deepagents — langchain.com/blog
  • MongoDB — The Agent Harness: Why the LLM Is the Smallest Part of Your Agent System
  • AlphaSignal — Most Developers Do Not Need Agent Loops Yet (production-step survey; Ralph Wiggum loop)
  • The Pragmatic Engineer / ignorance.ai — The Emerging Harness Engineering Playbook (Stripe Minions, OpenClaw, Codex numbers)
  • Microsoft Learn — AI Agent Orchestration Patterns (maker-checker / evaluator-optimizer)
  • OpenTelemetry — GenAI Observability with OpenTelemetry + GenAI Semantic Conventions (GenAI SIG); Greptime, Uptrace, Datadog write-ups
  • arXiv — SemaClaw (2604.11548), AgentSight (2508.02736), Toward a Safe Internet of Agents (2512.00520)
  • A2A: Google Cloud — Announcing the Agent2Agent Protocol + v0.3/v1.0 release notes; Linux Foundation — A2A Protocol Project launch; Agentic Web (arXiv 2507.21206) for the Agent Card / Task / Message / Artifact model
  • MCP & protocol security: OX Security STDIO disclosure (Apr 2026); Cloud Security Alliance Agentic MCP Security Best Practices; OWASP Top 10 for Agentic Applications (2026, ASI01); SentinelOne / General Analysis MCP threat-model guides
  • Sitepoint / Augment Code — 2026 agentic design-pattern catalogs; DEV — Open Source Toolkit for Building AI Agents in 2026
  • The Register — How AI agent harnesses like OpenClaw are changing LLMs, inference, and CPUs

Caveat: this is a fast-moving area. Terminology — especially multi-agent OTel conventions, code-execution tool calling, and specific framework APIs — is still consolidating; verify exact attribute names and signatures against current docs before building.