DEV FIELDNOTES
AI engineering field guideUpdated November 21, 2026

Harness Engineering for AI Agents That Can Verify Their Work

Build an AI agent harness with task contracts, scoped tools, recovery, evals, and a tested TypeScript gate that rejects stale or incomplete verification.

An AI processor in a tool frame connected to task records, checkpoints, and a gate that accepts current evidence and holds stale evidence

An AI coding agent changes a Next.js route, runs a test, and reports that the bug is fixed. Then someone finds that the test ran before the final edit. The implementation might be correct, but the evidence belongs to a different version of the code. A confident completion message leaves the team with another verification task.

Harness engineering addresses this kind of failure by designing the software and working environment around the model. The harness supplies context, executes tools, records progress, applies permissions, and decides which evidence allows a task to advance. For an AI engineering team, those responsibilities deserve the same review and testing as application code.

This guide uses a coding agent as the running example. We will define its task, constrain its tools, preserve progress across sessions, and build a TypeScript completion check that rejects stale or incomplete verification. The code runs locally without a model API key. It tests the completion decision, while your existing agent and CI system perform the actual coding and verification.

What is harness engineering?

An agent harness is the runtime that connects a model to tools and manages its execution. Harness engineering also reaches into the environment the agent works in, including repository documentation, repeatable setup, feedback, and checks. Teams use the term at different scopes, so describe the responsibilities you own whenever you discuss a harness.

OpenAI's engineering account describes making repository knowledge accessible to agents and enforcing architectural constraints through tooling. That is a useful example of the environment becoming part of the engineering work. Our example applies that idea to one smaller responsibility, deciding when a patch has enough verification to enter review. OpenAI's harness engineering account

Prompt engineering shapes the instructions supplied to the model. Context engineering selects and maintains the information available during a run. A harness puts those choices into an executable workflow with tool dispatch, persistent state, verification, and stopping conditions. Anthropic's context engineering guide explains why keeping context focused matters during longer tasks. Effective context engineering for AI agents

For an existing coding agent, much of the runtime already exists. Start by configuring its environment and measuring failures. A custom loop becomes worthwhile when your workflow requires controls or integrations the existing runtime cannot express.

Start with one observable task

Suppose a Next.js search endpoint returns an internal error when the search parameter is missing. The desired behavior is an HTTP 400 response with a stable error shape. A valid search should continue to return results.

Write the acceptance conditions before the agent edits the route. Include the exact endpoint, the failure to reproduce, and the checks that will prove the repair. Choose checks against the behavior callers depend on, including the existing successful path.

task.json
{
  "id": "search-missing-query",
  "goal": "Return HTTP 400 when GET /api/search has no q parameter",
  "acceptance": [
    "Missing q returns HTTP 400 with code MISSING_QUERY",
    "Whitespace-only q returns the same error",
    "A valid q still returns the existing successful response"
  ],
  "requiredChecks": ["typecheck", "search-route-contract"],
  "maxRepairAttempts": 3
}

This task file is an example application contract. Your controller must read and enforce the fields it uses. Writing maxRepairAttempts into a file only influences execution when the controller counts attempts and stops scheduling repairs at the limit.

Keep the accepted task and check definitions under the controller's control. An agent may propose a better test or explain that a requirement is ambiguous. Changes to the acceptance contract should be reviewed before the controller uses them to grade a candidate.

Give the agent a short route through the repository

The first context package should identify the route, its caller, nearby tests, and the commands that work in a fresh checkout. Load additional files when the investigation needs them. A run that starts with the entire repository can spend its context budget on information unrelated to the failure.

Use AGENTS.md to describe the repository's conventions and point to detailed documentation. In this search example, the instructions could name the API error helper, the integration-test location, and how the test database is prepared. Explain where the agent should look when a check fails.

Our AGENTS.md guide covers writing and maintaining those repository instructions. Keep the operational controls in the runner, where they can be enforced when a tool is called. AGENTS.md for AI coding agents

Make setup reproducible. Pin the runtime and dependencies, create a disposable test database if needed, and record failures that exist before the patch. When an agent cannot install dependencies or reach its test service, report that environment failure explicitly so the next attempt can recover from it.

Treat tool execution as an application boundary

A useful tool registry exposes operations the workflow needs, such as reading a repository file, applying a patch, running a named check, and inspecting its output. Resolve file paths inside the assigned workspace. Map a check identifier to a trusted command definition before starting the process.

Named commands alone provide limited isolation because a command can execute package scripts and application code from the candidate. Run candidate code in a disposable environment with scoped credentials and controlled network access. Keep production credentials outside the coding workspace. The executor must enforce these boundaries independently of text in the prompt.

Record a tool call's identifier, validated arguments, start and finish times, and outcome. A process that times out, receives a signal, or never starts needs an explicit result. Preserve bounded logs so the agent can inspect a failure without filling its next context window with the entire test output.

For an agent consuming remote tools, the same checks apply at the MCP server boundary. Verify the caller and permitted action before execution, and preserve the operation's identity across retries. Our remote MCP guide develops that integration in more detail. Remote MCP servers with TypeScript and Vercel

Make completion depend on current evidence

After the coding step, the controller snapshots the candidate into an immutable commit. A trusted verifier checks out that commit in a fresh environment, runs the required checks, and stores the results outside the agent's writable workspace. If the patch changes, produce a new candidate and verify it again.

The following module decides whether a verification record permits review. Copy it into harness-gate.mts. The controller supplies the expected attempt, candidate commit, and required check identifiers. These inputs and the verification record must come from trusted controller and verifier storage after runtime schema validation.

harness-gate.mts
export type CheckResult = {
  id: string
  exitCode: number | null
  timedOut: boolean
}

export type Verification = {
  attemptId: string
  candidateSha: string
  checks: CheckResult[]
}

export type ExpectedVerification = {
  attemptId: string
  candidateSha: string
  requiredChecks: string[]
}

export function completionState(
  expected: ExpectedVerification,
  evidence: Verification,
): 'needs_verification' | 'needs_changes' | 'ready_for_review' {
  const required = new Set(expected.requiredChecks)
  const received = new Set(evidence.checks.map(check => check.id))

  if (
    !expected.attemptId ||
    !expected.candidateSha ||
    required.size === 0 ||
    required.size !== expected.requiredChecks.length ||
    evidence.attemptId !== expected.attemptId ||
    evidence.candidateSha !== expected.candidateSha ||
    received.size !== evidence.checks.length ||
    received.size !== required.size ||
    !expected.requiredChecks.every(id => received.has(id))
  ) {
    return 'needs_verification'
  }

  if (evidence.checks.some(check => check.timedOut || check.exitCode === null)) {
    return 'needs_verification'
  }

  if (evidence.checks.some(check => check.exitCode !== 0)) {
    return 'needs_changes'
  }

  return 'ready_for_review'
}

A missing check, duplicate result, different candidate, or incomplete process leaves the task waiting for verification. A completed check with a failing exit code sends it back for changes. Passing every required check advances the candidate to review.

The controller should also record which verifier policy and environment produced the evidence. Version the required-check configuration when you change it, and create a fresh attempt. This example compares attempt and commit identity. It relies on the verifier to ensure that each reported check used the approved command, fixtures, and environment for that attempt.

An exit code of zero only establishes what the check actually tested. Configure the contract check to fail when it discovers zero tests, and ensure its assertions exercise the endpoint's status and response body. The gate cannot discover a missing assertion from the process exit code.

Test the completion failures before connecting a model

Copy the following test beside the module. It exercises the state transitions with deterministic evidence, including a timeout that arrives with a zero exit code and results reused from an earlier attempt.

harness-gate.test.mts
import assert from 'node:assert/strict'
import test from 'node:test'
import {completionState} from './harness-gate.mts'
import type {ExpectedVerification, Verification} from './harness-gate.mts'

const expected: ExpectedVerification = {
  attemptId: 'attempt-2',
  candidateSha: 'a'.repeat(40),
  requiredChecks: ['typecheck', 'search-route-contract'],
}

const passed: Verification = {
  attemptId: expected.attemptId,
  candidateSha: expected.candidateSha,
  checks: expected.requiredChecks.map(id => ({
    id, exitCode: 0, timedOut: false,
  })),
}

test('current passing evidence permits review', () => {
  assert.equal(completionState(expected, passed), 'ready_for_review')
})

const incomplete: Array<[string, Verification]> = [
  ['missing check', {...passed, checks: passed.checks.slice(0, 1)}],
  ['duplicate check', {...passed, checks: [passed.checks[0], passed.checks[0]]}],
  ['wrong check', {...passed, checks: [passed.checks[0], {
    id: 'unrelated-test', exitCode: 0, timedOut: false,
  }]}],
  ['older commit', {...passed, candidateSha: 'b'.repeat(40)}],
  ['older attempt', {...passed, attemptId: 'attempt-1'}],
  ['timeout', {...passed, checks: [passed.checks[0], {
    ...passed.checks[1], timedOut: true,
  }]}],
  ['no exit code', {...passed, checks: [passed.checks[0], {
    ...passed.checks[1], exitCode: null,
  }]}],
]

for (const [name, evidence] of incomplete) {
  test(name + ' requires verification', () => {
    assert.equal(completionState(expected, evidence), 'needs_verification')
  })
}

test('a failed contract check requires changes', () => {
  const failed = {
    ...passed,
    checks: [passed.checks[0], {...passed.checks[1], exitCode: 1}],
  }
  assert.equal(completionState(expected, failed), 'needs_changes')
})

test('an empty required-check policy cannot pass', () => {
  assert.equal(completionState(
    {...expected, requiredChecks: []}, {...passed, checks: []},
  ), 'needs_verification')
})

test('a duplicate required-check policy cannot pass', () => {
  assert.equal(completionState(
    {...expected, requiredChecks: ['typecheck', 'typecheck']},
    {...passed, checks: [passed.checks[0]]},
  ), 'needs_verification')
})

Run the example with Node.js 24. The .mts extension selects an ES module, and Node's built-in TypeScript support handles the erasable type syntax used here. Node executes this code without type-checking it, so a project should also run its TypeScript compiler. Node.js TypeScript support

Run the gate tests
node --test harness-gate.test.mts

There are 11 checks in this example. They verify the completion policy independently of model behavior. When you integrate the gate, add an integration check proving that the real verifier records the candidate it executed and that an agent cannot replace the stored evidence.

Preserve enough state to resume accurately

Store the task identity, current candidate, attempt number, last verified result, and next action between sessions. Keep a concise progress note explaining the investigation and remaining uncertainty. On resume, reload authoritative state and confirm that the referenced candidate still exists.

Anthropic's long-running-agent work uses initial environment setup, incremental feature work, and progress artifacts to carry work across sessions. The applicable lesson is to give the next session enough information to recover its place and check the current state. Effective harnesses for long-running agents

When verification infrastructure fails, retry that stage within a bounded policy. When a contract check identifies a behavior failure, return the relevant output to the coding step. A controller that sends both failures through the same repair loop can make the agent edit healthy code while its database remains unavailable.

For tools with external side effects, persist a business operation identifier before dispatch and reconcile an uncertain outcome before retrying. A fresh context window must preserve pending approvals and outstanding operations. Our production AI SDK guide shows these concerns in a workflow that prepares and sends a support reply. Production agents with Vercel AI SDK 7

Evaluate the model and harness together

Once the deterministic controls work, run the full agent on a small set of representative tasks. Include missing input, an existing failing check, unavailable infrastructure, an attempted tool action outside scope, and a task that needs clarification. Preserve the expected outcome for each case.

Record task success, time to a verified result, model usage, tool failures, repair attempts, and human interventions. Repeat tasks because a model can take different paths on different runs. Inspect the final application state alongside the execution trace. Anthropic's evaluation guide describes evaluating the model and its harness together and using multiple trials to understand variation. Demystifying evals for AI agents

Keep the model version and task set fixed when comparing a harness change. For example, compare the existing workflow with a version that gives the agent a focused route-contract test and current verification evidence. Check whether completed tasks improve and whether the cost per successful task changes. Keep a held-out group of tasks so repeated tuning does not simply optimize the examples you inspect every day.

For subjective work such as interface design, write a rubric that names the properties reviewers care about and calibrate it against human judgments. Anthropic's application-development experiments describe giving an evaluator explicit criteria and a separate feedback role. An evaluator model still needs calibration for the task you assign it. Harness design for long-running application development

Turn recurring failures into a specific improvement

If an agent repeatedly misses an error helper, improve the repository map and measure whether that reduces the mistake. If it claims success after a stale test, bind verification to the candidate. If it loops on an unavailable service, classify the environment failure and make recovery explicit.

For the Next.js search task, completion means a known candidate passed the agreed route contract and typecheck, with the evidence available to its reviewer. Start with that observable result. Expand the harness when the next recorded failure reveals a missing capability.

OpenAI — Harness engineeringAnthropic — Effective context engineeringAnthropic — Effective harnesses for long-running agentsAnthropic — Demystifying evals for AI agentsAnthropic — Harness design for long-running applicationsNode.js — TypeScript supportNode.js — Test runner