The benchmark hallucination
Claude Opus 4.6 in Claude Code scored 58.0% on Terminal Bench 2.0. The same model in Meta-Harness, a Stanford IRIS harness, scored 76.4%. That is an 18-point gap.1
The headline is wrong. Claude Code is used by millions of developers. Meta-Harness has no users outside its research paper. The market and the benchmark disagree, and the market has the better claim.
What the benchmark measures
Terminal Bench 2.0 has 89 curated tasks with automated verifiers.1 It measures terminal agents on isolated problems. That is useful for comparing models, not harnesses.
A harness is the runtime around a model: tools, permissions, context management, streaming, safety. A benchmark asks whether the agent produced the correct output, not whether the user understood what was happening or whether the session felt productive.
The benchmark rewards completion. A harness that commits to an answer, even wrong, outscores one that asks for clarification on an ambiguous task. Claude Code favors asking. Users prefer it, but the benchmark cannot tell.
The hallucination
Seen at 58% and 76%, the story writes itself: Meta-Harness is better. The number looks precise and the ranking looks objective, so the conclusion feels like evidence.
It is not evidence but a hallucination: projecting meaning onto data that cannot support it. Extrapolating from a benchmark score to a better harness is like an LLM fabricating a citation from a plausible name.
The hallucination persists because a leaderboard is clean and user retention data is not. Admitting that harness quality resists reduction to a single number means accepting that evaluation is slow and qualitative.
Specialization is not overfitting
OpenAI models are trained on patch-based file editing. Anthropic models use string replacement. Either model can use either format, but the unfamiliar one costs extra reasoning tokens and produces more errors.2
Calling this overfitting gets it backward: training a model to master one method is resource-efficient specialization; a fine-tuning budget split across formats produces a worse model.
The same applies to tool names and system prompt conventions. A consistent interface trained deeply produces better results than agnosticism across five surfaces.
What evaluates a harness
Real-world usage is the only evaluation that matters. The process is slow and noisy, but the signals converge on truth.
Adoption and retention tell you whether developers choose the harness and whether they stay. A benchmark can tell you whether a new model is worth testing in your harness. It cannot tell you whether your harness is good.