The Judge That Gates Your Agents

The instrument that decides which version of your agent ships carries preferences of its own, and in most pipelines nobody has measured them. It favors whichever candidate occupies a particular slot in the prompt, and asking it to explain itself, the first move most engineers reach for, makes that worse. Position consistency is the metric for position bias, and it quantifies how frequently a judge selects the same response after swapping the position of the candidate responses. GPT-3.5-turbo's position consistency sat at 68.78 percent. Asking the same model to supply an explanation alongside each verdict dropped its position consistency to 48.97 percent. Retrain on verdicts from a judge like that and the gate carries its preferences forward into every generation of agent behind it.

01The Judge Has an Error Profile

That behavior belongs to the design of model-graded evaluation, and it is documented well enough to build around. One recent review of automated evaluators pulled the known failure modes together and then ran its own meta-evaluation of the fixes teams reach for first. What comes out of it is a set of preferences that are named, stable, and measurable on a fixed dataset, which turns a general unease about trusting model output into an engineering problem with an engineering answer. Most teams running a learning loop already do the hard parts well, with frozen datasets and an outcome metric someone owns.

The evaluator is the part of that discipline nobody profiles. Ordering is the most familiar of its preferences and the easiest to correct. Concreteness bias is quieter: evaluators favor responses with specific details, including citation of authoritative sources and numerical values. An agent that learns to decorate an answer with figures and references climbs the score without changing what it does, and it learns that faster than it learns the task, because the gate rewards it sooner. Evaluators also prefer responses generated by themselves, which matters the moment the judge and the agent share a lineage. Compassion-fade bias describes the effect of the model names, so a version label left visible in a comparison hands the evaluator a shortcut with no bearing on the answer under review.

02What Held Up Under Measurement

The instinct when an evaluator looks unreliable is to make it think harder: ask for reasons, have it check its own verdict, run it several times and average the scores. Measured against human-labeled preferences, those instincts do poorly. Explanations help a human review a decision after the fact, but in terms of bias mitigation, they generally have a negative impact on both models tested. Self-validation, where the judge is asked to confirm its own result, shows minimal effectiveness, most likely due to the model's overconfidence. Taking the mean score across rounds or the best score across rounds did not improve the evaluation performance and even had some adverse effects, since both carry the skewed runs into the final number.

Two mechanical corrections came out of the same measurement intact, and their coverage has an edge worth reading at a glance.

A gate built on those two should carry that split in its own documentation, and the behavior inside a specific pipeline remains a separate measurement.

03Why a Learning Loop Compounds It

A single skewed verdict is a bad call on one comparison. A loop turns it into selection pressure. Using an evaluator for pseudo-labeling risks amplifying existing model biases, potentially leading to the generation of unverified and flawed training data, and a regression gate that decides which agent version ships is that same mechanism with a shorter feedback path. Whatever the judge rewards, the next generation of agent supplies more of, and the generation after that is scored by the same uncalibrated evaluator that shaped it. A learning loop with an unmeasured judge does not gate drift. It ratifies the judge's taste and reports the result as improvement.

The failure is quiet, which is what makes it expensive. Nothing errors. The gate passes, the merge lands, the score moves in the right direction, and the drift the gate was built to catch is sitting inside the training signal. Self-preference tightens the circle further: when the judge and the agent descend from the same base model, the evaluator's preference for its own output is already pointed at the thing it is scoring, and the loop reads the resulting agreement as quality. Curves drawn under those conditions still go up and to the right, and what they track is agreement with an evaluator nobody has calibrated, which is a hard thing to explain in a regulated deployment when someone asks how the improvement was established.

It ratifies the judge's taste and reports the result as improvement.

04What We Build Against It

The evaluator, then, gets specified the way any other load-bearing component gets specified.

QoherenceAI treats the evaluator as infrastructure with a specification. The regression gate runs on every change and blocks a merge that drops mean evaluation scores, introduces a new hard failure, or falls below an adversarial floor, and the evaluation datasets behind it are frozen and integrity-checked, so a moving dataset cannot be mistaken for a moving agent. The judge itself is bias-controlled and classification-routed: bias control is what the harness enforces against the failure profile above, and classification routing means the judge scoring a restricted-data evaluation is cleared for that data, so meta-evaluation does not become a side channel around the classification gate. The standard the gate is built to meet is that its verdict carries an agreement figure against human labels, so the claim that quality held is a number someone outside the team can check.

The Scope of the Claim Today

The scope of that claim matters as much as the mechanism. The gate is enforced in continuous integration today against a reference agent. The production agents and their baselines arrive as the first flows are wired, and the receipts that would show quality compounding across retrains come after that, once agents are running. We describe the mechanism because it is built and tested, and we will report the curve once agents are running against it.

Where the Authority Comes From

The authority of a learning loop rests on the measured error profile of the instrument that gates it. Order-swapped comparisons and agreement measured against human labels cost very little next to the price of discovering, three retrains in, that the curve was tracking the evaluator's taste. An evaluator is an instrument with a known error budget, not an authority taken on faith.