Drift That Never Flips a Decision
Decision-level review clears a release when every recommendation holds, even as the score distribution underneath has moved - and that gap compounds across thousands of decisions.
The most expensive drift in a deployed agent is the kind that never changes an answer. Edit one prompt module that has nothing to do with the task in front of it, and the scores the agent produces shift systematically while every recommendation lands exactly where it landed yesterday. The review that gates the release looks at recommendations, finds them identical, and reports all clear. The shift keeps accumulating across every hiring screen, credit decision, and triage call that agent handles for the rest of the quarter.
Why Decision-Level Review Reports All Clear
The teams running those checks are not being careless. Decision-level review is the correct instrument for the failure everyone expects, an agent that returns the wrong recommendation, and it catches that failure reliably. What it cannot see is a score distribution that has moved underneath a set of decisions that did not. Sub-threshold drift is invisible to a quality process built around decisions rather than score distributions, it compounds across the thousands of decisions a deployed agent makes, and it propagates into downstream ranking, prioritization, and aggregation systems where the magnitude matters whether or not any individual recommendation flips. The blind spot is structural rather than procedural. Adding more decision-level cases to the suite does not close it, because the suite is measuring the wrong quantity.
Where the Interference Comes From
The measurement holds the scored task fixed and varies three things about the surrounding modules:
- Volume: how much text was added to the shared context.
- Meaning: what that added text actually said.
- Format: how the same content was wrapped and laid out.
One controlled measurement on a deployed job-evaluation agent put numbers on this. The design held the scored task constant and perturbed only the modules that had nothing to do with it, along three separate channels: how much text was added, what that text meant, and how it was formatted. Adding a single irrelevant archetype to the shared rules file produced an effect size of d = 0.63, a paired shift of +0.17 with a bootstrap 95 percent confidence interval of [+0.03, +0.31] that excludes zero, and eight of the twelve job descriptions moving upward. The volume and format channels yielded confidence intervals that include zero, which rules out both the generic reading that any added context degrades performance and the reading that formatting alone explains the result.
Those two controls are what make the measurement useful rather than merely suggestive. The effect localizes to semantic content sitting in the same context window as the scored task, which puts the cause in the composition mechanism rather than in the model's capability. Self-attention computes pairwise interactions across the entire composed context, so a `### Persona` heading is a writing convention, not a namespace boundary. Delimiters and section titles are statistical hints the model has learned to honor most of the time. Nothing in the runtime enforces them, and nothing in the deployment surfaces the moment they stop being honored.
The blind spot is structural rather than procedural.
Prompt Modules Are Software Components
The operator-facing consequence is concrete. When scoring rubrics, workflow sequencing, and tool-selection criteria live in human-authored text that a model interprets, those files carry behavioral logic without any of the isolation guarantees a compiler or a module system would provide. They earn the same treatment code gets: a versioned behavioral suite, and a gate that re-runs it whenever anything in the composed context changes, including modules nobody thinks are related to the one under test.
Four categories of regression testing cover this ground:
- Compositional consistency: does the assembled context still behave the way its parts did alone?
- Module-interaction regression: does editing one module move behavior somewhere else?
- Format-perturbation robustness: does rewrapping the same content move the scores?
- Model-migration testing: does the suite still hold when the model behind it changes?
That is not where the widely deployed systems built on this pattern currently sit. None of the systems surveyed implements any form of regression test along any of the relevant categories, which are compositional consistency, module-interaction regression, format-perturbation robustness, and model-migration testing. Exhaustive coverage is not the alternative either. A public registry of more than ten thousand community-authored skill modules yields over fifty million pairwise combinations before anyone considers higher-order interactions, which makes full pairwise testing prohibitive. Sampled, risk-weighted checking against the combinations that actually ship is the practical posture, and it only works if the thing being sampled is the distribution rather than the verdict.
The Gate That Blocks on Score Movement
This is the class of failure our regression gate is built to catch. It runs on every change and blocks a merge that drops mean evaluation scores or introduces a new hard failure, with an adversarial floor underneath both, measured against frozen and integrity-checked evaluation datasets and scored by a judge that is itself bias-controlled and classification-routed. The unit it gates on is the score distribution, which is precisely the quantity a decision-level check cannot resolve.
Two limits sit on that mechanism, and both belong in the open.
Where the Gate Stands Today
The mechanism is built and running; the quality curve it will eventually draw is still ahead of it, and saying so plainly is part of the posture.
Scope the evidence honestly. That measurement covers one model and one deployed agent, so the magnitude is a proof of the protocol rather than a constant anyone can plan around, and how far the effect travels across model families remains open. The broader lesson does not depend on the number. If behavioral logic lives in text that shares a context window, module isolation is an expectation rather than a guarantee, and the only honest way to know whether an edit stayed inside its module is to measure the distribution before and after. Decisions are what the business acts on. Distributions are what tell you whether those decisions are still being produced the same way.
More from Insights
Which Memory Layer Are You Buying
Two systems both called memory can owe entirely different obligations: one holds state for a single run, the other has to be viewable, removable, tenant-isolated, and auditable years later.
Agent Failure Modes Are Architecture Decisions
Error cascades and undocumented decisions end regulated pilots, and both are settled at the first commit: what a buyer can check in the architecture long before an agent runs.
The Judge That Gates Your Agents
A judge with unmeasured preferences turns a regression gate into selection pressure, and swapping positions plus majority voting cost far less than three retrains of drift.
Orchestration that earns autonomy.
Measure the Distribution Before You Ship
Remember what worked. Apply it faster. Prove what you did.