The Context Window Is a Governed Boundary

If your privacy controls sit on the model's response, you are guarding the wrong door. Any runtime that hands real records to a model at inference, whether as retrieved context or as labelled in-context examples, has made those records the exposed asset, and the signal that identifies them never reaches the output. It lives inside the forward pass, in where the model concentrates its attention. The output filter reports clean for the whole duration of the leak.

01Guarding the Wrong Door

The gate most teams have already built sits one layer too early to catch this. Classification routed at the model router answers a real and load-bearing question: which class of data may be sent to which provider, checked before the call and re-applied on every failover hop, so a sensitive request cannot be quietly downgraded during an outage. We build that gate to fail closed. What it cannot answer is what happens once a request clears it, which records become eligible to enter the context window, and what the model's use of those records exposes to anyone holding a query interface.

The output filter reports clean for the whole duration of the leak.

The privacy defenses that grew up around classical membership inference all live at the output boundary. They cap the prediction vector, smooth confidence, or blunt the certainty a model expresses on records it has already seen. Those controls aim at a genuine signal, and against confidence-based attacks they work. Methods that restrict the prediction vector to the top classes, or enforce less confident predictions, are effective against classical confidence-based attacks and do not modify the attention patterns an attention-based attack reads. So a system can pass every output-side privacy check it has and still identify, row by row, what was sitting in its context.

02What Attention Concentration Reveals

The setup of the study that measured this is worth having at a glance before the results.

The aggregate numbers are worse than a marginal improvement. On the architectures where attention concentrated most sharply, the attention-based attack reached near-perfect discrimination measured by AUC. The low-false-positive behavior is the part worth sitting with, because that is the regime an adversary who intends to act actually works in. At a zero false positive rate the confidence-based attack collapses to near zero across all models, unable to name a single member without also flagging non-members. In that same regime, the attention score identified 19 percent of context members with zero false alarms on one model and 63 percent on another. It did so without relying on shadow models or auxiliary population data, requiring only access to the model itself.

03Synthetic Pre-Training Ends at the Weights

Tabular foundation models are commonly assumed to present limited privacy concerns because they are often pre-trained on large collections of synthetic data. The assumption holds for the weights and stops there. Synthetic pre-training shapes parameters, and parameters are where memorization of a training corpus would live. It has nothing to say about the real customer records a team hands the model at inference as labelled examples, which is exactly where the measured leakage originates. The variant of one model that was further trained on real data behaved almost identically under attack, which points at the architecture rather than the pre-training corpus as the source of the exposure.

That is not an argument against synthetic data, which remains a genuinely strategic capability for expanding test coverage, generating rare and adverse cases deliberately, and reducing dependence on sensitive production data. It is an argument about where the governance commitment ends. Synthetic data may reduce privacy risk, it is not automatically anonymous, and its benefit stops at the boundary of the weights it shaped. In diligence, "pre-trained on synthetic data" answers a question about the training corpus. The inference-time question, which real records the runtime holds and what it reveals about them, has to be asked separately.

Parameter adaptation adds a second and separable channel on top of that. Before fine-tuning, the relative-likelihood attack scored at 0.49 AUC, essentially chance; after fine-tuning on a private dataset it rose to about 0.57. What discriminated members from non-members was not the absolute confidence but how much that confidence changed through the update. The movement is modest, and it belongs in a risk register as its own line item rather than folded into the context exposure it sits on top of. It says nothing about removing anything from a trained model, which remains an unsolved problem the honest answer to is retraining from a cleaned dataset on a best-effort basis.

04The Defense Is a Routing Decision

The countermeasure the same evaluation tested runs in four steps, and a platform team will recognize every one of them.

The shape matches classification-aware model routing closely enough that a platform team can build both with the same discipline: a policy evaluated at a runtime boundary, a differentiated path for the requests that warrant it, and an audit record of which path each request took. Enforcement emitted by the system carries a cost that can be quoted to a compliance officer in a single number, and quoting it is the point.

The limits belong in the same breath as the mechanism. An adversary operating in real time can search for query versions that preserve membership information while falling below the high-risk threshold, thereby avoiding the defense entirely, which makes false negatives the surface the design itself creates. The grouping runs into its own constraint when high-risk labels are rare, which is precisely the minority-class case regulators press hardest on. And leakage may persist through residual output-confidence signals even after the context keys have been protected. A control with a stated operating envelope is still a control. It is the version you can defend under questioning, because you can say what it covers and what it leaves open.

A policy document cannot reach the forward pass. What can reach it is scoping built into the runtime: which records are eligible to enter a context window for which tenant and which classification, provenance carried on every record so that unvetted content cannot promote itself into trusted institutional memory, a risk score computed at the retrieval boundary before the model sees the batch, and an audit record showing which queries were served against a modified context. Each of those is a decision with a measurable cost. Measurable cost is what survives the audit.