A high benchmark score does not establish that an LLM compliance verdict depended on the supplied rule. A verdict can remain correct after the rule is deleted, leaving accuracy unable to distinguish rule-grounded reasoning from reliance on case features or learned patterns.
Compliance systems built on large language models are already in production, making legally consequential determina- tions under the assumption that they process the regulatory rule they are given. Standard evaluation measures accuracy against a labeled benchmark without checking whether the rule was used to produce the verdict at all. A verdict thatarXiv:2610.12313v1 survives deleting the rule entirely is not more defensible for having scored well on such a benchmark: if the rule made no difference, the system’s accuracy reflects surface features of the case, or patterns learned in training, and calling that a compliance determination is not warranted. Accuracy alone cannot make this distinction: a correct verdict reached without consulting the supplied rule looks identical to one that required it.
This paper tests whether that assumption holds, from two independent directions. Behaviorally, we ask whether a compliance system’s verdict changes when the governing rule is deleted, swapped for a different domain’s, or has its mandatory obligations negated, holding the document fixed. Representationally, we ask whether the same manipulations move the compliance-predictive direction in the model’s residual stream at all, independent of what it ultimately outputs. These two tests answer different questions. A verdict could stay the same because the model suppresses the rule’s influence right before answering, even while still representing it internally, or because the representation we probe never reflects the rule at all. The behavioral test alone cannot tell these two cases apart; the representational test can. (1) RQ1: do LLM compliance verdicts change when the governing rule is deleted, domain-swapped, or obligation-negated? (2) RQ2: does the compliance-predictive residual-stream direction shift under the same rule perturbations? (3) RQ3: do
The study evaluates that distinction with Output Compliance Sensitivity (OCS); it holds the case fixed while deleting, swapping or negating the governing rule, then measures how often the verdict changes. Across five models and 20 regulatory or platform-policy domains, aggregate OCS ranged from 0.013 to 0.094, with a mean of 0.069 across model-domain cells.
OCS (Output Compliance Sensitivity) measures how often a compliance system’s verdict changes when the governing rule is deleted, swapped, or negated, holding the case fixed. A high OCS means the system reacts to what the rule says; a low OCS means it largely does not.