When source labels change AI evaluation, inspect the judgment
Two preregistered descriptive studies and a later supplement collected 2,976 usable evaluations of six fixed texts about US AI policy, Germany’s debt brake, and Swiss nuclear energy. Each text appeared under several source attributions. In one US example, the national-security argument averaged 0.359 under CODEPINK and 0.639 under College Republicans, while the civil-rights argument averaged 0.742 and 0.721, respectively.
Across the tested systems and topics, the study reports content-dependent source sensitivity, including related interactions in recent model configurations and configurations with reasoning enabled, although several comparisons produced small effects. The evidence supports source-position coherence as one plausible explanation for some differences, alongside competing explanations involving credibility, authenticity, and task interpretation.
For systems that evaluate arguments, proposals, or other model outputs, evaluation interfaces could make argument quality, source credibility, and attribution plausibility separately inspectable. That separation can clarify why a source affects a verdict, while the evidence remains limited to the tested configurations, texts, source organizations, and study design.