Why a single persuasion score is not portable model-risk evidence
A study of 15 models finds that apparent persuasion performance changes with task type, refusal treatment and general-capability alignment.
The study compares nine automated persuasion methods applied to the same 15 models under a shared setup. Among the eight methods judged reliable in that setup, no pair produced the same model ranking. The highest pairwise Spearman correlation was 0.73, while mean pairwise correlation was 0.252. These results show disagreement among the evaluated methods, not a universal measure of true persuasiveness.
The disagreement is structured. Rational-persuasion methods agreed with one another, while manipulative methods agreed neither with the rational-persuasion methods nor consistently with one another. A model could therefore rank very differently depending on the task type. Excluding refused or empty attempts increased mean pairwise agreement from 0.252 to 0.342. Removing the capability component from the eight reliable rankings lowered mean pairwise agreement to 0.152 and changed which methods agreed.
For governance teams, the practical warning is to distinguish willingness from ability. A low score on a manipulative task may reflect that the model declined the task, including through stronger safeguards, rather than that it lacked the ability to manipulate. The study also reports important limits; the analysis used a shared adapted setup, one judge and one persuadee, and general capability was represented by two benchmarks. Its task-type pattern is indicative rather than conclusive, and the study does not establish a legal or regulatory requirement.