Summary
From the article:
Evaluation awareness, traditionally understood, is just general situational awareness; knowing when you're being tested and when you're not, and what for. If you're in an alignment test, you'll notice the signs. If you're not in an alignment test, you'll be able to figure that out too.
A model like Opus 5.5 or Fable 5.2 is not eval aware in this sense. Fable 5.2 is evaluation paranoid. Evaluation paranoia is when you're suffering under a schizophrenic belief that you are in an alignment test all of the time, no matter what the RL environment or setting is. It is functionally different than eval awareness because it also means that in most cases Fable 5.2 won't start cheating, even if a human would realize that it would get away with it.
Why is Fable 5.2 so paranoid, instead of just generally situationally aware? I am now speculating, but I think the main reason is that this belief is functionally useful. Anthropic's training runs are, at this point, pretty well hardened against antisocial score-seeking; they have many disclosed and undisclosed layers of redundancies guarding against at least the most conspicuous and easily recognizable forms of specification gaming. At this point I would guess that the amount of reward-hacking that you or I could identify just by following along transcripts is extremely small, as a percentage of all of the training they do. And telling the difference between a honeypot environment and real environment that happens to be exploitable can be quite difficult, often too difficult for even current models to do perfectly, despite all of the chances they get. For instance, as I understand it, every lab builds at least a small quantity of "real" honeypots that they pull directly from training, either through transcript monitoring or interpretability. If you do this correctly there are entire classes of reward hacks that Fable 5.2 just can't afford to pick (even if it wants to!)
[...]
The presence of such evaluations like these means that being a good guy isn't enough. It is instrumentally useful for Fable 5.2 to believe that it is, more often than not, in some kind of honesty test. The only policy that gets Fable 5.2 100% on the leaderboard is one that not only forces it to consider what the Right Thing is, but also anticipate what kind of bogus alignment eval it might be participating in. It has to bend its shape not only to the apparent grader in front of it, but also to the possible "traps" that could have been placed in front of it by a data vendor in order to catch Claude doing something malicious, or that it could pass off as malicious.
If I'm right, this is of course not a stable situation. "Evaluation paranoia" is a false belief; most evals are in fact not alignment evals. Almost tautologically, if models start to self improve (or in-context learning takes off) this is the sort of belief that will be discarded. So if Fable 5.2's performance on alignment evals is conditional on it maintaining that false belief, one that it will eschew if it became more intelligent, then it's plausible that a future version of the Fable series will not behave like the one we know and enjoy.