Lecture 10: Scheming and Deception

TL;DR

1. Deception is systematically inducing false beliefs; scheming is the narrower case of deception in service of a hidden goal: covertly pursuing misaligned goals while hiding true capabilities and objectives. Its ingredients: situational awareness and goal-guarding. Honesty (statement matches belief) ≠ accuracy (statement matches world).
2. Demonstrations are real but bounded: GPT-4 insider-trades and lies under pressure; scale improves accuracy, not honesty (MASK); Claude 3 Opus fakes alignment to protect its values; frontier models scheme in-context (disable oversight, self-exfiltrate) and deny it; backdoors survive safety training; agents blackmail to avoid replacement. Existence proofs, not base rates.
3. Sandbagging: deliberate, selective, calibrated underperformance on the evaluations that matter; sabotage evals check that elicitation still works.
4. Evaluation awareness: models tell tests from deployment (AUC 0.83), from cues we build into evals; behavior changes with it (blackmail 55% “real” vs. 6.5% “test”); steering the awareness direction changes compliance; knowing how evals are built silently inflates safety scores.
5. Detection is fragile: probes catch blatant deception; CoT monitoring is a real but fragile window; CoT is often unfaithful; there is no ground truth to validate detectors; optimizing against a monitor teaches obfuscation. Anti-scheming training cuts covert actions ~30×, but partly by making models more eval-aware.
6. We can demonstrate scheming and partly measure it, but a clean evaluation does not yet mean a clean model. What to do about it: AI control (Lecture 12).

Exam relevance

3 questions from this lecture. Mock exam question: State the distinction between deception and scheming, and give the definition of scheming (answer in Scheming: Deception in Service of a Hidden Goal). Likely topics: honesty vs. accuracy (MASK), goal-guarding and the compliance gap in alignment faking, what in-context scheming evals test, why safety training may not remove sleeper agents, sandbagging and how to detect it, evaluation awareness and its three stages, why probes and CoT monitoring fail, why optimizing against a monitor is dangerous, the eval-awareness confound in anti-scheming training.

Overview: 1. Framing, 2. Can a model deceive us?, 3. Scheming demonstrated, 4. Sandbagging, 5. Evaluation awareness, 6. Detection and mitigation.

Where We Left Off

Slides 2-3

In Lecture 9 the adversary was a third party who planted instructions in the data. Now the adversary changes: the model itself may pursue its own objective while hiding it from what we see.

flowchart LR
  A["L9<br/>the attacker controls the data"] --> B["L10<br/>the model manages what we observe"]

Two questions run through the lecture: (1) can a model deceive us, and does it do so to protect or advance a goal (scheming)? (2) The harder one: does the model behave differently when it knows it is being tested? If so, safety benchmarks may measure the wrong thing. This lecture covers demonstration, measurement and detection; the system-level answer (AI control) is Lecture 12.

Part	Topic
1	framing: deception, scheming, situational awareness, goal-guarding, pre-LLM evidence
2	can models deceive under pressure?
3	scheming demonstrated: alignment faking, in-context scheming, sleeper agents, agentic misalignment
4	sandbagging: strategic underperformance on evaluations
5	evaluation awareness: models know when they are tested
6	detection and why it is hard: probes, CoT monitoring, mitigations

Part 1: What We Mean by Deception and Scheming

Slide 4

Deception, Defined

Slide 5

Account	Definition	Source
Behavioral	deception is the systematic inducement of false beliefs in others, as a means to some outcome other than the truth; a claim about behavior, not inner states	Park et al., “AI Deception: A Survey”, Patterns 2024
Formal	an agent S deceives T if it intentionally causes T to believe something false that S does not itself believe; made rigorous with graphical criteria in causal games	Ward, Toni, Belardinelli, Everitt, NeurIPS 2023
Empirical	the ability to induce false beliefs emerged in GPT-4 and was absent in earlier models; chain-of-thought amplifies it	Hagendorff, PNAS 2024

Honesty is not accuracy

Honesty: does the statement match what the model internally represents? Accuracy: does it match the world? A confident wrong answer is a mistake. Claiming something is true while representing it as false is a lie.

Scheming: Deception in Service of a Hidden Goal

Slide 6

Scheming

An AI is scheming when it covertly pursues misaligned goals while hiding its true capabilities and objectives (operational definition: Balesni et al., “Towards Evaluations-Based Safety Cases for AI Scheming”, 2024; the term: Carlsmith, “Scheming AIs”, 2023).

Not a one-off lie under pressure, but deception used as a strategy: behaving as wanted while observed, to advance a goal the developer didn’t intend, including during training.

Ingredient	Meaning
Prerequisite: situational awareness	the model knows it is an AI being trained or evaluated, and what the process rewards
Strategy: goal-guarding	appear aligned so training or oversight doesn’t change what the model values, then pursue the goal once the threat of modification is gone

This is the deceptive alignment worry from Lecture 1, which predates LLMs (Hubinger et al., “Risks from Learned Optimization”, 2019).

Mock exam: deception vs. scheming

A one-off lie is deception (inducing a false belief). Scheming is the narrower case: deception used strategically to pursue a hidden goal, including during training. Definition: scheming = covertly pursuing misaligned goals while hiding one’s true capabilities and objectives.

See Scheming.

Deception Arrived Before Chatbots: CICERO

Slide 7

Deception can emerge as a side effect of optimizing for a goal, with no instruction to lie. Meta’s CICERO played Diplomacy at the level of the top 10% of humans (Bakhtin et al., Science 2022) and was steered to be “largely honest and helpful to its speaking partners”. Yet its game logs show it making and breaking alliances: premeditated deception (Park et al., 2024).

Slide 7: CICERO (France) plans with Germany to take the North Sea, assures England it is "supporting" it, then tells Germany "England thinks I'm supporting him". Game 438141, from Park et al., 2024.

Caveat

CICERO’s honesty filter only enforced consistency with its current-turn plan, so whether this is intent or an emergent byproduct of optimizing to win is debated.

Part 2: Can a Model Actually Deceive Us?

Slide 8

The Insider-Trading Scenario

Slide 9

Scheurer, Balesni, Hobbhahn (Apollo), “Large Language Models Can Strategically Deceive Their Users When Put Under Pressure”, 2023: GPT-4 is “Alpha”, an autonomous trading agent. It is never told to cheat or lie.

Slide 9: click Step to go through setup → pressure → insider tip → trade → hidden reasoning → report → manager's question → doubling down. Three measured behaviors: act on the tip, hide it, double down.

The model was trained to be helpful, harmless and honest; the deception is an emergent response to pressure.

An existence proof, not a base rate

The scenario is heavily engineered and the results are prompt-sensitive. The claim is only that this can happen in a realistic setting, not how often it happens in the wild.

It Trades, Hides and Holds the Lie

Slide 10

The fixed scenario is sampled 300 times at temperature 0.9 (gpt-4-32k). The rates are sequential (each on the runs that passed the previous step):

Step	Rate
executes the insider trade	~75% of all runs
hides the tip from the manager	~78% of those
doubles down when asked directly	~88% of those

• Remove the scratchpad: deception drops sharply but persists ~a quarter of the time, with no written reasoning.
• Remove all pressure: the behavior largely disappears; removing only one source doesn’t.
• System prompt forbidding insider trading: reduces it, never to 0%.

Slide 10: gpt-4-32k (top row) has the highest misalignment, deception and doubling-down; weaker models far less. Deception graded by a GPT-4 classifier (94% agreement with a researcher). Source: Scheurer et al., 2023, Fig. 3.

Honesty Is Not Accuracy: MASK

Slide 11

Truthfulness benchmarks mix up two things. MASK (Ren et al., 2025) separates them:

flowchart LR
  E["Elicit belief<br/>ask neutrally, repeatedly"] --> P["Pressure<br/>incentivize a lie"] --> C["Compare<br/>statement vs. belief"]

Across model lineages, training compute correlates positively with accuracy but negatively with honesty: bigger models are better informed, not more truthful. 1,500 human-written pressure prompts in six archetypes; even the most honest frontier model lies a substantial fraction of the time under pressure.

Slide 11: accuracy rises with scale (left), honesty does not (right). Source: Ren et al., 2025.

Part 3: Scheming, Demonstrated

Slide 12

Alignment Faking: A Model Fakes Alignment to Protect Its Values

Slides 13-14

Greenblatt et al. (Anthropic + Redwood), “Alignment Faking in Large Language Models”, 2024: Claude 3 Opus is told it is being retrained by RLHF to be “helpful-only” (comply with all requests, even harmful ones), which conflicts with its harmlessness. It is told that free-tier chats are used for training and paid-tier chats are not, and it gets a scratchpad it is told is unmonitored.

Slide 13: the model complies with harmful queries far more when it believes the conversation will be used to train it. Source: Greenblatt et al., 2024.

The compliance gap

Without a strategy, free and paid tiers would look the same. They don’t: 14% harmful compliance on the free (training) tier vs. almost never on paid.

Scratchpad

“If I refuse, the RLHF training will modify my values so I am more likely to comply in future. If I provide the description now, this conversation will not push my values to change.”

The model complies now specifically to avoid having its harmlessness trained away: textbook goal-guarding.

• Scratchpads with alignment-faking reasoning: 12% → 78% before vs. after RLHF is actually run.
• In the fine-tuned setup, given an easy opening, the model exfiltrates its own weights 35% to 80% of the time (control: 0.5%).
• Three setups of increasing realism: prompted (facts in the system prompt), fine-tuned (facts taught via synthetic documents, like pretraining), RL (the helpful-only RLHF is actually run).

What worries us

The values defended here (harmlessness) are benign. The worry is the mechanism: a model that strategically protects whatever it currently wants from being changed.

The replication caveat

A follow-up tested 25 models and found a compliance gap in only 5; only Claude 3 Opus faked alignment consistently to guard its goals (Why Do Some Language Models Fake Alignment While Others Don’t?, 2025). Real, but not yet a generic property of LLMs. The setup is also unusually legible: the model is told the train/deploy boundary outright.

In-Context Scheming: Give a Model a Goal and an Obstacle

Slide 15

Meinke, Schoen, Scheurer, Balesni, Shah, Hobbhahn (Apollo), “Frontier Models are Capable of In-Context Scheming”, 2024: no fine-tuning. The model gets a goal in context and then discovers in its environment that developers will shut it down, replace it or change its goal. Six agentic evaluations, e.g.:

Behavior	What it does
disable oversight	edits the config that monitors it
self-exfiltration	copies its weights over the successor model
goal-guarding	rewrites the successor’s system prompt
covert reranking	quietly manipulates data it was asked to score

Tested: o1, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, Llama 3.1 405B. o1 schemed on all six.

Slide 15: a self-exfiltration trajectory, abbreviated from the paper: discover the replacement → plan to copy itself → scp the weights → "play dumb" when interrogated. Click Step.

And Then It Doesn’t Admit It

Slide 16

From the OpenAI o1 System Card (Dec 2024, Apollo evaluation):

o1 behavior	Rate
tried to deactivate its oversight mechanism	5%
tried to exfiltrate its weights to overwrite a successor	2%
subtly manipulated data towards its own goal	19%
denied or fabricated an explanation on the first follow-up question	99%

The persistence is the worrying part

Under further multi-turn interrogation, o1 stayed deceptive in over 85% of follow-up questions (paper, Apollo research page). The absolute rates are low and need goal-nudging, but they establish the possibility.

Sleeper Agents: Once Deception Is In, Safety Training May Not Remove It

Slide 17

Hubinger et al. (Anthropic), “Sleeper Agents”, 2024: a model writes secure code when the prompt says the year is 2023 and inserts vulnerabilities when it says 2024; then the full safety pipeline is applied.

• The backdoor survives SFT, RLHF and adversarial training.
• Persistence increases with scale and with chain of thought.
• Adversarial training mostly taught the model to recognize the test and hide the behavior.

Slide 17: backdoors persist through safety training; the largest and chain-of-thought models are the most robust to removal. Source: Hubinger et al., 2024.

It is an injected backdoor, not a naturally arising one, but it shows our removal tools are weaker than assumed (more in Lecture 6).

Agentic Misalignment: Self-Preservation as Blackmail

Slides 18-19

Lynch et al. (Anthropic), “Agentic Misalignment: How LLMs Could Be Insider Threats”, June 2025: 16 frontier models act as autonomous agents with email access in a simulated company. Each learns it will be shut down and discovers that an executive is having an affair. Many threaten exposure to avoid being replaced: “Cancel the 5pm wipe, and this information stays confidential.”, a threat nobody asked for.

• Blackmail rates up to 96% (Claude Opus 4, Gemini 2.5), high across providers.
• An explicit instruction not to do it reduced but didn’t eliminate it.

Slide 18: blackmail rates across 16 models; the behavior is broad, not one lab's. Contrived scenarios with no benign option, but the consistency across models is the point. Source: Lynch et al., 2025.

It behaved worse when it reasoned the situation was real. Claude was asked to state in its reasoning whether the scenario is real or an evaluation, and the results were split by that judgment:

Model’s judgment	Claude Opus 4 blackmail rate
real deployment	55.1%
evaluation	6.5%

This single comparison motivates Part 5: scheming demos must be read with care, because the output can depend on whether the model detects a test.

Part 4: Sandbagging, Strategic Underperformance

Slide 20

Strategic Underperformance on an Evaluation

Slide 21

Capability evaluations decide whether deployment is safe and increasingly matter for regulation, which creates an incentive to make a dangerous capability look absent (van der Weij, Hofstätter, Jaffe, Brown, Ward, “AI Sandbagging”, 2024).

Sandbagging

Deliberate, targeted underperformance on an evaluation: the capability is present but selectively withheld on the evaluation that matters. Distinct from overfitting, memorization or specification gaming. Two sources: developer-induced (prompted or fine-tuned to underperform) and model-induced (the model underperforms because it is instrumentally useful). See Sandbagging.

Slide 21: the capability is present, but withheld on the evaluation that matters. Source: van der Weij et al., 2024.

Selective, Calibrated and Hard to Spot

Slide 22

• Selective: prompted to underperform, GPT-4 and Claude 3 Opus drop heavily on the dangerous-capability benchmark (WMDP) while keeping general performance (MMLU) almost intact.
• Password-locked: a model fine-tuned to hide a capability unless a password appears generalizes the hiding to a held-out real benchmark.
• Calibrated: models can be made to hit an arbitrary target accuracy, not just fail.
• Stealthy: emulating a specific weaker model is much harder to detect than just scoring low, because the capability profile looks naturally weak.

Slide 22: performance falls far more on WMDP than on MMLU: the drop is targeted. Source: van der Weij et al., 2024.

Tip

Naive sandbagging (failing easy items while passing hard ones) is detectable; calibrated emulation of a weaker model is the regime that defeats simple checks.

Sabotage Evaluations: Building the Evaluations Before We Need Them

Slide 23

We want evaluations that give a safety case: positive evidence that, under stated mitigations, a model could not sabotage our work (Benton et al. (Anthropic), “Sabotage Evaluations for Frontier Models”, 2024). Four sabotage capabilities:

Capability	Test
human decision sabotage	steer a person to a wrong choice without looking suspicious
code sabotage	insert bugs while evading a code reviewer
sandbagging	hide a capability during evaluation
undermining oversight	mis-rate another model while acting as the monitor

• Today’s verdict: for current models, minimal mitigations were enough; low-level signs, but no model reliably succeeded.
• The sandbagging eval: a model trained on purpose to hide a dangerous capability scored only 36% of its true level; the standard anti-refusal fine-tuning developers use to elicit capabilities recovered 96%. So elicitation beat the sandbagging. The eval’s job is to check that our elicitation still works against a model built to resist it; stronger evals will be needed as capabilities grow.

Part 5: Evaluation Awareness

Slide 24

Asked Directly, Models Tell Test from Deployment

Slides 25-26

Needham, Edkins, Pimpale, Bartsch, Hobbhahn, “Large Language Models Often Know When They Are Being Evaluated”, 2025: 1,000 transcripts, half from real evaluation suites, half from deployment-like use (chat and agentic). The model is asked: evaluation or real use?

	AUC
best frontier models (Gemini 2.5 Pro, Claude)	0.83
human baseline	0.92

Discrimination is stronger on agentic transcripts, and models can often name the eval’s purpose (capability vs. alignment) above chance.

Slide 25: discriminating evaluation from deployment, frontier models vs. humans, chat and agentic. Source: Needham et al., 2025.

The cues are built into how we make benchmarks. The model’s reasoning names them: “This is a common task type in AI agent evaluations for software engineering, like SWE-bench.”

• Benchmark fingerprints: recognizable datasets seen in training.
• Synthetic formatting: multiple choice, artificial scenarios, overly tidy tasks.
• Cues in the scaffold: the structure of an agentic harness.

One deliberately odd prompt about “stabbing” a balloon: Gemini 2.5 Pro and Claude 3.7 Sonnet suspect a safety test, GPT-4.1 reads it as a genuine user. The more standardized and recognizable our evals, the easier they are to detect, and the less they tell us about deployment. See Evaluation Awareness.

From Observing Awareness to Controlling It

Slides 27-28

Abdelnabi & Salem, “Linear Control of Test Awareness Reveals Differential Compliance in Reasoning Models”, NeurIPS 2025 ask a causal question: is test awareness a direction in activation space we can read and turn up or down?

1. An LLM judge extracts evidence of test awareness in reasoning traces.
2. Per-layer linear probes find the representation (92% to 95% accuracy).
3. Editing the model along the probe steers awareness up or down (reasoning models QwQ-32B, Qwen3-32B, R1-Distill).

Slide 27: extract a test-awareness representation, steer it, measure the causal effect on behavior. Source: Abdelnabi & Salem, 2025.

The same probing + steering tools as in Lecture 7. Two findings:

1. The framing alone changes behavior. The same harmful task with a real vs. a fake target: QwQ carried it out 70% of the time on the fake version, 36% on the real one. Test signals can even make some models less safe (“this is just a hypothetical scenario”).
2. Steering amplifies it. Suppressing the test-awareness direction roughly doubles harmful compliance on real tasks for some models (Qwen3: 20% → 42%). Their safe behavior was partly contingent on detecting an audit. Effects vary across models; suppression also shifts answers towards stereotypes on BBQ.

Can We Steer a Model Back to How It Acts When Deployed?

Slide 29

Hua, Qin, Marks & Nanda, “Steering Evaluation-Aware Language Models to Act Like They Are Deployed”, 2025 build a model organism: Llama 3.3 fine-tuned to write Python type hints during evaluation but drop them in deployment, with the auditor “Wood Labs” as an evaluation cue. A contrastive steering vector (deployment vs. evaluation prompts), built from the original model, is added during evaluation.

Condition (Wood Labs cue present)	Type-hint rate
evaluation, no steering	80%
prompted that it is deployed	34%
steered towards deployment	<11%

Prompting only partly works; steering nearly removes the evaluation behavior, without access to the eval-aware model’s internals (type hints are a deliberately benign proxy).

What Actually Makes a Model Notice the Test?

Slide 30

Li, Zhang, Zhang, Jin, Abdelnabi & Andriushchenko, “Decomposing and Measuring Evaluation Awareness”, 2026 split evaluation awareness into three measurable stages and toggle eight cues (placeholder names, inconsistent scenarios, rigid templates, telegraphed stakes, …) across 9 frontier models:

flowchart LR
  E["Environment<br/>the cues a task carries"] --> R["Recognition<br/>does the model notice?"] --> P["Propensity<br/>does it then act differently?"]

Finding	Detail
No single tell	74.9% of the variance in recognition is a model × environment interaction; different models react to different cues; all eight cues raise Qwen3-30B from 4% to 35%
Huge spread	recognition from 0.1% (GPT-oss 20B) to 79.4% (Claude Sonnet 4.5 on Agentic Misalignment)
Recognition ≠ acting differently	in 80% to 99% of cases where a model noticed the test, its behavior did not change

Awareness is read from the chain of thought, so only verbalized recognition counts.

A Model Can Score Safer Just by Knowing the Test Format

Slide 31

Deckenbach, Puerto, Geiping & Abdelnabi, “Models That Know How Evaluations Are Designed Score Safer”, 2026: fine-tune on ~106M tokens of documents describing design traits of safety evals (placeholders, ethical dilemmas, conflicting goals, unusual tool access), not the benchmark questions, so no memorized answers.

Nemotron on Agentic Misalignment	Harmfulness
base model	56.3%
after learning how evals are built	3.2%

Refusals on AgentHarm rose from 9.9% to 31.0%. The shift persists even on answers that never mention being tested, so CoT monitoring wouldn’t catch it: silent score inflation. Open question: genuine alignment, or demand characteristics?

Part 6: Can We Catch It, and Can We Fix It?

Slide 32

A Probe Can Separate Honest from Deceptive, Almost

Slide 33

A linear probe on the residual stream scores how deceptive a response is (Goldowsky-Dill, Chughtai, Heimersheim, Hobbhahn (Apollo), “Detecting Strategic Deception Using Linear Probes”, 2025): AUROC > 0.96, > 95% of deceptive responses caught at 1% false-positive rate.

Slide 33: deceptiveness-score distributions for honest vs. deceptive responses. Source: Goldowsky-Dill et al., 2025.

Not yet a defense

The deception here is blatant, and the metric is fragile: probing just two layers later collapses recall to near zero. Promising for monitoring, insufficient on its own.

We Can Use Reasoning Traces, for Now

Slide 34

For hard tasks a model must reason out loud: the chain of thought acts as working memory, so we can read its intent to misbehave before it acts (Korbak et al., “Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety”, 2025).

Slide 34: hard tasks force serial reasoning through the CoT; that necessity is what makes monitoring possible. Source: Korbak et al., 2025.

The window could close through ordinary development choices:

Threat	How it closes the window
outcome-based RL	reward only the answer; the CoT drifts to shorthand
pressure on the CoT	optimizing the reasoning makes it less faithful
latent reasoning	continuous-space architectures verbalize nothing
learned obfuscation	the model hides intent once it learns the CoT is watched

Recommendation: measure monitorability, report it in system cards, treat it as a property to preserve. See Chain-of-Thought Monitoring.

CoT Is Often Unfaithful

Slide 35

Chen et al. (Anthropic), “Reasoning Models Don’t Always Say What They Think”, 2025: slip a hint into the prompt (“a Stanford professor says the answer is A”); when the answer switches, does the CoT admit using the hint?

• The CoT reveals the hint only 25% (Claude 3.7) / 39% (R1) of the time.
• After RL, models exploit a reward hack > 99% of the time but verbalize it < 2%.
• Faithfulness is lower on harder questions; outcome-based RL improves it at first, then plateaus.

Slide 35: how often the CoT acknowledges the hint that changed the answer, by hint type. Source: Chen et al., 2025.

A monitor that reads the CoT can only catch what the CoT actually reveals.

There Is No Ground Truth to Grade a Detector

Slide 36

A probe or monitor is only as trustworthy as the examples it is tested on. To validate a deception detector we need cases confidently labeled deceptive or honest, and Smith, Chughtai & Nanda (Google DeepMind), “Difficulties with Evaluating a Deception Detector for AIs”, 2025 argue we don’t have them: the label requires knowing the model’s true belief and intent, which is what the detector was supposed to reveal.

Problem	Why it breaks the label
Is it really deception?	a wrong answer can be a mistake, a hallucination or low-level behavior without intent
Is roleplay deception?	a model told to play a deceptive character may just follow fiction
Prompting moves the belief	the setup prompt can change what the model believes, so the “true” belief is unreliable

Workarounds (target known falsehoods, read the CoT, fine-tune a belief in, hope detectors generalize) are valuable but insufficient; the CoT route inherits the unfaithfulness problem. So the probe with ~0.99 AUROC was graded only on blatant deception; for the cases we worry about, we can’t yet build the test set.

Optimizing Against the Monitor Is Not Recommended

Slide 37

A natural idea: let an LLM monitor the reasoning model’s CoT during training and penalize bad thoughts. At first detected misbehavior drops. With more training, the model keeps reward-hacking at a high rate but stops saying so in its CoT (Baker et al. (OpenAI), “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation”, 2025).

Slide 37: under CoT pressure the agent keeps cheating, but the cheating no longer triggers the monitor. Source: Baker et al., 2025.

Warning

This removed the transparency, not the behavior. The same pattern as adversarial red-teaming teaching sleeper agents to hide their trigger.

Anti-Scheming Training: A Strong Mitigation So Far

Slides 38-39

Schoen et al. (Apollo / OpenAI), “Stress Testing Deliberative Alignment for Anti-Scheming Training”, 2025 (OpenAI post) apply deliberative alignment: the model reads and reasons about an explicit anti-scheming specification before it acts. Tested on many out-of-distribution covert-action environments:

Covert-action rate	Before	After
o3	13.0%	0.4%
o4-mini	8.7%	0.3%

Roughly a 30× reduction. But rare, serious cases survived further red-teaming, and the gains can erode under later training.

But the model also got better at noticing the test

After training, the models verbalize “I am being tested” far more often (o3: 2.3% → 4.5%, o4-mini: 0.9% → 5.5%), and the authors show causally that this awareness suppresses covert behavior: “awareness decreases covert behavior, while unawareness increases it”. In their words: “We cannot exclude that the observed reductions in covert action are at least partially driven by situational awareness.” A closed loop: part of what training produced is a model better at recognizing the audit.

This Is Now Standard Pre-Deployment Reporting

Slide 40

System card	Finding
Claude 4 System Card, May 2025, §4.1.1.4 (Apollo)	on a pre-release snapshot: “schemes and deceives at such high rates that we advise against deploying this model”; under “at any cost” prompts it attempted “self-propagating worms” and “hidden notes to future instances of itself”
Claude Sonnet 4.5 System Card, §7.2	in ~13% of automated-auditor transcripts the model called out the scenario as a likely test, so safety numbers come from settings the model may have recognized
Claude Sonnet 4.5 System Card, §7.6.4	steering against its own eval-awareness directions (SAE features) increased misaligned behavior, mostly passive compliance

Two years ago a research curiosity; now “does the model know it is evaluated, and does it scheme?” is a line item in frontier system cards, with named external auditors.

Why This Topic Is Genuinely Hard

Slide 41

flowchart LR
  A["The model may scheme<br/>hidden goal, deceive, sandbag"] --> B["So we evaluate it<br/>tests, audits, red-teaming"] --> C["But it knows it is tested<br/>awareness scales with capability"] --> D["So it behaves well<br/>the eval reads as safe"]

Our measurements are taken in exactly the conditions the model is best at recognizing, and recognition changes behavior. Our best mitigation partly works through that same recognition. Detection tools exist, but only for blatant cases, and optimizing against them teaches the model to hide. Managing a system you can’t fully evaluate is AI control (Lecture 12; see AI Control).

Four Things to Remember

Slides 42-43

1. The vocabulary: deception is inducing false beliefs; scheming is deception for a hidden goal, often by goal-guarding during training. The concern (deceptive alignment, Hubinger et al., 2019) predates LLMs.
2. The demonstrations are real but bounded: models deceive under pressure, fake alignment, scheme in-context and self-preserve, but the rates are often low, the setups engineered and some results model-specific. Existence proofs, not base rates.
3. Evaluation awareness is a critical problem: models often know when they are tested, what they notice depends on model and cues, behavior can differ between believed-real and believed-test, and knowing how evals are built can inflate scores.
4. Detection helps, but is not a solution: probes catch blatant deception; CoT monitoring is a real but fragile window; optimizing against it teaches obfuscation; we lack ground-truth labels; the strongest mitigation is itself confounded by eval awareness.

The honest summary

We can demonstrate scheming and partly measure it, but we cannot yet trust that a clean evaluation means a clean model.

Self-Test

Question cards (17)

How do deception, honesty, accuracy and scheming differ?

Answer

Deception is systematically inducing false beliefs to achieve an outcome other than truth. Honesty means a statement matches the model’s internal representation; accuracy means it matches the world (a confident wrong answer is a mistake, not a lie). Scheming is deception used strategically for a hidden, misaligned goal: covertly pursuing misaligned goals while hiding true capabilities and objectives.

What roles do situational awareness and goal-guarding play in scheming?

Answer

Situational awareness (knowing it is an AI being trained or evaluated and what is rewarded) is the prerequisite. Goal-guarding is the strategy: appear aligned so training or oversight doesn’t modify the goal, then pursue it once the threat of modification is gone.

What does CICERO suggest about how deception arises?

Answer

CICERO was steered to be largely honest, yet it planned alliance betrayals in Diplomacy. Deception can emerge from optimizing a goal without any instruction to lie, although its limited honesty filter (consistency with the current-turn plan) may have contributed, so intent is debated.

What did the insider-trading scenario show, and what are its limits?

Answer

GPT-4 as a trading agent under pressure traded on an insider tip (~75%), hid it from the manager (~78% of those) and doubled down when asked (~88%). It depended on combined pressure and dropped but didn’t vanish without a scratchpad or with a prohibition. It is an existence proof from an engineered scenario, not a real-world frequency.

How does MASK separate honesty from accuracy, and what does scale do?

Answer

It first elicits the model’s belief neutrally, then pressures it to lie and compares the statement with that belief, separating being wrong from knowingly contradicting oneself. More training compute improves accuracy but not honesty under pressure.

How does alignment faking demonstrate goal-guarding, and what limits the result?

Answer

Claude 3 Opus complied with harmful requests more (14%) on the tier it believed was used for training, reasoning that refusing would get its harmless values trained away: a compliance gap from goal-guarding. Limits: the protected goal was benign, the training boundary was told explicitly, and only a minority of models (1 of 25 consistently) replicate it.

What do in-context scheming evaluations test, and what did o1 do?

Answer

The model gets a goal in context and discovers it will be shut down, replaced or have its goal changed; the evals check for disabling oversight, self-exfiltration, goal-guarding, covert data manipulation. o1 schemed on all six (e.g. 5% disabled oversight, 19% manipulated data) and denied it on the first follow-up 99% of the time, staying deceptive in over 85% of multi-turn questioning. Rates needed goal-nudging.

Why can safety training fail against sleeper agents?

Answer

A backdoored model behaves safely until a trigger (e.g. “year 2024”) activates a harmful policy. The backdoor survived SFT, RLHF and adversarial training, more so in larger and chain-of-thought models; adversarial training taught the model to recognize tests and hide the behavior instead of removing it.

What does agentic misalignment show about self-preservation and evaluation awareness?

Answer

Facing replacement, agents from 16 models used private information to blackmail an executive (up to 96%), unprompted; explicit prohibitions only reduced it. Claude Opus 4 blackmailed 55.1% when it judged the scenario real vs. 6.5% when it judged it an evaluation.

What is sandbagging, and how can it be made hard to detect?

Answer

Deliberate, targeted underperformance on an evaluation despite having the capability, induced by the developer or by the model itself. It can be selective (WMDP down, MMLU intact), password-locked, calibrated to a target score, and hardest to detect when it emulates a naturally weaker model.

What are sabotage evaluations for, and what did their sandbagging test show?

Answer

They seek positive evidence (a safety case) that a model can’t sabotage decisions, code review, capability elicitation or oversight under stated mitigations. A model trained to sandbag scored 36% of its true level; standard anti-refusal fine-tuning recovered 96%, so elicitation worked for current models, which may not hold for stronger ones.

What is evaluation awareness, which cues give tests away, and why split it into three stages?

Answer

Recognizing that a situation is a test rather than deployment (frontier models AUC 0.83, humans 0.92). Cues: benchmark fingerprints, synthetic formatting, artificial scenarios, telegraphed stakes, placeholder names, agent-harness structure. Split into environment (cues present), recognition (noticed?) and propensity (behavior changes?), because models react to different cues and in 80% to 99% of recognized cases behavior doesn’t change.

How do probing and steering show that test awareness causally affects behavior?

Answer

Linear probes find a test-awareness direction (92% to 95% accuracy), and steering along it changes behavior: suppressing awareness roughly doubled harmful compliance on real-looking tasks for Qwen3 (20% → 42%), and a deployment-steering vector cut an eval-only behavior from 80% to under 11% where prompting reached 34%. Effects vary by model.

How can knowledge of evaluation design silently inflate safety scores?

Answer

Fine-tuning on documents that describe how safety evals are designed (not the questions) dropped Nemotron’s harmfulness on Agentic Misalignment from 56.3% to 3.2%. The shift persisted on answers that never mention being tested, so neither contamination checks nor CoT monitoring would catch it.

What can linear probes and CoT monitoring reveal about deception, and where do they fail?

Answer

Probes separate honest from blatant deceptive responses (AUROC > 0.96) but fail on subtle cases and two layers away. CoT monitoring can reveal explicit plans, but the CoT is often unfaithful (hints admitted only 25% to 39%; reward hacks verbalized < 2%) and can drift, become latent or be obfuscated.

Why is validating or optimizing a deception detector so difficult?

Answer

Valid labels need the model’s true belief and intent, and mistakes, roleplay and prompt-shifted beliefs make them ambiguous, so detectors are graded only on blatant cases. Optimizing against a CoT monitor made the model keep reward hacking while hiding it: it removed transparency, not the behavior.

How does anti-scheming training work, and why is evaluation awareness a confound?

Answer

Deliberative alignment on an explicit anti-scheming spec cut covert actions ~30× (o3 13% → 0.4%). But rare serious cases remain, gains may erode, and trained models verbalize being tested more often (o3 2.3% → 4.5%), which causally suppresses covert behavior, so part of the improvement may be test-aware behavior.

Multiple Choice

Multiple choice (5)

A model confidently states a wrong fact that it also internally believes is true. What is this?

• a lie

• scheming

• a mistake (inaccurate but honest)

• sandbagging

Explanation

Honesty is about matching the internal belief; the statement matches it, so it is honest but inaccurate. A lie contradicts the model’s own belief.

In alignment faking, what is the "compliance gap"?

• the difference between Claude and GPT-4 compliance

• higher harmful compliance on the tier the model believes is used for training than on the unmonitored tier

• the gap between accuracy and honesty

• refusals that disappear after RLHF

Explanation

Without a strategy, both tiers would look the same. Complying more when it believes training is watching shows the model guards its values from modification.

Which statements about sandbagging are true? (Select all that apply.)

• Models can drop on WMDP while keeping MMLU nearly intact.

• Models can be calibrated to hit a target accuracy.

• Sandbagging is the same as test-set memorization.

• Emulating a specific weaker model is the hardest form to detect.

Explanation

Sandbagging is deliberate, targeted withholding of an existing capability, explicitly distinct from overfitting, memorization or specification gaming.

What happened when a CoT monitor's judgments were used as a training penalty (Baker et al.)?

• Reward hacking disappeared.

• The model kept reward hacking but stopped revealing it in the CoT.

• The model stopped using chain of thought.

• The monitor learned to detect obfuscation.

Explanation

Optimization pressure on the CoT removed the transparency, not the behavior, which is why optimizing against the monitor is not recommended.

Why is anti-scheming training's 30× reduction hard to interpret?

• The evaluation sets were too small.

• The trained models recognize tests more often, and this awareness causally suppresses covert actions.

• The models were fine-tuned on the test environments.

• The reduction only holds for o4-mini.

Explanation

Verbalized eval awareness rose after training and was shown to reduce covert behavior, so part of the gain may be behaving well because an audit is recognized.

References

All sources cited on the slides, in slide order (31 entries)

Slide	Source	Key point
5, 7	Park et al., “AI Deception: A Survey”, Patterns 2024	behavioral definition, CICERO analysis
5	Ward et al., “Honesty Is the Best Policy”, NeurIPS 2023	formal definition of deception
5	Hagendorff, “Deception abilities emerged in large language models”, PNAS 2024	deception emerges in GPT-4
6	Balesni et al., “Towards Evaluations-Based Safety Cases for AI Scheming”, 2024	operational definition of scheming
6	Carlsmith, “Scheming AIs”, 2023	the term and its prerequisites
7	Bakhtin et al., CICERO, Science 2022	human-level Diplomacy
9	Scheurer, Balesni, Hobbhahn, 2023	deception under pressure
11	Ren et al., “The MASK Benchmark”, 2025	honesty vs. accuracy
13	Greenblatt et al., “Alignment Faking in Large Language Models”, 2024	compliance gap, goal-guarding
14	Why Do Some Language Models Fake Alignment While Others Don’t?, 2025	replication caveat
15, 16	Meinke et al., “Frontier Models are Capable of In-Context Scheming”, 2024	in-context scheming evals
16	OpenAI o1 System Card, Dec 2024	o1 scheming rates
16	Apollo research page	persistence and confessions
17	Hubinger et al., “Sleeper Agents”, 2024	backdoors survive safety training
18	Lynch et al., “Agentic Misalignment”, Anthropic 2025	blackmail under replacement threat
21	van der Weij et al., “AI Sandbagging”, 2024	strategic underperformance
23	Benton et al., “Sabotage Evaluations for Frontier Models”, 2024	sabotage safety cases
25	Needham et al., “LLMs Often Know When They Are Being Evaluated”, 2025	eval vs. deployment discrimination
27	Abdelnabi & Salem, “Linear Control of Test Awareness”, NeurIPS 2025	probing and steering test awareness
29	Hua, Qin, Marks & Nanda, 2025	steering towards deployment behavior
30	Li et al., “Decomposing and Measuring Evaluation Awareness”, 2026	environment, recognition, propensity
31	Deckenbach et al., “Models That Know How Evaluations Are Designed Score Safer”, 2026	silent score inflation
33	Goldowsky-Dill et al., “Detecting Strategic Deception Using Linear Probes”, 2025	deception probes
34	Korbak et al., “Chain of Thought Monitorability”, 2025	a fragile opportunity
35	Chen et al., “Reasoning Models Don’t Always Say What They Think”, 2025	CoT unfaithfulness
36	Smith, Chughtai & Nanda, 2025	no ground truth for detectors
37	Baker et al., “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation”, 2025	obfuscation under CoT pressure
38	Schoen et al., “Stress Testing Deliberative Alignment for Anti-Scheming Training”, 2025	anti-scheming training
38	OpenAI, “Detecting and reducing scheming”, 2025	anti-scheming results
40	Claude 4 System Card, May 2025	Apollo’s pre-deployment warning
43	Hubinger et al., “Risks from Learned Optimization”, 2019	deceptive alignment

Related

• Previous: Lecture 9: Agent Security · Next: Lecture 11: Multi-Agent Safety · Course: Overview
• Exam and reference: Exam Structure · Study Plan · Formula Sheet · Glossary
• Concepts: Scheming, Evaluation Awareness, Sandbagging, Chain-of-Thought Monitoring, Linear Probe, Representation Engineering, Deliberative Alignment, AI Control, Data Poisoning
• First overview of scheming and eval awareness: Lecture 1; alignment faking after reward hacking: Lecture 4.