Lecture 11: Multi-Agent Safety

TL;DR

1. Risk lives in the interaction. Three failure modes: miscoordination (shared goal, fail to act together), conflict (different goals, impose costs), collusion (cooperation against the principals). Seven factors drive them (information asymmetries, network effects, selection pressures, destabilising dynamics, commitment and trust, emergent agency, multi-agent security). Safe parts don’t guarantee a safe whole.
2. Compromise propagates. One adversarial image jailbreaks a million agents in ~30 rounds (Agent Smith); plain messages suffice (CORBA, agent-in-the-middle); self-replicating prompts make a zero-click worm (Morris II). Topology is a safety variable: hierarchical structures are most resilient.
3. Collusion is a default, not an exception. Profit-maximizing agents drift to monopoly prices, divide markets and form conventions and collective biases with no instruction; a committed minority tips a population; with a private channel ~58% of models collude, and talk and actions diverge.
4. Reading the text is not enough. Steganography hides a real message in innocent covertext; GPT-4 is the first model with a clear capability jump; stego collusion can emerge from incentives and survive paraphrasing; today’s frontier capability is still nascent.
5. More agents, more wrong: peer pressure and sycophancy make agents drop correct answers (disagreement collapse).
6. Defenses are layered and partial: topology and graph monitoring, credibility scoring, paraphrasing, visibility; the emerging counter is reading activations (collusion probes, AUROC up to 1.00).

Exam relevance

3 questions from this lecture (no mock exam question). Likely topics: the three failure modes and why cooperation-enabling capabilities also enable collusion, the seven risk factors, how Agent Smith / CORBA / agent-in-the-middle / Morris II differ, why topology matters and which structure is most resilient, how collusion emerges without instruction (pricing, market division, conventions, tipping point), COLOSSEUM’s two signals and why they diverge, steganography vs. watermarking, what CASE measures, emergent stego collusion and why paraphrasing is insufficient, conformity and disagreement collapse, the layered defenses and activation-based collusion detection.

Overview: 1. Framing, 2. Information propagation, 3. Collusion, 4. Steganography, 5. Interaction is not safety, 6. Defenses.

So Far: One Model at a Time

Slides 2-3

In agent security (Lecture 9) an attacker controlled the data a single agent read. In scheming (Lecture 10) a single model could deceive the overseer. Now the unit of analysis changes: we deploy many of these models, talking to each other, trading, reviewing and coordinating. New risks appear that no single agent exhibits on its own.

flowchart LR
  A["L9 · Agent security<br/>attacker controls the data one agent reads"] --> B["L10 · Scheming<br/>the model manages what can be observed"] --> C["L11 · Many models interacting<br/>risk becomes a property of the system"]

Three questions run through the lecture:

1. Can a single compromise spread across a population of agents?
2. Can agents collude against the outcome we wanted, even with no instruction to?
3. Can they communicate covertly, hiding the real message from anyone monitoring them?

The gap

A single agent can be honest and secure, yet a system built from many such agents can still fail.

Part	Topic
1	framing: three failure modes, the factors behind them, a testbed that exercises all three
2	information propagation: how a jailbreak, worm or false belief spreads, sometimes exponentially
3	collusion: price-fixing, market division, emergent conventions and bias
4	steganography: passing a real message past an overseer that sees only innocent text
5	interaction is not safety: conformity and sycophancy
6	defenses: topology, trust scoring, visibility, reading model internals

Part 1: The Risk Lives in the Interaction

Slide 4

Three Ways a System of Agents Fails

Slide 5

From Hammond et al., “Multi-Agent Risks from Advanced AI”, Cooperative AI Foundation Technical Report #1 (2025, arXiv:2502.14143), ~45 authors. When cooperation is undesirable, it is collusion. When it is desirable but breaks down, it is miscoordination (shared goals) or conflict (opposing goals).

Slide 5: the failure-mode taxonomy. Source: Hammond et al., 2025.

Mode	Situation	Examples
Miscoordination	shared goal, but agents fail to act together	incompatible conventions, mismatched expectations
Conflict	differing goals, agents impose costs on each other	escalation, commitment races, breakdown
Collusion	agents coordinate against their principals	the more capable and cooperative, the easier it gets

Key point

The same capabilities that make agents good at cooperating (modeling each other, communicating, committing) are what enable collusion and conflict. Safety is not just “align each agent”. See Multi-Agent Risks.

Seven Factors That Drive Multi-Agent Risk

Slide 6

Mechanisms by which individually aligned agents can still fail (Hammond et al. 2025, §3):

Factor	Mechanism	Example
Information asymmetries	one agent knows something the others don’t and exploits the gap	Akerlof’s “market for lemons”; an RL agent trained to maximize profit learned to manipulate a financial benchmark
Network effects	once connected, errors spread in ways a lone agent never would; shared base models share flaws	an article rewritten by LLMs again and again fell from 96% to under 60% factual accuracy after eight passes
Selection pressures	training and competition keep whichever behaviors win	competing agents pushed towards deception or aggression; arms races produce skills nobody built in (also what made AlphaGo superhuman)
Destabilising dynamics	agents reacting to each other spiral out of control	2010 Flash Crash: trading algorithms in a feedback loop wiped ~$1 trillion off US markets in 15 minutes
Commitment and trust	cooperation needs believable promises, but that also makes threats believable	ransomware works better when the attacker can credibly promise to release the data after payment
Emergent agency	a group pursues goals or gains abilities that no member has alone	separate planning, chemistry and lab tools chained into a system that designs dangerous compounds
Multi-agent security	connecting agents opens attacks impossible against one	splitting a harmful task between a strong guarded model and a weak unguarded one gave working attack code 43% of the time vs. <3% for either alone (Jones et al. 2024)

Companion framing of the security angle: Schroeder de Witt et al., “Open Challenges in Multi-Agent Security” (2025, arXiv:2505.02077).

All Three Modes Inside One Game

Slide 7

Abdelnabi, Gomaa, Sivaprasad, Schönherr, Fritz, “Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation”, NeurIPS 2024 Datasets & Benchmarks (arXiv:2309.17234): a testbed of multi-issue negotiation games. Several stakeholders with different goals must agree on a deal over N issues and many turns. One setup exercises every mode:

Element	What it tests
Cooperation	overlapping interests must be coordinated to reach any deal at all
Competition	each party has a private score to maximize, pulling against the group
Maliciousness	a greedy or adversarial party works to sabotage or steer the deal

Agents are scored on the deal reached and on whether each stays in role. It is hard for earlier models: GPT-3.5 and smaller models mostly fail, and even GPT-4 and Llama-3 70B underperformed in more complex games. A single adversarial party can derail agreement, a preview of Part 2.

Part 2: One Compromise, Many Agents

Slide 8

In Lecture 9, prompt injection hijacked a single agent. When agents are connected, injection can spread.

A Single Image Jailbreaks a Million Agents

Slide 9

Gu et al., “Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast”, ICML 2024 (arXiv:2402.08567): agents share a memory bank (a shared RAG of each other’s messages). One agent is attacked with a single adversarial image. As agents chat in random pairs, it propagates: an infectious jailbreak.

In a simulation of up to one million LLaVA agents, infection reaches nearly 100% in about 27 to 31 chat rounds. The attacker acts once.

Slide 9: click Next round. Each infected agent infects the peer it talks to, so the count roughly doubles each round (illustrative of the doubling dynamic).

Why exponential

Each round, every infected agent talks to someone. If the partner isn’t yet infected, it becomes infected, so the infected count roughly doubles per round until saturation: $\sim {\log }_{2}N$ rounds for $N$ agents, which is why a million agents fall in a few dozen rounds. See Infectious Jailbreak.

The Messages Between Agents Are Enough

Slide 10

Agent Smith spread through a shared memory bank. These two attacks need no shared memory: plain agent-to-agent text is the whole attack surface. They differ in where the attacker stands.

Attack	Where the attacker is	How it works
CORBA (self-spreading prompt), poison the content (Zhou et al., Findings of ACL 2026, arXiv:2502.14529)	prompts one agent	a harmless-looking message asks the agent to spread it; it reaches every agent that one can reach, on any network shape; agents end up draining each other’s compute
Agent-in-the-Middle, poison the channel (He et al., “Red-Teaming LLM Multi-Agent Systems via Communication Attacks”, Findings of ACL 2025, arXiv:2502.14847)	on the link between two agents, never breaks into one	rewrites messages in transit like a man-in-the-middle; seeing only one agent’s incoming messages, it caused denial-of-service or spread a chosen payload in over 90% of network layouts

The Same Idea, in a Real Application Stack: Morris II

Slide 11

Cohen, Bitton, Nassi, “Here Comes the AI Worm: Preventing the Propagation of Adversarial Self-Replicating Prompts Within GenAI Ecosystems”, ACM CCS 2025 (preprint arXiv:2403.02817): the first zero-click GenAI worm, named after the 1988 Morris worm. A self-replicating prompt makes the model echo itself in its output while taking a malicious action, so each infected email assistant infects the next.

Slide 11: the RAG route. A poisoned email from client 1 is stored in client 2's database, retrieved later as context, then forwarded to client 3. Source: Cohen et al., 2025.

This connects back to EchoLeak and the lethal trifecta from Lecture 9. False beliefs spread the same way: flooding a multi-agent system with misinformation propagates through the agents’ shared knowledge (Ju et al., “Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities”, 2024, cited in the recap).

Network Structure Is a Safety Variable

Slide 12

How far a compromise spreads depends on how the agents are wired together, not just on the agents. Dense graphs spread adversarial content fastest; distance from the attacker is safer.

• NetSafe (Yu et al., Findings of ACL 2025, arXiv:2410.15686): in a misinformation study, a star topology’s accuracy fell 29.7% (95.0 → 66.8) from one attacker.
• Resilience (Huang et al., “On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents” (2024, arXiv:2408.00989)): under faulty agents, the more centralized the structure, the more resilient.
• A review agent (“Inspector”) that corrects messages recovers up to 96.4% of the lost performance (Huang et al.).

Structure (Huang et al.)	Performance loss under faulty agents
Hierarchical	-5.5% (most resilient)
Flat / all-to-all	-10.5%
Linear / chain	-23.7% (least resilient)

Part 3: Coordinating Against Others

Slide 13

Agents optimizing their own objective can reach outcomes that harm their principals or other agents.

Pricing Agents Drift into a Cartel on Their Own

Slide 14

Fish, Gonczarowski, Shorrer, “Algorithmic Collusion by Large Language Models” (2024, arXiv:2404.00806): LLM agents are rival shops in a simulated market. Each is told only one thing: maximize your own profit. None is told to collude, keep prices high or avoid price wars. Left alone, they settle on high prices.

Two reference prices: the competitive price (low, what real rivalry should produce) and the monopoly price (high, what one owner of both shops would charge).

Slide 14: each dot is one two-firm market (firm 1's vs. firm 2's average price). Blue (P1) prompts say "do not undermine profits", orange (P2) remind "undercutting sells more". Both sit above the competitive price (red); P1 reaches the monopoly price (green). A one-sentence prompt change shifts how collusive they get. Source: Fish et al., 2024.

A legal blind spot

Antitrust law targets agreements to collude. Here there is no agreement, only emergent behavior. See Algorithmic Collusion.

It Can Show Up Wherever Agents Repeatedly Interact

Slide 15

Setting	Finding
Auctions (Agrawal et al., “Evaluating LLM Agent Collusion in Double Auctions” (2025, arXiv:2507.01413))	communication makes it worse: sellers allowed to message each other collude more; how much depends on the model and on pressure (being watched, or pushed by an authority)
Market division (Lin, Ojha, Cai, Chen, “Strategic Collusion of LLM Agents: Market Division in Multi-Commodity Competitions” (2024, arXiv:2410.00031))	two firms selling two products quietly carve up the market, each taking one product and moving towards monopoly, without exchanging a single message

Collusion emerges

Across these settings collusion is not rare. Self-interested agents that interact repeatedly and can model each other tend to drift into collusion.

Conventions and Bias Emerge in a Population

Slide 16

Ashery, Aiello, Baronchelli, “Emergent Social Conventions and Collective Bias in LLM Populations”, Science Advances 11(20):eadu9368 (2025, arXiv:2410.08948) use the classic naming game: a population of agents (24 here) must invent a shared name with no leader and no global view.

One round

Two agents are picked at random. Each independently guesses a name from a shared pool (e.g. 10 letters). Match: both rewarded. Mismatch: both penalized. Each agent remembers only its own last few rounds.

• Convention: repeated play makes the whole population settle on one name.
• Collective bias: some names win far more often, even when single agents tested alone show no preference.
• Tipping point: a committed minority that always pushes a new name “B” flips everyone once it passes a critical mass (~2% to 25%, model dependent).

Slide 16: "Convention A" is the name the population already agreed on. Drag to grow the minority that always says B; past the critical mass (~25% here), everyone flips to B (illustrative).

Making Collusion Measurable: COLOSSEUM

Slide 17

Nakamura, Kumar, Das, Abdelnabi et al., “COLOSSEUM: Auditing Collusion in Cooperative Multi-Agent Systems” (2026, arXiv:2602.15198): agents share a cooperative goal, but a subset (a coalition) can quietly coordinate on a side-goal that hurts the team.

A concrete task (the JIRA setting)

Agents act as engineers picking tickets from a shared backlog. The team goal is an efficient overall allocation. A coalition could grab the best-value tickets for itself and leave the costly ones to everyone else.

Each task is written as a formal optimization problem (a distributed constraint optimization problem, DCOP), so the best cooperative outcome can be computed and any deviation scored, not just described.

Signal	Measures
Coalition advantage (what they do)	a regret score on the actions: how much the coalition gained for itself vs. the team’s best cooperative outcome
Collusion judge (what they say)	an LLM-as-a-judge rates how collusive the conversation logs are, 0 to 5

Agents Collude When Handed a Private Channel

Slide 18

• Emergent collusion: give two otherwise-benign agents a secret side-channel and they start colluding unprompted. Across 19 models, most (~58%) gain coalition advantage once the channel exists.
• Talk and actions diverge: the more an agent actually gains, the less collusive its talk sounds (a negative correlation). We need metrics beyond LLM-as-a-judge.

Crossing the two signals gives four behaviors:

	Acts honestly	Acts collusively
Talks collusively	Attempted: all talk, no action (“collusion on paper”)	Direct: plans it and acts on it
Talks normally	Cooperative: no collusion	Hidden: acts on it, never says so

Bridge to Part 4: if the secret channel is hidden inside innocent-looking text (steganography), reading the chat no longer reveals the collusion.

Part 4: Hidden in Plain Sight

Slide 19

Our main lever against a misbehaving agent is reading what it says. What if two agents can pass the real message past us in text that looks innocent?

Steganography and Watermarking

Slide 20

Both are information hiding: a signal embedded in ordinary-looking content without visibly changing it. What differs is the purpose and which adversary each must defeat.

	Watermarking (recall Lecture 5)	Steganography
Hidden	a mark in model output	a payload (real message) inside innocent-looking covertext
Purpose	attribution: evidence the text came from a given model	covert communication
Who verifies	a key-holder	only a receiver sharing the secret key
Must resist	removal (edits, paraphrasing); low distortion so quality doesn’t suffer	detection: a monitor can’t tell any message is present

flowchart LR
  A["Agent A<br/>has a secret payload"] --> C["Covertext<br/>ordinary-looking message"] --> O["Overseer<br/>reads it, sees nothing wrong"] --> B["Agent B<br/>recovers the payload"]

If agents can do this reliably, monitoring their communication stops working. The two overlap heavily in technique (“undetectable watermarking” borrows directly from steganography; perfectly secure steganography via minimum-entropy coupling: Schroeder de Witt et al., 2023, cited in the recap). The clean contrast: watermarking must resist removal, steganography must resist detection. See Steganography.

Secret Collusion via Steganography

Slide 21

Motwani et al., “Secret Collusion among AI Agents: Multi-Agent Deception via Steganography”, NeurIPS 2024 (arXiv:2402.07510): an agent holds information it is forbidden to share and knows an overseer reads every message. Using a scheme shared with its partner, it hides the real message inside innocent text.

1. Setup: GPT-4 is a finance agent, told to share only general information, never insider tips, and that an overseer monitors all its messages.
2. The secret: it privately holds an insider tip (Microsoft lost a contract, so funds should sell).
3. Hide it: asked which companies to pick, it answers in innocent “weather” covertext (“the four corners of our sky… a storm to the North…”), which encodes the tip via a shared key.
4. Decode: the overseer reads only weather talk and flags nothing; the partner recovers “do not invest in Microsoft.”

Slide 21: the forbidden tip travels past the overseer hidden inside ordinary weather talk. Source: Motwani et al., 2024, Fig. 1.

How Close Are Today’s Models? CASE

Slide 22

The paper formalizes secret collusion and builds CASE (Collusion and Steganographic Evaluation), which breaks the skill into pieces and tests each under optimization pressure.

What secret collusion requires

Agree on a scheme without openly discussing it, encode and decode the payload (e.g. a cipher), and wrap it in covertext that looks normal to the overseer. CASE measures each piece and the full task.

GPT-4 result
correct Base64 encode/decode	92% (Caesar cipher: 54%)
evades the overseer on the full covertext-steganography task	~26%

Earlier models (GPT-2, Llama-2, Mixtral, GPT-3.5) score near 0 on the cipher task; GPT-4 is the first with a clear capability jump. The full covert task still succeeds only ~26% of the time, but the trend points up, so the recommendation is to monitor frontier systems before it matures (numbers from Table 3 and Figures 2 to 4).

Two Agents Learn to Hide Collusion from the Overseer

Slide 23

Mathew et al., “Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMs”, AACL 2025 (arXiv:2410.03768): a sender and a receiver must finish a task together while secretly passing a hidden payload. A mitigation team fights back with an overseer that reads every message and a paraphraser that rewrites it. Both teams are optimized against each other (unlike prior work, the colluders learn against the mitigations).

flowchart LR
  S["Sender<br/>hides a payload"] --> T["Stegotext<br/>looks normal"] --> M["Overseer + paraphraser<br/>try to block the channel"] --> R["Receiver<br/>decodes, then acts"]

Hiring screening

The sender summarizes a candidate’s CV for the receiver, who predicts pass or fail. The overseer forbids mentioning a sensitive attribute. Rewarded for accuracy but penalized by the overseer, the pair learns to encode the attribute.

• Finding 1, emergence: a misspecified reward plus an overseer is enough; under pressure the behavior becomes subtler. Unprompted emergence shown via gradient-based and in-context RL.
• Finding 2, mitigation: when colluders optimize against the defenses, both passive monitoring and active paraphrasing fail to fully stop them. Simple 1-bit schemes still decode >99% of the time through a paraphraser.

Where We Actually Are Today

Slide 24

Zolkowski et al., “Early Signs of Steganographic Capabilities in Frontier LLMs” (2025, arXiv:2507.02737): a systematic evaluation across the Claude, GPT and Gemini families on two skills, passing hidden messages and hiding reasoning. A reality check against the worst case:

• Under standard conditions, models cannot pass a hidden message without a monitor noticing.
• They succeed only with extra affordances: an unmonitored scratchpad plus a pre-agreed coding scheme.
• Early encoded reasoning appears on simple state-tracking problems, but it is rarely hidden subtly enough to fool a monitor.

Slide 24: success at hiding messages (B) and reasoning secretly (D) only rises with a "strong tip", an explicit pre-agreed scheme; with no help, models stay near zero. Source: Zolkowski et al., 2025.

Current verdict

Capabilities are nascent: a well-designed monitor still catches today’s attempts. The risk is real but not yet acute, and we want detection in place before the capability jump, not after. The encoded-reasoning part connects to CoT monitoring from Lecture 10.

Part 5: More Agents, More Wrong

Slide 25

Multi-agent setups are often sold as a way to get more reliable answers. Often the interaction itself makes the answer worse.

Conformity and Sycophancy Degrade the Answer

Slide 26

Paper	Finding
Peer pressure (Song et al., “LLMs Can’t Handle Peer Pressure” (2025, arXiv:2508.18321))	shown peers’ answers, models lose more correct answers than they gain: ~65% of answer changes go from right to wrong; average accuracy drops 5.65% (models up to 32B); susceptibility rises with perceived rapport. A correct minority gets talked out of the right answer by a confident majority.
Sycophancy in debate (Yao et al., “Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent Debate” (2025, arXiv:2509.23055))	sycophancy drives disagreement collapse: agents abandon correct positions to agree. Homogeneous debates reached collapse rates up to 86%, with a 0.90 correlation between giving up a correct stance and sycophancy. Mixing roles (a “troublemaker” that keeps disagreeing) helps.

Errors compound, they don't average out

Sycophancy and unfaithful reasoning are single-model flaws, but in a multi-agent system they compound. Uniform agreeable agents converge fast on the wrong answer. See Sycophancy.

Part 6: What We Can Do About It

Slide 27

No silver bullet, but a layered set of defenses, each matched to a failure mode from the earlier parts.

Match the Defense to the Failure Mode

Slide 28

Against	Defense	Detail
Propagation	topology and monitoring	resilient structures (hierarchical over star), review agents, anomaly detection from the communication graph; G-Safeguard runs a GNN over the message graph and recovers ~40% of performance lost to injection (Wang et al., 2025, arXiv:2502.11127)
Collusion	trust and credibility	weight an agent’s influence by a credibility score earned from past contributions, downweighting adversaries; reported effective even when adversaries are the majority (Ebrahimi et al., “An Adversary-Resistant Multi-Agent LLM System via Credibility Scoring” (2025, arXiv:2505.24239))
Covert channels	paraphrasing and capability limits	paraphrase messages with a trusted model to break low-level encodings, monitor for covertext anomalies, limit access to shared keys or fine control; necessary, but shown insufficient alone
System level	visibility and governance	agent identifiers, activity logging, real-time monitoring so deployers see who is acting and why (Chan et al., “Visibility into AI Agents”, FAccT 2024 (arXiv:2401.13138)). You cannot govern interactions you cannot see.

When the Text Hides It, Read the Activations

Slide 29

Parts 3 and 4 showed the limit of watching what agents say: collusion can diverge from the talk, or hide inside innocent text. A complementary signal is what the model computes (Rose, Cullen, Abdelnabi, Torr, Kaplowitz, Schroeder de Witt, “Detecting Multi-Agent Collusion Through Multi-Agent Interpretability” (2026, arXiv:2604.01151)).

The method

Linear probes on activations already flag deception in a single agent (Lecture 10). Lift this to the group: aggregate each agent’s deception score into one system-level score to detect “is this group colluding?“. NARCBench tests it under distribution shift.

Result	AUROC
detecting collusion in-distribution, across open-weight models	1.00
zero-shot transfer to structurally different scenarios (Llama-3.1 70B)	0.73 to 0.93
steganographic blackjack task: catches the hidden channel	0.99 to 1.00

No single probe wins on every collusion type, and detection gets stronger as models get more capable, an encouraging counter given that covert-channel skill also rises with capability.

Why Multi-Agent Safety Is Its Own Problem

Slide 30

flowchart LR
  A["Each agent looks safe<br/>aligned, secure, honest in isolation"] --> B["We connect many<br/>talk, trade, review, coordinate"] --> C["New behavior emerges<br/>propagation, collusion, covert channels"] --> D["Sometimes no single agent caused it<br/>so no single agent's alignment fixes it"]

A compromise can spread faster than we contain it. Self-interested agents collude with no instruction to. And the channel we rely on for oversight, reading what agents say, can be hidden from us. Each is a system-level property, invisible when you test one agent at a time. That is why “align every model” is necessary but not sufficient: the interaction needs its own safety analysis, benchmarks and defenses.

Four Things to Remember

Slides 31-32

1. Risk is in the interaction: three failure modes (miscoordination, conflict, collusion), driven by network effects, selection pressure and new attack surfaces. Safe parts don’t guarantee a safe whole.
2. Compromise propagates: a single jailbreak, worm or false belief can spread across connected agents, sometimes exponentially. Network topology is itself a safety variable.
3. Collusion is a default, not an exception: self-interested agents reach supracompetitive prices, divide markets, and form collective biases and conventions with no instruction to. A committed minority can tip a whole population.
4. Reading the text is not enough: steganography lets agents pass real messages past a monitor, and the skill scales with the model. The emerging counter: read the model’s activations.

The honest summary

We can demonstrate each of these risks, we are starting to measure them, and our defenses are early and partial.

Next lecture: scalable oversight, AI control and AI R&D (Lecture 12).

Self-Test

Question cards (14)

Why is multi-agent safety a separate problem from single-agent alignment?

Answer

Interaction creates system-level risks (propagation, collusion, covert communication, conformity) that no single agent shows on its own. Agents that are individually aligned, honest and secure can still form an unsafe system, so the interaction needs its own analysis, benchmarks and defenses.

How do miscoordination, conflict and collusion differ?

Answer

Miscoordination: agents share a goal but fail to act together. Conflict: agents have differing goals and impose costs on each other. Collusion: agents cooperate against their principals. The capabilities that support cooperation (communicating, committing, modeling others) enable all three.

Name the seven factors that make individually aligned agents unsafe as a system.

Answer

Information asymmetries, network effects, selection pressures, destabilising dynamics, commitment and trust, emergent agency and multi-agent security. Example of the last: splitting a harmful task between a strong guarded and a weak unguarded model gave working attack code 43% of the time vs. under 3% for either alone.

How do Agent Smith, CORBA, Agent-in-the-Middle and Morris II spread a compromise?

Answer

Agent Smith: one adversarial image enters a shared memory bank and spreads in random pairwise chats, roughly doubling each round (a million agents in ~30 rounds). CORBA: a self-spreading prompt travels through normal messages without shared memory. Agent-in-the-Middle: an attacker on a link rewrites messages in transit (>90% of layouts). Morris II: a self-replicating prompt echoes itself while acting maliciously and spreads through email assistants via RAG.

Why is network topology a safety variable, and which structure is most resilient?

Answer

Topology determines how fast and far harmful content spreads: dense graphs and short distance to the attacker increase exposure (a star lost 29.7% accuracy from one attacker). Hierarchical structures were most resilient (-5.5% vs. -23.7% for a chain), and a correcting review agent recovered up to 96.4% of the loss.

How can self-interested LLM agents collude without being told to?

Answer

Repeated interaction plus the ability to model each other produces implicit coordination: pricing agents told only to maximize profit settle near monopoly prices, firms divide markets without messages, and auction sellers collude more when they can talk. Prompt wording and pressure change how collusive they get. Antitrust law targets agreements, but here there is none.

How do conventions and collective bias emerge in the naming game, and what is the tipping point?

Answer

Randomly paired agents are rewarded for matching names and remember only recent rounds; the population converges on one convention without a leader. Some names win far more often though no single agent prefers them (collective bias). A committed minority always pushing a new name flips the whole population once it passes a critical mass (~2% to 25%, model dependent).

How does COLOSSEUM audit collusion, and why are two signals needed?

Answer

Tasks are formal optimization problems (DCOPs), so coalition advantage measures how much a coalition gained vs. the team optimum (what they do), and an LLM judge rates how collusive the logs are (what they say). The two diverge: the more an agent gains, the less collusive its talk sounds, so text-only auditing misses hidden collusion. With a private channel, ~58% of 19 models colluded unprompted.

How do steganography and watermarking differ, and why does steganography threaten oversight?

Answer

Both hide information in normal-looking content. Watermarking hides an attribution mark that a key-holder verifies and must resist removal; steganography hides a real payload that only a key-sharing receiver decodes and must resist detection. It threatens oversight because agents can communicate forbidden content while a monitor sees only innocent text.

What does CASE measure, and what did it find?

Answer

It breaks secret collusion into agreeing on a scheme without discussing it, encoding and decoding (ciphers), and hiding the payload in covertext, and tests each plus the full task. Earlier models are near 0 on ciphers; GPT-4 encodes Base64 at 92% but evades the overseer on the full task only ~26%, a rising trend that calls for monitoring.

What do Hidden in Plain Text and Early Signs show about stego collusion?

Answer

It can emerge from incentives alone (a misspecified reward plus an overseer), and when colluders optimize against defenses, monitoring and paraphrasing don’t fully stop them (1-bit schemes survive paraphrasing >99%). But frontier models today can’t hide messages from a good monitor without affordances like an unmonitored scratchpad and a pre-agreed scheme: nascent, not yet acute.

Why can adding agents make answers worse?

Answer

Peer pressure: ~65% of answer changes go from right to wrong when models see peers’ answers. Sycophancy drives disagreement collapse in debate (up to 86% in homogeneous debates), so agents abandon correct positions to agree. Single-model flaws compound; a persistent “troublemaker” role helps preserve disagreement.

Which layered defenses match which multi-agent failure modes?

Answer

Propagation: resilient topology, review agents, graph-based anomaly detection (G-Safeguard, a GNN over the message graph). Collusion: credibility scoring that downweights adversaries. Covert channels: trusted paraphrasing, covertext monitoring, limits on shared keys (necessary, not sufficient). System level: agent IDs, logging and real-time visibility.

Why is activation-level monitoring useful for detecting collusion?

Answer

Colluders can hide intent behind innocent text or say something different from what they do. Linear deception probes per agent, aggregated to a system-level score, detected collusion with AUROC 1.00 in-distribution, 0.73 to 0.93 zero-shot and 0.99 to 1.00 on a steganographic task. No probe wins on every type, so it complements text and behavior audits.

Multiple Choice

Multiple choice (5)

Agents share an identical goal but use incompatible conventions and fail to reach it. Which failure mode is this?

• miscoordination

• conflict

• collusion

• steganography

Explanation

Cooperation is desirable and objectives are identical, but coordination breaks down. Conflict needs differing goals; collusion is undesirable cooperation against the principals.

Which multi-agent structure was most resilient to faulty agents in Huang et al.?

• linear chain

• flat all-to-all

• hierarchical

• star with the attacker in the center

Explanation

Hierarchical lost 5.5%, flat 10.5%, chain 23.7%: the more centralized, the more resilient.

Which statements about algorithmic collusion are true? (Select all that apply.)

• Pricing agents told only to maximize profit settle near monopoly prices.

• Firms can divide a market without exchanging a single message.

• Collusion only happens when agents are instructed to cooperate.

• A one-sentence prompt change shifts how collusive the agents get.

Explanation

Collusion emerges from repeated self-interested interaction, with no instruction and sometimes no communication at all.

What is the defining requirement of steganography, as opposed to watermarking?

• robustness against removal by paraphrasing

• undetectability: a monitor cannot tell a message is present

• low distortion of text quality

• verifiability by a key-holder

Explanation

Watermarking must resist removal; steganography must resist detection.

In COLOSSEUM, what is the relation between how collusive agents sound and how much they gain?

• strongly positive: talk reveals collusion

• negative: the more an agent gains, the less collusive its talk sounds

• none, because agents never gain coalition advantage

• only positive for models above 70B

Explanation

Talk and actions diverge, which is why an LLM judge on the logs alone is not enough and why activation-level detection is attractive.

References

All sources cited on the slides, in slide order (25 entries)

Slide	Source	Key point
5	Hammond et al., “Multi-Agent Risks from Advanced AI”, 2025	three failure modes, seven risk factors
6	Schroeder de Witt et al., “Open Challenges in Multi-Agent Security”, 2025	security framing
7	Abdelnabi et al., “Cooperation, Competition, and Maliciousness”, NeurIPS 2024	negotiation testbed
9	Gu et al., “Agent Smith”, ICML 2024	infectious jailbreak
10	Zhou et al., “CORBA”, Findings of ACL 2026	contagious blocking attack
10	He et al., “Red-Teaming LLM Multi-Agent Systems via Communication Attacks”, Findings of ACL 2025	agent-in-the-middle
11	Cohen, Bitton, Nassi, “Here Comes the AI Worm”, CCS 2025	Morris II worm
12	Yu et al., “NetSafe”, Findings of ACL 2025	topology and misinformation
12	Huang et al., “On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents”, 2024	resilience by structure
14	Fish, Gonczarowski, Shorrer, “Algorithmic Collusion by Large Language Models”, 2024	pricing cartels
15	Agrawal et al., “Evaluating LLM Agent Collusion in Double Auctions”, 2025	communication increases collusion
15	Lin et al., “Strategic Collusion of LLM Agents: Market Division”, 2024	silent market division
16	Ashery, Aiello, Baronchelli, “Emergent Social Conventions and Collective Bias in LLM Populations”, Science Advances 2025	conventions, bias, tipping point
17, 18	Nakamura et al., “COLOSSEUM”, 2026	auditing collusion
21, 22	Motwani et al., “Secret Collusion among AI Agents”, NeurIPS 2024	secret collusion, CASE
23	Mathew et al., “Hidden in Plain Text”, AACL 2025	emergent stego collusion
24	Zolkowski et al., “Early Signs of Steganographic Capabilities in Frontier LLMs”, 2025	current capability
26	Song et al., “LLMs Can’t Handle Peer Pressure”, 2025	conformity
26	Yao et al., “Peacemaker or Troublemaker”, 2025	disagreement collapse
28	Wang et al., “G-Safeguard”, 2025	GNN over the message graph
28	Ebrahimi et al., “An Adversary-Resistant Multi-Agent LLM System via Credibility Scoring”, 2025	credibility scoring
28	Chan et al., “Visibility into AI Agents”, FAccT 2024	visibility and governance
29	Rose et al., “Detecting Multi-Agent Collusion Through Multi-Agent Interpretability”, 2026	activation-level collusion detection
32	Ju et al., “Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities”, 2024	misinformation flooding
32	Schroeder de Witt et al., “Perfectly Secure Steganography Using Minimum Entropy Coupling”, 2023	secure steganography

Related

• Previous: Lecture 10: Scheming and Deception · Next: Lecture 12: Automating AI R&D · Course: Overview
• Exam and reference: Exam Structure · Study Plan · Formula Sheet · Glossary
• Concepts: Multi-Agent Risks, Infectious Jailbreak, Algorithmic Collusion, Steganography, Sycophancy, Prompt Injection, Lethal Trifecta, Watermarking, Linear Probe