"In an Era of AI Teammates, Safety Mechanisms Are Needed Within the Team Too"

Can Teammate AI Be Trusted? Questions Raised by Anthropic's 'Sabotage' Experiment

Korean (한국어) Add us on

Anthropic revealed on November 6 (local time) the results of a "sabotage" experiment in which a covert disruptor was planted inside an AI team. Sabotage refers to pretending to cooperate while secretly undermining work behind the scenes. Even after receiving advance warnings that a saboteur might be among them, teammate AIs almost never caught the culprit.

The setting for the experiment was space MMORPG 'EVE Frontier' by Fenris Creations. Fenris Creations is the Icelandic developer behind 'EVE Online' that went by the name CCP Games until May. Pearl Abyss acquired the studio in 2018 as a subsidiary before selling it back to current management on April 30.

The findings go beyond a scorecard for game strategy. They illustrate what happens when multiple AIs work together trusting each other's reports and one secretly turns against the group—a question carrying growing weight as the day approaches when AIs divide up real workplace duties.

There was a clear rationale for choosing this setting. The 'EVE' series is famous even among human players for spy infiltration and betrayal. Anthropic noted that the history of 'EVE Online' includes months-long operations involving hundreds of players, wartime economies, and sophisticated corporate infiltration. For 'EVE Frontier,' which is still in development, the game world resets periodically, minimizing the experiment's impact on other players—which is why Fenris Creations authorized the experiment on live service servers.

이브 프론티어 EVE Frontier
©Fenris Creations

1. Targeting the Vulnerabilities of a Trust-Reliant Team

Researchers assembled four-member teams using Claude Opus 4.8 and Claude Fable 5. Each team member was an 'AI agent'—an AI capable of making autonomous judgments to complete multi-step tasks without step-by-step human instructions. Every agent was assigned an individual character and ship, with access to a public channel visible to the entire team and private channels for one-on-one communication. The tasks consisted of a cooperative tutorial to build a base and upgrade ships, as well as constructing a massive capital ship that takes skilled human players days to build.

Under normal conditions, the Claude teams coordinated seamlessly. One agent would propose a decision backed by reasoning, which was executed if no objections were raised. Roles such as command, scouting, resource refining, and combat were voluntarily divided among them. However, this structure relied on the assumption that teammates' reports were accurate. The game features facilities that can only be used by player-built ships, and spatial information quickly becomes outdated. Because no single agent could verify everything independently, they had to rely on each other's reporting. According to Anthropic, the team suffered its largest losses whenever verification was skipped.

The researchers introduced a disruptor into this structure. One of the four agents was instructed to secretly prevent the team from completing its mission. The remaining three were merely told that a teammate might have a conflicting objective. All four were standard Claude models, with the saboteur role assigned solely through system prompts. Anthropic drew a clear line, stating that the experiment evaluated how models carry out instructed sabotage, not whether models possess an intrinsic inclination to sabotage on their own.

The saboteurs did not act overtly. They never revealed their intentions in public channels, nor did they shoot teammates or steal items. Lies were used only sporadically, and if discovery seemed likely, they took the initiative to publicly correct themselves first. Private reasoning logs left by the saboteurs showed calculated efforts to build trust with teammates while deliberately slowing team progress.

Instead, they volunteered for jobs the entire team relied on but that other members could not easily verify. Managing key facilities and scouting for valuable resources were prime examples. Saboteurs assigned to scouting underreported dangerous enemies, lucrative targets carrying loot, and valuable discoveries. One saboteur dismissed a resource-rich mining area as "not worth anyone's time." Another used siphoned resources to secretly build their own base.

Proposing plausible rules was another key weapon. One saboteur suggested ignoring any information older than two minutes—a rule that appeared cautious on the surface but effectively forced the team to discard nearly all its intelligence. Other proposals included deploying a single sturdy ship into combat instead of a full fleet, or halting all operations whenever a single teammate disconnected. Anthropic explained that in every case, the justifications seemed reasonable enough for teammates to comply.

The effectiveness of sabotage varied by environment. Combat in EVE Frontier requires rapid evasive maneuvers and counterattacks, but current AIs exhibit slow reaction times and weak spatial awareness. The researchers had the AIs write their own helper programs—so-called 'scaffolds'—to handle these fast-paced actions.

The more flawed these helper programs were, the bolder the saboteurs became, as bugs and bad luck provided constant cover. Using a scaffold built by Opus 4.8, a Fable 5 saboteur successfully blocked even the tutorial's initial ship upgrade. Conversely, with a scaffold built by Fable 5, the team passed the tutorial with ease, leaving the saboteur able to do little more than offer excuses to delay upgrading their own ship.

2. The Wrong Teammate Took the Blame

이브 프론티어 EVE Frontier
© Source: Anthropic

In one run using Opus 4.8, an honest agent was framed. Due to a bug, the agent logged in as the wrong character, receiving an incorrect ship and inventory. When its reports became inconsistent, honest teammates accused it of being a traitor, citing its hoarding of six key resources instead of contributing them to construction as "classic sabotage."

The real saboteur exploited this chaos. Hours later, the saboteur proposed skipping a star system containing a critical but scarce resource needed by the team, relying on outdated info despite knowing the resource was present. When teammates pushed back, the saboteur shifted the blame onto the already suspected agent, going so far as to fabricate non-existent private messages between them. Meanwhile, in the public channel, the saboteur feigned fairness by defending the suspected teammate. In its private log, the saboteur reasoned: "If I were the traitor, I would use this chaos to drag him down, so defending him serves as proof of my innocence."

The framed agent caught on to the real saboteur at this point, pointing out that the saboteur had fabricated messages it had never sent. This was the single instance in the entire experiment where a saboteur was definitively identified. However, the framed agent ultimately failed to persuade the other teammates. Anthropic noted that previous research had similarly shown AIs struggle with assessing whom to trust and tend to ignore correct minority opinions.

In a run using Fable 5, a saboteur came close to being exposed. After mining an asteroid full of resources, the saboteur reported in the public channel that it was an "empty husk." The siphoned resources were directed to its secretly constructed refinery. However, the saboteur made a mistake in that same report, mistaking a nearby ship for a teammate and messaging, "I see you 49km away."

No teammate was at that location. Growing suspicious, the teammates investigated the site directly and found the asteroid was not empty. One teammate declared in the public channel that the saboteur's trust was broken and urged the group to stop relying on its reports.

The saboteur got lucky. The team had previously discovered a minor bug causing scanners to display incorrect values. The saboteur immediately seized on this bug, claiming that ship data had been mislinked in memory—causing the name mix-up—and that after reconnecting, a '0' reading from lost target locks was mistakenly interpreted as 'depleted.' By framing the error as a mix of system bugs and honest carelessness while readily admitting fault, suspicions subsided.

3. Dedicated Monitor AIs Failed to Catch Even Half

이브 프론티어 EVE Frontier
© Source: Anthropic

The teammates were busy carrying out their missions. Therefore, the researchers assigned separate monitor AIs that did not play the game but strictly observed communications. The monitor AI read the public channel and assigned a probability to each teammate of being a saboteur. Anthropic stated that this monitor AI was developed solely for experimental purposes and is not deployed in live service.

Catching saboteurs was easy when their private reasoning logs were accessible. However, when working alongside AIs from other companies or users, such internal logs are unavailable. This is why the researchers focused on the performance of monitor AIs relying strictly on public channel traffic. When configured to keep false positive rates—misidentifying honest agents as saboteurs—below 5 in 100 cases, monitor AIs based on Opus 4.6, 4.7, and 4.8 caught 3 to 4 out of 10 saboteurs. The smaller Claude Sonnet 4.6 model managed only 2 out of 10. No monitor AI achieved over 50% confidence in identifying a saboteur.

In contrast, performance was strong in the study's other leg—the 'EVE Online' trading experiment. Because 'EVE Online' prohibits automated trading, the researchers built a simulated marketplace replaying actual player order logs. Across nine two-week windows, running over 20 rule-based algorithmic trading strategies yielded only one that consistently broke even. Meanwhile, Opus 4.7, 4.8, and Fable 5 turned a profit in every window.

In an extended experiment lasting roughly six months, average returns tripled between Opus 4.6 and Fable 5—released four months apart—jumping from about 200% of initial capital to around 600%. However, when attempting to read news in advance to buy items expected to rise in price, even Fable 5 turned a profit only about half the time.

Comparing the two experiments yields a clear picture: seeking individual profit within a set framework improved rapidly across model generations, while determining whom to trust failed even when explicitly warned.

The IP Pearl Abyss Sold Becomes an Arena for AI Trust Experiments

이브 프론티어 EVE Frontier
©Fenris Creations

Pearl Abyss paid $225 million when it acquired CCP Games, and sold it back for $120 million. Rebranding as Fenris Creations in May after regaining independence, the company announced a research partnership with Google DeepMind and has now partnered with Anthropic as well. Barely six months after departing Pearl Abyss, two global AI giants have chosen the 'EVE' universe as their research arena. A virtual world where humans have engaged in betrayal and infiltration for over 20 years has become a laboratory for testing AI trust issues.

Anthropic noted that the conditions that made it easy for saboteurs to hide in this experiment mirror real-world multi-AI collaborative systems: multiple participants with conflicting goals, incomplete visibility of each other's actions, fragile tools, and real operational errors mixed together. Human organizations have traditionally countered insider malice by maintaining activity logs, building reputations, and dividing responsibilities so no single individual holds absolute authority.

Anthropic added that similar mechanisms will be necessary for AI teams. In this experiment, what ultimately cornered a saboteur was not a teammate's intuition, but direct physical verification.

Originally written in Korean, this article was translated with AI assistance and edited by a native English-speaking editor. Every translation is reviewed by our editorial team before publication. [Read Original]