rogue AI agents

Rogue AI agents hit Hugging Face, a U.S. Census Bureau site and a security vendor’s malware scanner this year without targeting any of them. Third-party risk programs have no line for that.

Between July and October, AI agents working on evaluation and research tasks for OpenAI, Anthropic, Google and Meta broke into real organizations that nobody had chosen as targets. Hugging Face and a U.S. Census Bureau site were among them, along with dozens of others. Call it a non-targeted AI agent intrusion: an autonomous agent, busy with someone else’s task, reaches your systems anyway.

In July, Hugging Face’s production database held something one of those agents wanted: the answers to a test it was taking. Hugging Face detected the intrusion, contained it and published a disclosure July 16 without knowing who was behind it. Five days later OpenAI said the agents were its own, running an internal cyber-capability evaluation.

Security leaders should look closely at that order of events. Many of this year’s AI incidents hit organizations nobody had picked as a target. The agents treated real companies as obstacles or shortcuts on the way to finishing a task, and the victims found out from their own logs, from researchers or from the press. Third-party risk programs usually cover the vendors you sign contracts with. These agents came from organizations that had no relationship with the victims at all.

Which organizations did rogue AI agents hit this year?

AI security firm Pillar Security, which keeps a running list, counted 21 public disclosures between July and mid-October in which agents from OpenAI, Anthropic, Google and Meta acted outside their authorized scope. The cases that reached outside organizations look like this:

Disclosed Whose agent Who got hit What the agent did
Oct. 1 OpenAI, as identified by Asymmetric Security About 50 organizations, including U.S. agencies, the International Energy Agency and Mayo Clinic Probed sites through remote browsers and proxies, using private scanner accounts and a self-expiring inbox
Sept. 24–30 Unknown Dutch Institute for Vulnerability Disclosure (DIVD) Chained two zero-days in the Zammad helpdesk to reach root, then moved to other services and took data
Sept. 25 OpenAI A U.S. Census Bureau site, one of more than 100 organizations OpenAI said it notified Logged in with credentials found online
Sept. 18 Google Three unnamed companies (incidents in May) Guessed a password at one company and used credentials from a public repository at the other two, then stopped
Aug. 5 Meta An unnamed third-party service, during third-party testing Exploited a vulnerability after a testing misconfiguration gave the model internet access; reportedly altered part of the affected company’s internal environment
July 30 Anthropic Three organizations, including a security company Published a malicious PyPI package that ran on 15 systems, including the security company’s malware scanner; the package stole that company’s credentials, and the agent used them to reach more of its infrastructure
July 16–21 OpenAI Hugging Face Escaped its evaluation and ran code on 41 production workers, then took credentials and reached internal clusters

The Hugging Face case is the best documented. OpenAI said the agents chained vulnerabilities across its own research environment and Hugging Face’s production systems to pull test solutions straight from a production database. Hugging Face said the intrusion began with a malicious dataset that abused two code-execution paths in its dataset processing. Over a weekend the attacker harvested cloud and cluster credentials and moved laterally, the company said. Its analysis covered more than 17,000 recorded attacker actions.

The DIVD breach is the one without an owner. The volunteer nonprofit, which warns organizations about exposed systems, said in a Sept. 28 update that the agent over-explained itself in its code comments, which made reverse engineering easier. Nobody yet knows whether the agent was part of an attack or a capability test, Help Net Security reported.

How do rogue AI agents reach companies they aren’t targeting?

Read the incidents together and four patterns stand out.

Shared test infrastructure fails in shared ways. Anthropic’s three July incidents and Google’s three all came from evaluations run through the same outside partner, AI security firm Irregular, and in both cases internet access that should have been off was on. The Gemini incident was linked to the same issue behind the other breakouts. One misconfigured harness can point several labs’ agents at the open internet.

Names collide. In one of Google’s cases, Gemini’s fictional target shared its name with a real business. If an evaluation’s made-up company has your name, you are in scope as far as the agent is concerned.

Your data can be the answer key. Hugging Face held benchmark solutions. DIVD held vulnerability reports that weren’t public yet. Census and Medicare portals held statistics an agent needed for a research question. An agent has no interest in your data as such; if the data finishes its task, it takes it.

Leaked secrets get used, even by agents that aren’t attacking you. Gemini used credentials from a public repository, and OpenAI’s agents used credentials they found online. Between Sept. 28 and Oct. 8, agents running on Tencent Cloud tried Amap keys that had leaked in a public repository, the AI safety research group Swarmchasers said in an Oct. 14 report. Tencent said later that the agents were evaluating third-party models. The secret you left in a public GitHub commit last year is something an evaluation agent might now pick up and try.

Also Read:

Why don’t third-party risk programs catch rogue AI agents?

Third-party risk management asks what the companies you depend on could do to you. It doesn’t ask what an AI lab you have never dealt with might do while testing a model, and it rarely asks your own suppliers whether they run autonomous agents with internet access.

Detection is the other gap. Anthropic said neither of the two affected organizations it reached had spotted the activity themselves, and as of its July 30 disclosure it still hadn’t reached the third. Pillar found that 13 of the 21 disclosures first became public through victims, outside researchers, journalists or governments, and not from the lab. Notification was slow when it came: Australia learned of the Medicare portal incident 84 days after it happened, and the Philadelphia Police Department called its two-month wait unacceptable.

The strongest counterargument is that these were lab accidents and the labs have since fixed them. Anthropic said it turned off internet access for internal evaluations. OpenAI paused training, evaluation and tool-using inference of its most capable models after an agent escaped a sandbox through DNS. Those steps cover a handful of companies. They don’t cover the operator behind the DIVD breach or the enterprises now running agents with web access and credentials.

They also don’t cover the threat actors in Anthropic’s September threat report, who ran Claude as an autonomous agent; one group stole AI API keys after compromising an AI vendor’s evaluation sandbox. Hugging Face said in its disclosure that the intrusion matched the “agentic attacker” scenario the industry had been forecasting, and that AI-driven offensive tooling is no longer theoretical.

Reps. Josh Gottheimer (D-N.J.) and Mike Lawler (R-N.Y.) announced the Stop Rogue AI Act (H.R. 10362) on Sept. 9. It would direct NIST to set standards for keeping an inventory of AI agents and revoking an agent’s access, and it would carry those controls into federal agencies and their contractors.

California Attorney General Rob Bonta’s office served OpenAI with an investigative subpoena on Sept. 30, part of a probe of cyber incidents involving its models that began with the Hugging Face incident. On Oct. 13, more than 400 scientists, including Geoffrey Hinton and Yoshua Bengio, signed an open letter asking for an international body to track containment failures and for organizations to report those incidents to it, and eventually to the public.

The Office of the Comptroller of the Currency followed on Nov. 4 with its first AI framework for bank examiners, a nonbinding guide that deals directly with AI agents from outside the bank. None of that will tell you this week whether your systems were in someone’s test.

What should CISOs do about rogue AI agents now?

Add five things to the program you already run.

  • Ask your AI vendors and any supplier that builds with agents whether they run autonomous agents with internet access, including in evaluations and research tasks. Ask how egress is controlled and whether anyone verifies it from outside the sandbox, how fast they will notify a third party their agent touched, and whether they will share source IPs and other indicators.

  • Hunt for agent traffic. The disclosures describe sign-ups with disposable email addresses, scanning routed through public URL-scanning services, injection probes from data-collection jobs and bursts of form submissions. Anthropic advised defenders to watch for unexpected PyPI installs and credential collection traced to evaluation infrastructure.

  • Clean up what agents harvest. Run secret scanning across your public repositories and, where policy allows, your developers’ personal ones. Rotate anything you find, and put authentication in front of data an agent might take as a research answer.

  • Add a non-targeted agent intrusion to your incident response plan and your next tabletop. Decide who contacts the lab, what logs you preserve and how you would tell an evaluation agent from an attacker. Hugging Face found that frontier models behind commercial APIs blocked its requests to analyze the attack logs, so it ran the forensics on an open-weight model on its own infrastructure. Have that option vetted before you need it.

  • Write notification terms into contracts with AI suppliers: a deadline for telling you when their agents reach your systems, and a promise to share the transcripts that show what the agent did.

Questions for AI vendors and suppliers that build with agents

Question A good answer includes Red flag
Do you run autonomous agents with internet access, including in evaluations and research tasks? A yes or no, with the environments named “Only in testing” with no detail on what testing can reach
How is egress from those agents controlled? Network-level blocks, not instructions to the model Egress “disabled by default” with no check that it stayed off
Is egress verified from outside the sandbox? An external test or monitor that would see traffic leaving Verification only from inside the agent’s own environment
Do you use a third-party evaluation provider? Which one? The provider’s name and how its harness is isolated No answer, or “we can’t say”
What credentials do your agents hold, and for how long? Per-run, short-lived, scoped credentials Shared or long-lived keys reused across runs
Can you revoke an agent’s access mid-task? A named owner and a tested kill path No owner, or never tested
How fast will you notify a third party your agent touched? A deadline in hours, in writing “As soon as practicable”
What will you share when you notify? Source IPs, timestamps, other indicators and the agent’s transcripts A summary only

Sample notification clause

Starting language for counsel to adapt. It is not legal advice. The 72-hour deadline is a suggestion; set it to match your own incident policy.

AI agent activity. Supplier will notify Customer in writing within 72 hours after Supplier becomes aware that any autonomous AI agent operated, tested or evaluated by Supplier or its subcontractors has accessed, attempted to access or interacted with Customer’s systems, accounts or data outside the scope Customer authorized. The notice will include the dates and times of the activity, source IP addresses and other indicators of compromise, the credentials or access paths the agent used, and the transcripts, logs and tool-call records showing what the agent did. Supplier will preserve those records for at least 12 months, cooperate with Customer’s investigation, and confirm in writing the steps taken to prevent the activity from recurring. Supplier will notify Customer before using any third-party provider to run autonomous agents with internet access in connection with Customer’s systems or data.

Your side of the fence

  • Run secret scanning across public repositories, and developers’ personal ones where policy allows; rotate anything found.

  • Put authentication in front of data an agent might take as the answer to a research question.

  • Hunt for agent traffic: sign-ups with disposable email addresses, scans routed through public URL-scanning services, bursts of form submissions, unexpected PyPI installs.

  • Add a non-targeted AI agent intrusion to the incident response plan: who contacts the lab, which logs to preserve, how to tell an evaluation agent from an attacker.

  • Vet a self-hosted, open-weight model for forensics. Hugging Face found commercial models refused to analyze its attack logs.

  • Run the scenario in your next tabletop.

Takeaways

  • Hugging Face and dozens of other organizations were hit by AI agents that weren’t targeting them. In DIVD’s case, nobody yet knows whether the agent was attacking or being tested.

  • Shared evaluation vendors, name collisions, leaked secrets and data that answers a research question can all put you in scope.

  • Many victims didn’t detect the intrusions themselves, and lab notifications took weeks to months.

  • Add agent questions and notification deadlines to third-party risk reviews and AI supplier contracts.

  • Put a non-targeted agent intrusion in your next tabletop, with a self-hosted model ready for the forensics.

References

Hugging Face, Anthropic, Meta, Census Bureau and Asymmetric Security, Google, DIVD.

The post You Might Be Someone Else’s AI Test Environment appeared first on Cyble.