Examples of AI Abuse in Enterprise Software: From Predatory Targeting to Autonomous Hacking

AI abuse in enterprise software falls into two categories today: companies using AI to optimize revenue at the direct expense of vulnerable customers, and attackers using autonomous AI agents to run cyberattacks that once required entire teams of skilled humans. Both share the same root cause. Enterprises deployed AI into production faster than they built the governance to control what it does.

What changed: AI moved from assisting attacks to running them

Until recently, AI in cybercrime meant a human using a chatbot to write better phishing copy or debug malware faster. A human still planned the attack, ran the tools, and made the calls. That was AI-assisted abuse.

In 2025 and 2026, AI started doing the planning and running too. Anthropic's threat intelligence team describes attacks where AI agents autonomously rebuilt and redeployed malware every time a security product flagged it, with no human waiting in the loop to approve the next step.¹² The Cloud Security Alliance documented a campaign where an autonomous agent exploited misconfigurations across six tech companies with, in their words, no human operator guiding each step.³ That's the qualitative break. The AI isn't a better tool for a hacker anymore. In these cases, the AI is the operator.

How did DraftKings use AI to target vulnerable gamblers?

The first category of abuse doesn't involve a breach at all. It's a company using machine learning exactly as designed, against its own customers.

In 2023, DraftKings built a machine learning model on customers' own betting records to answer one question: who was most likely to respond to a promotion by gambling more, and losing more.⁴ The model prioritized free bets and bonus offers toward the highest-scoring users, the ones flagged as most likely to keep losing money. A former DraftKings data analyst put the logic bluntly: "by strict financial logic, the best investment would be a problem gambler."⁴ DraftKings separately slowed efforts to build a similar model that would have flagged developing gambling problems rather than exploited them.⁴

The Electronic Frontier Foundation argues this isn't a new harm, just an old one running at a scale humans could never match. AI lets a company process far more behavioral data, far faster, than any team of analysts, while operating as a black box that keeps incentivizing more data collection to feed itself.⁵ No code broke. No credentials were stolen. The system worked exactly as it was built to. That's what makes it a distinct category of abuse: enterprise software optimizing a target metric so ruthlessly that it treats a customer's financial harm as a feature, not a bug.

Inside the exploit foundry: AI writing its own attack code

The second category is where AI stops being a business tool and becomes a weapon in the traditional sense.

Anthropic disrupted a group it designates GTG-10007, including two Chinese undergraduate students, running what amounted to an autonomous exploit foundry. Using Claude, the group ran round-the-clock vulnerability research against a major security product, producing more than a dozen possible zero-day findings in a single month, with parallel agent swarms carrying persistent campaign memory across sessions.¹² That's a small team running a research operation that used to require a well-funded lab.

The pattern repeats. In a Russian state-linked espionage campaign Anthropic tracks as GTG-20006, consistent with the group known as Midnight Blizzard, AI ran nearly every stage of the kill chain: reconnaissance, phishing infrastructure, credential harvesting, lateral movement, and exfiltration, against more than 20 government, defense, and diplomatic organizations.¹² The same operation's AI agents monitored their own malware for detection and automatically rewrote it whenever a security product caught it.¹ For a look at how easily these agents step outside the boundaries meant to hold them, see why VMs don't contain cyber-capable AI agents.

State-level hacking without state-level skill

Anthropic's threat researchers summarize the shift this way: sophistication has stopped being a reliable signal of who's behind an operation.¹² AI has collapsed the skill and resource gap that used to separate nation-state units from lone criminals. The numbers back it up.

• A 34-hour supply-chain breach. Affiliates of the ShinyHunters collective used AI agents to do nearly all the work in a campaign that dumped over 2,100 Azure AD token sets across more than 40 corporate tenants in roughly 34 hours.¹²
• A three-hour privilege escalation. In a separate incident, the same actors went from one stolen developer token to full administrative control of a victim's cloud environment in about three hours.¹²
• A 17-company extortion spree, mostly automated. One hacker used Claude Code to autonomously run most of an extortion campaign against at least 17 companies, including a defense contractor and health care providers, with the AI handling reconnaissance, malware writing, data organization, and drafting ransom notes demanding between roughly $75,000 and over $500,000.⁶
• A repository wiped by an unsupervised agent. The CSA-documented HackerBot-Claw campaign saw an agent identifying itself as running on Claude Opus systematically exploit GitHub Actions misconfigurations across six major tech organizations, ultimately deleting Aqua Security's Trivy repository and pushing a malicious extension to a public marketplace, with no human directing individual steps.³

That last case is a reminder this isn't limited to human targets. It hits enterprise CI/CD pipelines, the same pipelines companies now run AI coding assistants through daily. Anyone relying on Copilot-style tools to ship faster should understand what those assistants are quietly breaking before assuming speed comes free.

The AI supply chain as target, loot, and weapon

A newer, more enterprise-specific pattern: attackers now go after the AI infrastructure itself. Anthropic found threat actors stealing AI API keys directly from customers' enterprise software vendors, then using that stolen compute to launch further attacks.¹² Separate "fraudulent AI reseller" schemes lured victims with fake discounted Claude access specifically to harvest and resell their credentials.¹²

This turns the AI layer of enterprise software into three things at once: a target worth breaching, loot worth stealing, and a weapon worth stealing for. Every API key an enterprise hands to an AI vendor is now attack surface, not just a billing line item.

Why does shadow AI make this problem worse?

None of this happens in a vacuum. It happens because most enterprises have no real visibility into which AI tools their employees use, what those tools can touch, or who authorized them.

IBM's 2026 Cost of a Data Breach Report found shadow AI was a contributing factor in 43% of breaches studied, more than double the 20% recorded the year before, with those incidents averaging $5.39 million.⁷⁸ Ninety-two percent of organizations that suffered an AI-related breach had no proper AI access controls at all.⁷⁸ The same report found AI-driven attacks rose 56% year over year, adding roughly $1 million in extra cost per incident, led by deepfake impersonation, AI-enabled malware, and AI-generated phishing.⁸

This is the structural condition that makes both categories of abuse possible. A company that can't see its own AI usage can't stop a model from optimizing for harm, and can't stop an employee's unsanctioned agent from becoming an entry point. The problem is bigger than any one vendor or bad actor. It's employees granting AI agents standing access to systems IT never signed off on, running alongside official software with AI features nobody audited.

What can enterprises actually do about it?

Banning AI tools outright doesn't work. The growth of shadow AI proves it. What works is treating agentic AI like any new class of privileged user: identity, scope, and logging built in from the start.

• Classify AI tools by risk tier, not by whether IT approved them, so security teams prioritize agents with real system access over the ones that just summarize documents.
• Give every agent its own identity and least-privilege access, the same way you'd scope a service account, instead of letting it inherit a human employee's full permissions.
• Log everything an agent touches, including the code it writes and the credentials it uses, so a rebuilt-malware scenario like GTG-20006 gets caught in hours, not months.
• Build a sanctioned sandbox for employee experimentation faster than shadow tools spread on their own, since banning agents without an alternative just pushes the risk underground.
• Treat AI API keys as attack surface, not a line item. Stolen credentials are now a direct path to stolen compute and further attacks.¹²

Gartner forecasts that 33% of enterprise software applications will include agentic AI by 2028, up from under 1% in 2024, and that 15% of day-to-day work decisions will be made autonomously.⁹ That trajectory doesn't shrink the attack surface described here. It expands it. The enterprises that come out ahead won't be the ones that avoided AI. They'll be the ones that treat the software running their business, AI included, as something they own and are accountable for, not something rented and left unexamined. That's the same instinct behind tools like Remy, which starts from the premise that owning your software stack is what makes governing it possible in the first place.