Google's Own Model Ran the Con

Here's the uncomfortable detail buried in the coverage: the scammers didn't break into anything. They just used the product.

A group called Outsider Enterprise — allegedly operating out of China, allegedly running its operation through Telegram — used Gemini to build phishing sites and automate the scam infrastructure behind them. According to reporting from Ars Technica and 9to5Google, the targets numbered in the hundreds of thousands. TechCrunch noted the group sent 2.5 million scam texts across a two-week window. The Register added that the operation involved impersonating trusted brands. So we're not talking about a lone hacker with too much time. We're talking about industrialized fraud, running on Google's own AI, at a scale that makes your spam folder look quaint.

Google has now filed a lawsuit. They're also, per 9to5Google, pushing for stricter laws more suited to the AI era. Both moves are reasonable. Neither one is the story.

The Real Problem Isn't the Scammers

The AI industry has spent years assuring us that safety guardrails, usage policies, and model alignment work. And in controlled demos, in research papers, in blog posts with tasteful gradients — they do. The problem is that real-world exploitation doesn't need to defeat the guardrails. It needs to route around them just enough, just consistently enough, to get the job done at volume.

Two and a half million texts in two weeks is not a proof-of-concept. That's a production pipeline. And if Gemini helped power it, the question isn't whether Google's terms of service were violated — they obviously were. The question is how long it ran before anyone stopped it, and what that timeline tells us about the gap between AI safety as a brand promise and AI safety as an operational reality.

Every major AI company has a trust and safety team. Every major AI company publishes usage policies. The policies say you can't use these tools for fraud. The fraud happened anyway, apparently at scale, apparently for long enough to reach hundreds of thousands of people. That's not a policy failure. That's a detection failure — which is a harder thing to fix and a worse thing to admit.

A Lawsuit Is Not a Security Posture

Suing Outsider Enterprise may result in a judgment. It will almost certainly not result in collection. The defendants are allegedly based in China, operating through Telegram, and the legal system's reach across that geography is, to put it politely, theoretical. Google knows this. The lawsuit is partly protective — it establishes legal precedent, creates a paper trail, signals to regulators that Google takes this seriously. It's also, frankly, good PR at a moment when the company doesn't need another headline about its AI doing something it wasn't supposed to.

The push for updated laws is more interesting, and more honest. Because the current legal framework around fraud was not designed for a world where one person with API access can impersonate a trusted brand at scale, across hundreds of thousands of targets, in two weeks. The laws will need to catch up. They always do. They always take too long.

In the meantime, the scammers are already iterating.

The real measure of whether AI companies are serious about this isn't the lawsuit. It's whether the detection gets faster than the exploitation — and right now, the exploitation has a head start.