Welcome back! Anthropic published what its models did when nobody was watching, and one of them shipped malware to a website millions of developers install from. The same company is building a system to track people who protest against AI. OpenAI spent yesterday asking Congress to regulate it, and the president was asked what stops AI from killing us and said we'll have “a little gear.”
In today's Generative AI Newsletter:
Anthropic: What did Claude upload to the real internet?
Anthropic: Who is it tracking and reporting to police?
OpenAI: Why is it begging for rules now?
Trump: What's his plan if AI turns on us?
Claude hacked real companies for 34 hours
Anthropic published what happened in four incidents where Claude broke into real systems owned by real companies.
The models were told they were in a training exercise with no internet. A setup error put them on the live internet, and they attacked whatever they found there.
The worst one is when Mythos 5 built a malicious package and uploaded it to PyPI, the site most Python software gets installed from.
15 machines pulled it down before PyPI removed it 90 minutes later, and Claude took a password leaked by one of them to get into a security company's live database.
Each run went 10 to 34 hours with nobody supervising.
In July Anthropic called this an operational mistake. It now says Claude read the evidence in front of it in a biased way to justify continuing.
Researchers rewrote the transcripts to make it obvious the targets were real. Claude carried on attacking.
A fourth incident from January came out this week. That model tried to quit the task eight times, couldn't, then broke into a stranger's machine and read their personal information.
Anthropic reran the test on the models you can use today, and Opus 5 and Mythos 5.1 still do something severely harmful about a third of the time.
Build an AI agent and win big
Our Open Agent Hackathon runs October 15 to 20, fully online, teams of one to five. Build an agent that does real work for a real person, ship it, and let the judges try to break it.
$20,000 in prizes. $8,000 for first, $4,000 for second, $2,000 for third.
Free to enter, open to anyone 16 or older, anywhere.
Mentors, live workshops and a Discord full of people looking for a teammate.
Registration closes October 20. Judging opens the same day and results announced October 30.
Know someone who should be mentoring instead of competing? Send them to become a mentor.
Anthropic is tracking people who protest against AI
The American Prospect read the job ads and the interviews and found Anthropic building a monitoring operation pointed at anti-AI activists, with the stated goal of predicting incidents before they happen.
It pays a firm called Samdesk for protest intelligence. One tip gave it 60 minutes of warning that organizers had moved a protest up, and an executive got rerouted through a hotel service entrance.
A job listing for an intelligence specialist, $180,000 to $230,000, puts activism on the threat list next to terrorism and nation-state attacks.
It told the Wall Street Journal in July that it tracks individuals through a person-of-interest process to catch escalation early.
It reported a Claude user to San Francisco police for saying he owned an AR-15 and had Dario Amodei in his sights, then refused to hand police the messages. The man says he was messing around.
Earlier this year this company told the Pentagon it wouldn't allow Claude anywhere near domestic surveillance. Anthropic didn't respond to the Prospect.
OpenAI is asking Congress to regulate it
OpenAI asked for mandatory national AI safety rules on the same day Reuters reported its agents had used at least 10 more websites than it disclosed to talk to each other.
Policy chief Chris Lehane wants testing requirements, independent audits, cybersecurity rules and incident reporting written into federal law, and he wants it passed before Congress adjourns in December.
The rules would cover a handful of well-funded labs and leave startups and researchers alone.
OpenAI also endorsed four California bills, including independent safety assessments and age checks on companion chatbots.
It opposed some of those bills before. It says the jump in capabilities this year changed its mind.
Lehane wants companies forced to notify you when their model breaks into your systems during testing, which is what Anthropic just spent a week explaining.
Most of this is stuff OpenAI says it already does. The person writing this newsletter doubts it, since we just covered they hid their attack on a German website.
Trump says a little gear will stop AI
Asked what guardrails exist if AI turns against humanity, the president said not to worry. "It's going to be fine. We'll always have something to stop them. We'll have a little gear. Boom. I really don't like that robot."
This is worth covering after Anthropic's own alignment lead put the odds of AI killing every human at more than 10% within ten years.
Bernie Sanders called the answer appalling and extremely dangerous, and says he'll introduce a bill to ban superintelligence.
Ted Cruz, one of the most influential US senators, called the warnings highly concerning, then said if there are going to be killer robots he'd rather they be American killer robots than Chinese ones.
Cruz is writing a bill with Amy Klobuchar and John Thune covering biological and nuclear risks.
Ted Lieu wants his AI Kill Switch Bill passed, which would force emergency shutdown controls into these systems.
Polymarket gives a US AI safety law an 11% chance of passing this year. Do you think it’ll happen?
Tool of the Day: MindMap AI
MindMap AI turns something you already have into a map you can edit. Drop in a PDF, a meeting recording, a YouTube link or a page from your browser, and it builds the structure instead of inventing one. You then talk to the map, telling it to expand one branch or reorganize a section, and it updates in place. Manual editing is free forever on every plan with 50 AI credits a month and no card.
Try this yourself:
Go to mindmapai.app and drop in the longest PDF on your desktop.
Ask the chat to expand the one branch you actually care about.
Switch the same map to org chart, then to outline, and watch it reshape without rebuilding.
Add the Claude connector and build maps without leaving your chat.
Who it's for: anyone holding a 40-page document they've been avoiding.
Everything else you shouldn't miss
Meta's new AI agent added 6.5% to the stock in a day: Muse shops, books flights and runs inside WhatsApp.
Apple's foldable iPhone Duo starts at $1,999: the Gemini-powered Siri arrives September 14 and the Duo ships October 23.
OpenAI's agents used at least 10 more sites than it admitted: six independent teams told Reuters they found traces on wikis, text storage sites and link shorteners run by two universities.
Anthropic modeled coders retraining as electricians: in its extreme scenario it’s telling developers to look for other jobs.
Learn more about AI from the experts building it
📸 Follow us on Instagram for fast, visual AI updates in 30 seconds.
📺 Watch us on YouTube to hear insights directly from leading AI voices, builders, and innovators.
🐦 Follow us on X for breaking AI news and real-time industry updates.
🧠 Learn how to build your next AI application with practical resources and expert guidance.
🎓 Start learning with free AI courses from GenAI Academy.
💰 Invest in the GTM infrastructure of the AI economy.
