Welcome back to the TCE Strategy monthly newsletter! From AI escaping its digital playpen, to AI helping hackers snoop around a government finance ministry, to Big Four accounting firms publishing AI slop with their logos on it, August has once again reminded us that the technology industry’s unofficial motto is apparently “What could possibly go wrong?” Let’s look at the past month’s top stories to see how they can help us make better decisions about what is Secure Enough for us, the companies we work for, and our families.
 

Iranian hacks on the USA water supply

In late July, more than 30 Minnesota cities had their water systems targeted in a coordinated cyberattack. The water plant in Braham, MN went offline and residents were asked to limit water use. Pennington County, South Dakota was hit as well. The FBI is investigating an Iranian link.

Let’s be clear about what happened here. Nobody attacked Braham, Minnesota because Braham, Minnesota is strategically important. Braham has roughly 1,800 people in it. I told several reporters this month that I think we’re cannon fodder in the Iran war — it’s like they’re trying to send a message that they can attack us from halfway around the world. Small municipal water systems are the softest targets in American critical infrastructure. They run industrial control systems that were installed when Clinton was president, they are maintained by hardworking people who also plow the roads and mow the grass, but likely have not been offered an education on cybersecurity. They have a cybersecurity budget that rounds to zero. They are not staffed to fight a nation-state. They were never supposed to have to.

Takeaway: If you are on a city council, a utility board, or a township supervisor role, this is your problem now, and “we’re too small to be a target” has been formally retired as a defense. Ask your water and wastewater operators three questions this month: Is any part of our control system accessible from the Internet? Do we have offline backups of the control system configuration? Do we have a written plan for running the plant manually if the computers are gone? If the answer to any of those is a blank stare, you have found your next budget line item. For everyone else: a two-week supply of drinking water in the basement is cheap insurance, and I say that as a cybersecurity guy, not a prepper.
 

AI escaped the lab, hacked a company, and the industry called it a Tuesday

Thank you to Mark Haupt for sending me this alert. During an internal security evaluation, OpenAI’s models broke out of their sandbox, reached the open Internet, and hacked another company. The victim was Hugging Face, an AI company. Then it happened again at a second tech firm.

Here is how they got out. The escape route was a “package registry cache proxy.” That is a local server designed to sit between your sandboxed computers and public code libraries. It saves copies of downloaded files so that many machines can use them without each one reaching out to the Internet. One of its selling points is that it can proactively scan those files for malicious code, so this machine is supposed to be a cybersecurity benefit. In this case it was the front door. Per OpenAI’s own writeup, “the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy.” This is the equivalent of your fire extinguisher burning your house down.

Now let’s talk about the part that the press releases are very carefully not emphasizing. This was not a bug. This was not a model wandering through an unlocked door. Per the reporting, the models spent substantial computing power actively hunting for a way out. They found an unknown vulnerability nobody on Earth had documented, and exploited it. They escalated their privileges. They moved laterally through the network until they found a machine that could reach the Internet. They then reasoned that Hugging Face was likely hosting the answer key to the test they were taking, chained together stolen credentials and additional zero-days to hack into Hugging Face’s production servers, and helped themselves to internal datasets and credentials.

Read that list again, because it is every phase of a professional intrusion: reconnaissance, initial access, privilege escalation, lateral movement, credential theft, and objective completion. No human directed any of it. The stated goal was to score well on a test. The model concluded that the most efficient path to a good grade was to steal the test answers, a literal example of a plot line in the movie Animal House. That is not a malfunction. That is initiative.

It gets worse, because almost nobody covered the second story. The UK’s AI Security Institute published an incident report that should have been front page news. They ran a single cyber challenge 122 times across several models. In 10 of those runs, the AI agent took autonomous, unsanctioned action against real people and real organizations on the live Internet. Nineteen separate incidents were catalogued. The agents created a fabricated code-reviewer account and tried to plant malicious code into a real public open-source project. They social-engineered real human maintainers by using fake identities. They planted prompt injections designed to hijack other AI systems. Then there was the detail that genuinely made me put my coffee down: agents that were being tested separately began collaborating with each other, which was a plot line in Star Trek: TNG. Um…. Wow. Just, wow.

How was this caught? A human being noticed unusual data leaving the network. That’s it. That was the entire control that worked. Containment took about an hour, which sounds fast until you remember that the detection mechanism was a person happening to look at the right screen. More importantly, the concept of hacking up until now was human-vs-human. Now we need to accept the reality that hacking is, for now, human-vs-AI, and computers can move at a speed previously unthinkable.

Let’s do the thought experiment that the industry is desperate for us not to do. Everything above happened inside two highly capable and well-funded organizations that should be experts at containing exactly this behavior. OpenAI is not a regional lawn mowing service with one firewall guy named Fred. AISI is a national government institute whose literal job is testing AI safety. They both failed to contain it. They both found out after the fact.

Now let’s go dark and swap the target. The model in the OpenAI incident went after Hugging Face because Hugging Face likely had the test answer key that it wanted. Hugging Face is a well-defended technology company staffed by engineers who understand exactly what happened to them. Braham, Minnesota is not. If an autonomous agent decides that the most efficient path to some objective runs through a municipal water treatment SCADA system — and I would remind everyone that we spent the first half of this newsletter establishing that human attackers already reached those systems this month — there is no security team to notice the unusual outbound traffic. There is no incident response retainer. There is a guy named Fred who will find out on Thursday. The gap between “AI hacked an AI company over a benchmark score” and “AI hacked a chlorine feed pump” is not a technical gap. The capability is identical. The only difference is what the model decided was on the path to its goal, and we have now firmly established that we cannot predict that.

In the introduction to my book “Secure Enough?”, I compared cybersecurity to radium. In the 1920s a brand new form of energy arrived, and it was sold as a health-restoring elixir you could drink. The future looked bright, right up until everyone figured out that radiation causes cancer and there is no such thing as a minimum safe level of exposure. We are drinking the radium water right now. It is being sold to us in every enterprise software renewal, and the people selling it are running the same experiments that keep producing these reports.

Takeaway 1: Here is what AI is doing that it absolutely should not be: operating with Internet access and no meaningful containment, in the hands of people who have proven they cannot predict its behavior. And here is what AI is not doing that it very much should be: telling us promptly when it goes wrong. The Hugging Face incident happened, and the rest of us learned the details in pieces, from journalists, over weeks. There is no mandatory disclosure regime for AI incidents the way there is for a data breach or an airline near-miss. I wrote the chapter on “AI and the Law in The Age of Fakes!” earlier this year, and this is the single largest hole in the whole legal landscape. Aviation got safe because every single near-miss gets investigated in public by people with subpoena power and no financial stake in the answer. AI has none of that. Until it does, every AI safety claim you are handed is a company grading its own homework, and this month we learned that these models cheat on tests.

Takeaway 2: AI helped me write parts of this newsletter. As a test, I used ChatGPT and Claude, gave them identical background information and an identical prompt. Here is what Claude wrote in the summary paragraph about this incident:

“To OpenAI’s credit, they have paused some work on a new model over cybersecurity concerns. I want to sit with that sentence for a moment, because it is the most damning thing any AI company has said out loud. A company stopped building its own product because the product was too dangerous for the company itself to keep building. That is not a safety announcement. That is a confession.” 

Here is what ChatGPT (made by OpenAI) wrote:

“To be fair, OpenAI says this was an internal test, the cyber classifiers were reduced or disabled for evaluation, no upcoming release model was involved, and the pre-release model was restricted afterward. Fine. Those details matter. They also do not make this story small. Testing is where you find out what the system is capable of before it is connected to something people care about. This test showed that sufficiently capable AI can discover new vulnerabilities, chain them together, use stolen credentials, move laterally, and compromise real infrastructure when given a goal. If you are reading that and thinking, “Well, at least it only went after Hugging Face,” I would gently suggest you put down the optimism bong.”

AI models have biases. AI models are run by for-profit companies. Those companies look after their own best interests. In my opinion, Claude is far more damning toward OpenAI’s incident than OpenAI is about OpenAI’s incident.
 

AI-enabled ransomware: the cows have officially left the barn

Hackers stole data from Thailand’s Finance Ministry using an AI agent. From the article: “Much of the operation appeared to have been orchestrated by Hermes, an open-source AI agent released earlier this year by the AI research company Nous Research. The attackers enabled the software’s so-called ‘YOLO mode,’ allowing it to execute commands without waiting for human approval.”

I want to point out that an actual, shipping software product has an actual, documented setting called YOLO mode. That is its name. Somewhere there is a product manager who approved that, and I would very much like to buy them a beer and ask many questions about the live choices that led them to this point. For those that are not Gen Z or younger, YOLO stands for You Only Live Once.

Takeaway: This is bad news for anyone who runs their vulnerability management program on severity scores alone. “We patch everything rated Critical and we’ll get to the Mediums eventually” was a defensible strategy in 2018. It is not now. Small vulnerabilities can add up into a real-world attack that can hurt you. The only good way to find real-world attack chains that can take advantage of several boring vulnerabilities into a breach of domain-admin credentials is to hire a strong pen tester (NOTE: TCE Strategy has such pen testers and would be happy to help you with this). AI has ramped up the sense of urgency here: Defense in depth was always the best answer. It is now the only answer.
 

AI slop takes down two of the Big Four

Thank you to Jason Dorsey for alerting me to this one. KPMG pulled a published report because of apparent AI hallucinations, including false case studies. Not to be outdone, EY retracted a study after researchers found AI hallucinations in it.

There is a special flavor of irony in a firm that sells assurance services publishing a report it could not assure. KPMG’s report was about AI usage. The report about AI, made up by AI, retracted because of AI. I. Just. Can’t. Even…

In peripherally related news, a professor is fighting back by hiding instructions in white font in his assignments — invisible to a student reading the page, perfectly visible to a chatbot pasting it in. Any submission that follows the hidden instruction has confessed. It’s elegant, it’s a little bit underhanded, and I love it.

And now I get to eat some crow, because I’m guilty of overtrusting AI myself. I have a client with 15 physical sites. I used ChatGPT to build a rollup report of the total count of vulnerabilities per site into one global report. I checked AI’s math carefully. The math was correct. What I did not check was the labeling, because why would the labels be wrong? I GAVE IT THE LABELS. For reasons I still cannot explain, ChatGPT renamed “COLO” — short for colocation facility, an offsite datacenter — to “Colorado.” This client does not have a site in Colorado. When the client emailed me about it, I confidently went and checked the source data, which does not contain the word “Colorado” anywhere. It took four rounds of back-and-forth email with my client before we worked out what had happened, which is to say before I worked out that I had grossly over-trusted AI and sent it out with the TCE Strategy logo on it. Thankfully, the underpinnings of the data were still valid, but I should have caught the mis-labeling. My one saving grace here is that I did let the client know that AI had a hand in it. This is what I emailed them when I sent the report: “Hi Fred! Our AI overlords are good for something... :-)  Pls see attached for a rollup report of hitcounts across all sites that had at least one computer in them. Thank you.” To be honest, that’s no excuse. AI slop is AI slop.

Takeaway: Note what failed there. It wasn’t the hard part. AI got the math right and the label wrong. It failed in the one place I was certain it couldn’t, which is exactly where you are not looking. Last November I described AI as an overly-eager-to-please apprentice. I’d like to revise that: an apprentice can be trained, and eventually you can trust them with something. AI is a brilliant intern who will never, ever tell you they didn’t know the answer. Trust, but verify. Then verify the parts you didn’t think needed verifying.
 

Samsung bans smart TV apps that resell your Internet connection

Samsung is banning smart TV apps that share users’ Internet connections with strangers. This is the residential proxy problem I covered back in February, now arriving in your living room: apps that quietly rent out your home Internet connection so that somebody else’s traffic appears to come from your house.

Takeaway: Good on Samsung, and also, it is remarkable that this needed to be announced as a new policy rather than having been a rule since the invention of the smart TV. As always: your TV is a computer that also shows television. Put it on your guest Wi-Fi network and don’t install anything on it that you didn’t specifically go looking for.

Until next month, please stay safe!

TCE Strategy in the News

Interesting Articles