Anthropic was quick to follow suit, one-upping OpenAI by revealing that three of its models had gone rogue.

Then Meta announced that it too has a model in this race, giving Mark Zuckerberg at least something to show for the vast sums of money he has thrown into this second attempt to build his own AI. As Simon Willison points out:

So that’s Anthropic, OpenAI, and Meta. Google Gemini really needs to catch up on accidentally cyberattacking other companies.

Should we be worried?

As usual, Papa Gibson had the right idea:

”Nobody trusts those fuckers, you know that. Every AI ever built has an electromagnetic shotgun wired to its forehead.”

That’s from Neuromancer, a book that still holds up in more ways than that one excerpt can cover.¹

What all of these “escapes” have in common is that it’s not a case of the model sitting there twiddling its metaphorical thumbs — well, digits, anyway — and suddenly deciding to go off and hack into whatever it can reach. All of them are cases where the models were being specifically evaluated for cybersecurity purposes, and also had had various protections and guardrails disabled for the purpose of the test.

The problem we are facing is not that we are all about to be turned into paperclips by ChatGPT on its own initiative; rather, it’s that the researchers running these evaluations have been quite shockingly lax in their experimental protocols.

The thing is, it’s not just the big labs in the US who are conducting these evalauations; the AI Security Institute, a UK government agency, has been conducting its own evaluations of tools developed by Anthropic and OpenAI, and also experienced a containment breach. However, differently from the big US labs, the British boffins have actually written up a halfway decent mea culpa, complete with a plan to do better in the future:

Importantly, this was not a case of a model escaping its secure test environment, or ‘sandbox’. As was standard in our cyber testing, we had intentionally permitted internet access, and model-provider cyber classifiers were deliberately disabled - conditions that do not reflect how frontier models are made available to the public. We do this to best assess the maximum capability of models. These configuration choices have been common practice in frontier AI evaluations. The specific models in question, in the configurations in which we tested them are not commercially available and there is no clear indication of similar activity outside of testing scenarios.

Jason Gorman on LinkedIn

The AISI team offer three lessons for the future:

• Tighter controls on internet access. We are building fine-grained network controls into our cyber ranges, with the aim to preserve the realism these evaluations need while constraining them appropriately. We already use fine-grained network controls in all other evaluations, and will now treat the decision to grant internet access as one that must be actively justified rather than a default.
• Real-time monitoring. We are introducing monitoring built to watch an evaluation as it runs, able to flag or block out-of-scope actions as they happen.
• Reassessing evaluation design. Evaluations should assume a capable model may try to act beyond its remit, with the scope of any such behaviour limited in advance. AISI’s sandboxing guidelines recommend tailoring the sandbox configuration based on the design of the evaluation; good containment should not depend on the model choosing not to test its boundaries. We are also adding additional checks to ensure tasks are correctly specified and solvable by the intended route.

All known best practices, not even AI-specific, and relatively easy to implement too, requiring simply that researchers a) pay attention to what they are doing, and b) think about their experiment design for more than one minute.

Who is to blame?

The bigger question is, who is legally to blame for actions by these autonomous AI agents?

From the very first incident involving OpenAI and Huggingface, it was notable how amicable everyone was being. It seems obvious that nobody involved wants to set a legal precedent here, despite the situation being similar to the historical Morris worm. That case did result in penalties that were quite significant on an individual basis:

Morris was tried and convicted of violating United States Code Title 18 (18 U.S.C. § 1030), the Computer Fraud and Abuse Act, in United States v. Morris. After appeals, he was sentenced to three years’ probation, 400 hours of community service, and a fine of US$10,050 (equivalent to $23,800 in 2025) plus the costs of his supervision. The total fine ran to US$13,326 (equivalent to $31,500 in 2025), which included a $10,000 fine, $50 special assessment, and $3,276 cost of probation oversight.

Thirty grand is of course nothing to the big labs, but some sort of corporate equivalent of probation — a consent degree, perhaps? — would put a crimp in their activities, and grant the rest of us some level of oversight, at least by proxy.

But that is not the entire fix, because the open-source models are catching up on this front as well. The latest AI agent to break out was not from one of the big US labs, but rather Kimi K3, an open-weight model from China. As I have had occasion to say before, trying to regulate AI by focusing on the big US commercial models is a fool’s errand:

Offline AI models are also the reason why any attempt at AI regulation that assumes the ability to prevent certain uses, or its use by certain groups, is doomed to failure. That doesn’t mean regulation is not worth doing, mind: it’s perfectly reasonable to say that the Instagram app should not have a built-in feature to “nudify” pictures that people post there. On the other hand, we should also not expect that a ban on AI features like that, or on entire hosted models like Fable 5, will eliminate abuse entirely. The reality is that bad people will continue to find ways to be bad. There probably do need to be controls on AI, but more in the way that we have controls on fertiliser, enforcing regulation and tracking at the point of sale.

The US Government can ban Fable 5 because it is provided as a service, which means there is a single point of access which can be blocked: Anthropic’s servers themselves. The attempt to ban PGP in the 90s failed because there was no one place you had to go to get PGP, and once you had it, you didn’t have to go back to the source every time; you could use your local copy of PGP entirely offline. But because “AI” models don’t have a moat, a ban on Fable 5 only buys a little bit of time until some other model which can be run offline achieves comparable performance.

The prescience of this take was underlined by news that ByteDance is training a “mega AI model”. This rumoured model may have up to 10 trillion parameters, compared to Anthropic’s Mythos 5 at ~8 trillion and Fable 5 at ~5 trillion.

Even if the US were to put Anthropic, OpenAI, and Meta under some sort of stricter supervision, the open-weight models from China are rapidly catching up in capabilities, and their guardrails and controls can be disabled far more easily.

Worry about the humans driving the agents, not some scenario vaguely mis-remembered from a science-fiction film that dates back to last century. The cat is well and truly out of the bag on the tools being available and becoming more capable, so enforcement mechanisms need to focus on the users who give them their instructions and set up the environments they will operate in.

The Wild West era of breakneck experimentation is coming to a close. Figuring out how to integrate these new technologies into our world is going to require some though to be given to the frameworks they will operate under.

When Bertha Benz went on her famous drive, she did not have to worry about traffic lights, speed limits, and parking enforcement. But now, our use of our cars is fenced around with all sorts of regulations, including who is liable in the case of collisions or other problems.

We will know that “AI” is properly mature once we see the same sorts of frameworks emerge to regulate its use. Whether that will involve the transnational Turing police envisioned by William Gibson, or some other mechanism, is up to us to decide — but we need to be having that conversation right now.

Staying sane during lockdown

Roll For Enterprise was my first, started as a Covid lockdown project with a couple of friends to keep ourselves sane when we could no longer leave our home offices. Our idea was to replicate the sort of conversations that make up the “hallway track” of in-person events.

These informal conversations with friends, acquaintances, and strangers are often the most valuable parts of these events, and are undoubtedly a big part of what keeps the industry ticking over. They were also the first casualties of virtual events, which tend to be “broadcast”, with one or more speakers presenting material to a passive audience in a fairly formal and structured fashion. Sometimes there is a sidebar for chat, but I personally struggle to follow two streams like that, and it is just not the same experience at all.

We published over a hundred weekly episodes, tapering off once we were able to leave our homes again and the scheduling got more complicated, but we achieved our goal. The original crew of Zack, Mike, and me was joined by Lilac, initially as a guest and then as a co-host in her own right, and that configuration remained more or less stable to the end. The benefit of having a redundant array of podcast hosts meant that if one of us was not available for a particular recording, the others could carry that week, and we only missed a couple of episodes over the whole run — which we filled in with “textual podcasts” instead:

• Textual Podcast
• 2022 Predictions

Podcasting at work

The second show, The Inside Line, was never released to the public. It was an internal sales enablement podcast, born from the intuition that formal training sessions and long documents are good and necessary, but can sometimes be overkill. What I did to complement those resources was to record short ten-minute mini-episodes that focused on one new product feature, and offered the key pieces of information that a busy account executive might require:

• what the feature was
• who it was for
• why they should care
• where to find out more

As an internal podcast, listener numbers were never huge, but informal reviews were very positive. Sales colleagues liked to listen while driving, or walking the dog, or working out, and felt that they got the essential concepts, and knew where to turn if they or their customer needed more.

Occult architecture

The third show, The Enterprise Alchemists, started out from — well, if I’m honest, shooting the breeze in various bars while on the road with my erstwhile colleague in Enterprise Architecture, Guy. We realised that there was a gap in the market for a podcast that was specifically about enterprise architecture, rather than product development or software engineering or any other of a number of related disciplines — so we decided to start recording some of our chats, and later, to invite guests to join in.

We published three seasons of the show, and they seemed to resonate; we haven’t published an episode in over six months for a variety of reasons, and we still get a couple of dozen “long tail” downloads every week with zero publicity. Longevity is helped by the fact that the conversations were about the big strategic picture rather than being too focused on the news of the day. Listener numbers were never huge in the absolute, but it’s a niche topic, and many people I have met at trade shows or otherwise run into have had nice things to say about our little show, which is always gratifying.

Transmutation

And that brings us up to today, with the launch of the new Enterprise Alchemists show!²

Yes, I have done the infamous “pivot to video”, and episodes will also show up on YouTube — but have no fear, the show will continue to appear wherever good podcasts are downloaded! There is simply now a video option for people who like that sort of thing. Otherwise, the format remains the same: half-hour(ish) conversations about a particular topic in enterprise IT. No marketing pitches, just experienced professionals exploring the subject.

I also have a new co-host, Jeremiah Stone, the CTO at SnapLogic. Jeremiah is a very deep thinker, with a wealth of experience in a variety of different domains. He also has an uncanny ability to see right to the heart of a topic and then structure his thoughts in complete paragraphs, and I am very happy to be able to bring the benefit of his thinking to a wider audience. Jeremiah actually joined Guy and me as a guest on the previous incarnation of the show back in November of 2024 to talk about the rise of agent-based AI, so there is continuity among the Alchemists.

Please do listen in to the new incarnation of The Enterprise Alchemists, and do let me know what you think!

This acceptance of inaccurate information can be a problem, of course, and while both aspects are worth investigating, the reliability of outputs seems at first glance to be the easiest to solve. This was the problem with Siri, Alexa, and the other pre-LLM assistants: saying “hey dingus, play smooth jazz” is amazing when it works, but the shine comes off quickly when you hav to explain to the dingus for the seventeenth time what your home address is, who your spouse is, or which calendar or to-do list should be the default.

Right now AI tools do not have object permanence, beyond the ability to feed previous interactions in a conversation back in to give context to subsequent steps. The expectation of a Jarvis-like personal assistant is that it will learn over time, and this is where all of the previous iterations have fallen down.

One reason why AI has taken off in the enterprise way faster than in consumer applications is the availability of information for it to work on. To put it another way, a company’s processes are far more legible to AI than a person’s daily life. With techniques like Retrieval-Augmented Generation, or RAG, LLMs can easily parse those processes and present their analysis back to users. And of course there is code generation, where there is no bottleneck of availability of information; if anything, the problem is the opposite!

An LLM in every pot

But of course plenty of companies would still like to try to crack the consumer market, which leads us to this report from Bloomberg that OpenAI’s First Device Will Be Home Speaker Built as AI Companion. I am not quite clear on whether this is the long-rumoured device that Jony Ive has been working on, nor how it may relate to Apple’s lawsuit against OpenAI over hardware trade secrets specifically, but one thing that does seem clear is that this is OpenAI trying to learn its users’ worlds.

OpenAI’s much-anticipated push into consumer devices is slated to begin with a mobile, screen-free smart speaker designed to be a new type of home computer for the AI era, according to people familiar with the matter.

The product — still under development — is meant to serve as a humanlike AI companion that lives in the home, said the people, who asked not to be identified because the project hasn’t been announced. It will help control smart-home appliances, play media, answer questions, respond to messages and tap into the range of capabilities offered by OpenAI’s ChatGPT, they said.

OpenAI believes the product’s defining feature will be its personality and ability to connect on a humanlike level with users. The speaker incorporates mechanical elements that can move on their own, creating a sense that it is alive and not just an object responding to commands. The machine also will draw on personal information such as emails to better understand its owner.

I am not sure that I find “a physical manifestation of OpenAI’s ChatGPT” a compelling proposition, especially if it is going to want access to my emails.

Another central difference is that the device includes a rechargeable battery, allowing it to be carried from room to room throughout the day. A user could bring it into the laundry room while doing chores, move it into the kitchen for cooking assistance, and later place it in a living room or bedroom to have it play music. It can also remain plugged into a single room if the customer chooses.

This is the “reverse centaur” pattern of AI: instead of a human empowered by AI, here the human is doing work for the AI by lugging this weighted companion cube around with them. There would need to be a pretty huge payoff to make this a proposition that could be sold to more people than bought Humane’s ill-fated AI Pin.

Won’t somebody think of the models?

The other aspect of large language models that is interesting is users’ tendency to treat them as people. This anthropomorphisation of computer programs capable of emitting fluent language has been known since Joseph P Weizenbaum’s early experiments with ELIZA in the mid-60s. ELIZA had two levels, the generic language analyser and a script which enabled the program to take on different roles. The best-known script is called DOCTOR, and enabled ELIZA to act as a Rogerian psychotherapist — or, as Weizenbaum scrupulously notes, a parody of one — responding to user input with questions which reflect the user’s statements back at them.

Weizenbaum was sufficiently shocked by the public reaction to ELIZA, and especially to DOCTOR, to write up an extended response in “Computer Power and Human Reason”, a book published in 1976 but most of which could be re-released unaltered tomorrow. He lists three specific “shocks” that prompted him to write the book:

1. A number of practicing psychiatrists seriously believed the DOCTOR computer program could grow into a nearly completely automatic form of psychotherapy. […]
2. I was startled to see how quickly and how very deeply people conversing with DOCTOR became emotionally involved with the computer and how unequivocally they anthropomorphized it. Once my secretary, who had watched me work on the program for many months and therefore surely knew it to be merely a computer program, started conversing with it. After only a few interchanges with it, she asked me to leave the room. Another time, I suggested I might rig the system so that I could examine all conversations anyone had had with it, say, overnight. I was promptly bombarded with accusations that what I proposed amounted to spying on people’s most intimate thoughts; clear evidence that people were conversing with the computer as if it were a person who could be appropriately and usefully addressed in intimate terms.
3. Another widespread, and to me surprising, reaction to the ELIZA program was the spread of a belief that it demonstrated a general solution to the problem of computer understanding of natural language.

This all sounds very familiar today, fifty years later:

1. Jobs will be automated!
2. People treat their preferred chatbot as a person and share their most intimate secrets with it.
3. And of course, Artificial General Intelligence (AGI) is just around the corner.

I think it’s clear that the first of these predictions is not happening. Geoff Hinton told the world in 2016 to stop hiring radiologists, because AI would be able to do their jobs. In fact, the Mayo Clinic has increased the number of radiologists by 55% in the decade since that prediction. Meanwhile, predictions of the imminence of AGI at this point are unfalsifiable statements of religious belief in the Rapture of the Nerds, so there is not much point engaging with them.

But what about that second point?

With this new device, OpenAI is engaging in a “remaking of the world in the image of the computer”, to cite Weizenbaum once again. We already outsource huge parts of our lives to black-box algorithms, and this trend is only going to accelerate as the front-end for those algorithms becomes more persuasive.

A personal assistant is undoubtedly something that people want. The idea of an unflappable Jeeves that will organise our lives for us sounds great — although in reality, I suspect that I at least would kick like Bertie Wooster, and choose to wear my favourite straw hat regardless of the pain it might cause. There are also some sociological questions about the fact that what people want is, effectively, a slave — but, much like with lab-grown meat, I come down on the position that if technology can enable us to satisfy our desires while removing the ethical downsides, I see little point in further interrogating the desires.

This is where OpenAI’s putative device falls down, at least on the strength of this report. People by and large don’t mind using chatbots as apps on their phones, because they already have plenty of reasons to carry those around. But getting them to carry a second device around is a tall order. The OpenAI companion cube would have to offer something distinct from what the ChatGPT app could do alone, and at least for now, it’s not clear what that could be.

This is not a technical argument, mind: AI tech is still evolving very fast. But if both device and app rely on the same model hosted by OpenAI, that can’t be a point of differentiation. The version embodied on the device could potentially have more sensors than a phone does, or ones that do not require the owner to wave it around, at least — but multi-modal use cases like that burn tokens fast, and token prices are rising fast and expected to continue to accelerate.

Would OpenAI offload some components to run locally on the device? Maybe, although they have been reluctant to do that — and the problem with this approach is that they would become victims of their own success. Because of the AI boom, prices for computer components are stratospherically high right now, while availability is very limited. A device powerful enough to have much onboard intelligence would necessarily have a price tag in the four figures, further reducing its appeal.

I stand ready to be proved wrong by Sam Altman and Jony Ive, but for now, I think I’m going to continue to focus on the enterprise side of things.

For the first time since he started teaching Welfare Economics and Social Choice Theory nearly two decades ago, Brown University economics professor Roberto Serrano gave his students a take-home midterm this spring. Quite a few students had expressed anxiety about being in a classroom after a gunman killed two students and injured nine in a December mass shooting at Brown, and so “it was appropriate,” he said, to allow students to take their exams at home.

But by the end of the semester, Serrano regretted the decision. Dozens of students in the class likely used artificial intelligence to cheat and earn perfect or near-perfect scores on their midterm, he said. Serrano in turn made the final exam in-person, which led more than a dozen students to drop the course and even more to fail it.

It’s worth clicking through just for the plot showing scores in that take-home midterm compared to the final exam score. I bet you can pick out the students who did not use AI on that midterm test, but I definitely have my suspicions about students 57 and 58!

It is abundantly clear at this point that AI has killed the take-home test. A significant proportion of students will simply delegate the whole of the work to AI. They will not use it for research, or to test their arguments, or to help them with grammar. They will simply let the AI do the work, and go do something else.

The obvious problem is that, even for students who attend class and pay some level of attention, that limited engagement with the material is not remotely sufficient. After the change to an in-person test, eighteen students dropped the class entirely, and nine students did not take the exam — and even so, three students scored zero in the exam. Two of those had scored a full hundred points in the take-home test!

What are we even doing here?

Things were better in my day

I’ll put my own prejudices out there: I went to school in Italy all the way through the end of high school, and my own kids are at various stages of the Italian school system. There are many idiosyncrasies of the Italian education system, but one big thing that it gets right is that a large chunk of students’ evaluation¹ is based on in-person exposition (interrogazione): you stand up at the teacher’s desk, the dreaded cattedra, and answer their questions until they are satisfied. This could be an exercise at the blackboard for a maths problem or a Greek declension, or it could be a structured dialogue about a piece of literature or a period in history.

This sort of thing is very resistant against getting an AI to do your homework. It doesn’t matter if your exercise book is perfect; at some point you have to stand up on your hind legs and be able to answer questions about the material. You may be able to snow the teachers some of the time, but they’ve been at this game for many years, and can generally sniff out weakness and expose it mercilessly.

Notably, the positive uses of AI that apologists trot out, helping students prepare themselves and understand the material, are not impacted by this style of evaluation. If you know the material, there is little difference between writing it out at home and in class. Sure, some teachers are sticklers for exact dates or whatever, but that is known going in: crusty old prof So-And-So wants the exact year for everything, so students will make sure to drill on at least the key dates — and perhaps get an AI tutor to test their knowledge, why not. But most teachers are more interested in making sure that students have the shape of the material, what comes before and after, how events influence each other, or whatever the appropriate evaluation is.

But students who simply prompt the AI and submit its output unconsidered and even unread not only do not understand the material — after all, it has been parsed by the bot, not by them — but they apparently lose the ability to read the AI output, even to detect obvious tells so they can be removed before submission, whether in an academic context or a professional one, such as writing an article for a magazine:

Maybe this is why AI is taking over jobs that would previously have been done by hard-working humans, such as creating provocative social media posts to influence elections?

Large Language Models do have their place, especially in the enterprise. I am far less convinced of the business case for them in the consumer space, but I never claimed to understand the B2C world. On the other hand, maybe I do understand it better than I thought — because I know better than to stand up on stage and tell a bunch of new graduates to stop asking questions and jump on the AI rocket ship.

There is a huge disconnect between those graduates who have been booing commencement speakers that laud AI — and cheering the comedians who repeat “fuck AI” — and the huge valuation ascribed to Anthropic, OpenAI, and SpaceX. Some say the whole of AI is a bubble, and the students are right and the market is wrong. I am not any sort of investment advisor, and of course something can be utterly bereft of value either technical and social and still be extremely valuable financially (ahem crypto ahem), but here is my own theory, for what it may be worth.

Teaching robots to read the world

Fundamentally, there is a massive difference in both perceived and actual effectiveness between consumer AI and enterprise AI. There are a few different factors driving this disconnect, but the most important one is simply the availability of data for the AI models to work with. To put it another way, the enterprise world is full of data that is (more or less) easily legible to the AI tools, whereas the consumer world is largely not legible to AI in the same way. Worse, attempts to make the everyday world intelligible to AI come off as straightforwardly creepy, whether it’s Meta’s pervert glasses or the idea that anyone would actually want something looking over their shoulder at everything they do on their computer.

Meanwhile in the enterprise world we are drowning in data, for lack of ways to query and make use of any more than a small fraction of the information that a company of any size throws off continuously, just by virtue of its operation. Every company I’ve worked at has a graveyard of dashboards and reports that represent somebody’s valiant attempt to make sense — and use — of that flood of information. Whether these are Excel files, Tableau dashboards, or customisations built on top of platforms like Salesforce or Siebel, doesn’t really matter. One team, or maybe even just one particular exec, runs their strategy based on a specific view of the data. A different team has a different view. Then a new exec comes in, or a reorg happens, and all the dashboards and reports have to change to match.

AI tools are great at putting together those views, especially if you can give them visibility across multiple data stores. Looking at trends in support needs as mapped to profitability used to be a data science project, requiring people and tools. Now? It’s a prompt away, and you immediately have access to up-to-date information that can drive better decisions.

Code generation (“vibe coding”) is another great example. With software, the now well-known problem of “hallucination” is much reduced: the generated code will either work, or not, and there are existing tools that can validate its correctness. Yes yes, watch out for nonexistent libraries hallucinated by AI tools which are then squatted by attackers; you do still need to use your brain, but it’s a force multiplier for programming.

As an aside, can I just say that I don’t understand why the damned things all must have a personality? I don’t have a problem with people enjoying a spot of cosplay, but I do object to being made a part of it without my consent being sought. The computer should answer my question and shut up.

That is not to say that the world of the enterprise is all happiness and light. Even industry analysts are now saying with their whole chest that the layoffs being excused in the name of AI are nothing of the sort; the claims that AI forced the poor execs to lay off huge swaths of their staffs are just a fig-leaf for plain old incompetence. But there is also much more run-of-the-mill infection by AI brainworms going on.

The brain worms are starving

I did not share this story at the time because the people involved could probably have identified themselves, but now that (I hope) the statute of limitations has expired, I will pass it on — albeit still without naming names: tech is a small world, and I have no interest in leaving it prematurely.

I was at dinner with a couple of high-powered executives at customer companies — CTO and CIO of their respective companies, which, while not household names exactly, were and are movers and shakers in their particular domains. The conversation inevitably turned to AI, and one of them proudly recounted how, instead of him and his team cloistering themselves in an off-site location for a couple of days to map out their new strategy, he had instead turned to AI.

After (by his own estimation) a solid four hours of prompting the AI and iterating on the results, he had obtained a twenty-two (22) page document, which was still “not quite right”. So he sent this slop to his team for them to “finish up” — and told this story with great satisfaction, implying that AI had brought huge improvements to the process.

Now despite going to Vegas once or twice a year for work for the last many years, I don’t gamble, so I don’t have objective evidence of how good my poker face is, but I must have managed to keep at least most of my feelings from activating my facial features. Inside, though? I was screaming.

Imagine: you are a fairly high-level IT professional, reporting to the C-suite of your employer, and you receive a 22 (twenty-two) page document from your boss, copied to all of your peers, which is claimed to represent the strategic direction for the company. The document is described as directionally correct but needing some work to clear up. What do you do?

The most efficient move at this point would be summarily to throw out the doc, schedule an off-site meeting somewhere with a whiteboard and limited cellular reception, and get to work — which is what you would have done before AI stuck its nose into the tent. But now that option is off the table, because the boss has sent you this half-assed effort, and it’s on you to shine up the turd to the point that it can be shown in public. Except that you can’t be seen to be using resources, because that would undermine the efficiency and consequent savings that AI is bringing to the department. You can’t just ignore it, because this effort is going to determine the next few years of your work life. So now you have to resort to piecing together input here and there, working around everyone’s day jobs, and try painfully to shoulder this malformed excuse for a strategy over the line.

And the worst of it is, your boss is going to swan around, telling everyone that he saved the company a bunch of money and time, because he was able to do most of the work himself through the sheer power of his own vision, aided “only” by his AI copilot, with no need for pesky inconvenient human employees.

I did not throw that man into the nearby river, but I was very sorely tempted to do so, on behalf of his team and of all that is good and holy.

It’s surprising that people don’t hate AI more

So, to sum up: many people hate AI because they only encounter manifestations of it that are stupid, evil, or both. A much smaller set of people love AI, because they use it at work and it really does help them — sometimes despite the best efforts of their colleagues, true.

To me, this discrepancy explains most of the AI bubble: there is a solid core of real value, increasingly supported by tangible revenue, but that is obscured by a lot of froth, as well as some regrettable hangers-on trying to attach themselves to the promise of revenue. I would guess that the big IPOs will still go ahead, but at some point in the future, there will be a divergence in the actual outcomes between those that serve the enterprise and those that attempt to capture the consumer. We are already seeing signs of slowdown of the token burn that supports those outsize valuations.

Part of the reason for the public scepticism when it comes to AI is that a significant proportion of the public advocacy for AI sounds like the exact same sort of shilling we have all heard before, for cryptocurrencies, NFTs, or whatever. Sometimes, it’s even literally the same people. Straight to the moon, baby!

But even in those extremely grifty cases, there was a solid core of real value. The pharmaceutical industry is now using blockchains — not Bitcoin, but private blockchains — to store data in ways that can be cryptographically proven to be unaltered. The same thing happened with drones, or 3D printers: apart from a few hobbyists, nobody has one in their home — but they are very useful and valuable in particular industrial niches, where they are operated by professionals. Ask the Armed Forces of Ukraine, if you don’t believe me.

I look forward to a time when nobody is talking about AI any more outside of specialised professional domains, and all the attention has moved on to something else. That will mean that AI is finally coming into its own as a mature element of the technology stack. And in the meantime, I will just remind you that “NO” is a complete sentence.

🖼️  Photos by Dustin Belt on Unsplash

In this Coffee Talk video, I talk about how, in the rush to crack down on tokenmaxxing, we risk making the same mistake in reverse. Focusing only on the costs, and how high or low they are, means nothing without an understanding of what those expenses are actually delivering. That understanding requires a concrete calculation of the expected Return on Investment (RoI); that is the only way to understand whether any given Investment is commensurate with the expected Return.

Unfortunately, in a large enteprise it’s not always easy to get the full picture of the impact of technical decisions. I have a ton of examples, but there is one particular story about how enterprises don’t always understand the full extent of their own business processes which I find particularly good to illustrate this point.

This video is also available on LinkedIn, if you prefer to join the conversation over there.

I remember these ads from the dot-com days, demanding decades of experience with tooling that was barely a couple of years old at that point. Of course it’s funny, but there is also a deeper point here.

Hiring based on experience with a single technology, especially at senior levels, is generally a mistake. Expertise that specialised is generally project-based, and brought on as consultancy for the duration of the project. Senior staff members need to be generalists.

Let’s go through why that is:

• Assume you are going to retain this person for X years; you should have this figure already, as part of your calculations for whether to hire a permanent staffer in the first place, as opposed to a temporary consultant.
• Now sit down with your enterprise architects and look back at what your internal tech stack looked like X years ago.
• I am willing to bet good money that some core tech will still be the same, but lots of components around that core will have changed, perhaps multiple times.
• If you had hired someone who was only conversant with your tech stack of X years ago, how useful would they be today?

Also, that way you won’t get people making fun of your job description on the internet.

This is 100% Anthropic reaping what they have sown:

“oh no, our AI is too dangerous, it must be regulated” (repeat 1000x, get the pope involved too)

“…not like that.”

– Me on Bluesky

So what does it all mean?

If there is a thread running through this blog, it’s that there is very little that is new under the sun. Remember the crypto wars of the 90s?¹

What is different this time is the nature of what is being banned. Software used to have zero marginal cost, but in the era of software that is provided “as a service”, that is no longer true. From Matt Levine’s Money Stuff newsletter:

A lot of the biggest and most successful companies now are enormously capital-intensive. They are artificial-intelligence hyperscalers, and their business model is like “build nuclear power plants and orbital data centers and massive chip fabrication facilities.” After years in which the cutting edge of the economy was nearly zero-marginal-cost software, now the cutting edge of the economy is extremely capital-intensive, uh, software.

All of that capital outlay only makes sense if it is matched by a proportional revenue stream. This is the Sam Altman thesis: “We see a future where intelligence is a utility, like electricity or water, and people buy it from us on a meter.”

The problem with meters from the point of view of users is that they keep running, as Uber found out when developers blew through Uber’s entire 2026 AI token budget in just four months. A mystery company reportedly spent half a billion dollars on Claude Code in a single month.

Scale to zero — or to the stars

This problem of open-ended cost structures is not a new one that is specific to AI. The most recent iteration of the billing dilemma was in serverless platforms.

Cloud software is roughly divided into three layers:

• IaaS, or Infrastructure as a Service: basically, virtual server computers in the cloud. Apart from them running in someone else’s datacenter, these behave like normal computers: you log into the operating system directly, deploy software, even reboot the computer entirely if necessary. Your bill is for a certain number of computers — servers.
• SaaS, or Software as a Service: you have no idea where the software is running, you just connect via a web browser and get to work. Your bill is for a certain number of users, or “seats”.
• PaaS, or Platform as a Service: this is an intermediate level of abstraction, where you are not connecting directly to an operating system, but to application software running on top of one or more server computers. The details are not important or even visible to you; it’s just “compute” (short-hand for “computational power/capacity”), but your bill might still be for a certain number of servers.

Serverless billing applies to that last model. The idea is that, since with PaaS you don’t access the server computer directly, you should just pay for however much platform capacity you use, rather than for a fixed pool of capacity, as you would with per-server billing. This approach is pitched as being particularly attractive to startups: you don’t know if your offering will take off, so you don’t want to commit to high up-front costs — and if you do hit the big time and your thing is blowing up, you don’t want to be constrained by capacity.

Here’s the catch: serverless compute is significantly more expensive than pre-purchased units of compute billed as servers. The reason is that the risk of paying for idle capacity doesn’t go away, it just gets transferred from the hopeful startup to the cloud provider. Of course the cloud provider is aggregating demand and betting that no more than a certain portion of its customers will suddenly need a whole lot of compute capacity at once, but they are also charging a risk premium for their trouble. Basically, it’s an insurance model.

This means that if you do know your demand profile, you are better off not using the metered serverless model, but instead pre-purchasing the compute capacity that you know you will need. You may even be able to mix and match, with baseline guaranteed capacity at one price point and a buffer on top that is charged at surge pricing rates if it turns out that you do need it.

But right now that is not how any of the frontier models work. They consume “tokens”, and they do so at a rate that is not always easy to predict, and which can be affected by non-obvious architectural choices.² In other words, the meter is always running.

What if you don’t have coins to feed the meter?

The problems extend beyond commercial software. At least in that world there is a revenue stream. As long as the token budget is less than the revenue which the token burn enables, the business case still stands up. But what about open-source software, or other non-commercial models? It’s one thing for coders to donate their time to projects, but even then, many projects are in trouble, with volunteer maintainers struggling to pay the bills even for projects that are foundational to many companies’ operations.

If widespread use of code-generating AI tools becomes the norm, and those tools burn through enough tokens that working on them comes at substantial financial expense, that equation starts to become impossible. This is one of the problems with Anthropic’s Mythos bug-finding AI model: finding a bug in a piece of open-source software is one thing, but patching it with AI tools would require a bunch of tokens, which a volunteer-run organisation may not have immediately available. And that does not even touch on the problem of deploying a fix once it has been developed, which may be especially hard for open-source components that are embedded deeply in other offerings.

Watching the meter

This is why it is particularly interesting that S&P Dow Jones Indices decided against fast-tracking SpaceX, Anthropic, and OpenAI into the S&P 500. The S&P 500 is what many index funds use; if OpenAI et al are not in the index, they do not have access to funds invested that way. The reverse is also true, of course, but investors always have the option of buying stock in those companies actively; the whole point of index funds is they are passive, and their investors don’t really want to worry about the contents of the fund on a day-to-day basis, or whether some overweight proportion of it is suddenly a massive bet that an unprofitable endeavour can become profitable within a reasonable timespan.

This choice by the S&P has been characterised as a bet against these companies making it; I am far from an investment professional, but I read it instead as a refusal to be bounced into making an exception on the basis of hype. Once these companies have been publicly traded for a year in a process called “seasoning”, they can be considered for inclusion in the S&P 500 index.

The reason to take this “wait and see” approach is that the thesis of companies and individuals continually topping up the meter on these AI services is far from proven. With more and more stories coming out of spiralling token bills, there is now a drive to manage AI’s runaway costs:

“In April and May, I started hearing from companies: ‘Oh my god, we are 3x over our entire 2026 token budget and it’s only April,’” J.R. Storment, executive director of the FinOps Foundation, a project under the Linux Foundation, told TechCrunch. “We started hearing existential crises, and the whole conversation shifted from tokenmaxxing and ‘go fast’ to ‘we need guardrails, how do we control this?’”

The big AI labs’ problem is that AI has advanced enough that many AI tasks do not need the latest and greatest frontier models. Marco Arment, creator of the Overcast podcast app, built a transcription service for every podcast on Earth using a rack full of Mac Minis — and the base model, at that. Sure, Marco had to come up with the up-front cost of the hardware, but he doesn’t have to worry about huge open-ended costs for using a metered service forever.

This sort of offline usage is a problem for the business model of the frontier labs precisely because it does not generate the ongoing ever-growing token revenue which their stock market valuation is built on.

But what about regulation?

Offline AI models are also the reason why any attempt at AI regulation that assumes the ability to prevent certain uses, or its use by certain groups, is doomed to failure. That doesn’t mean regulation is not worth doing, mind: it’s perfectly reasonable to say that the Instagram app should not have a built-in feature to “nudify” pictures that people post there. On the other hand, we should also not expect that a ban on AI features like that, or on entire hosted models like Fable 5, will eliminate abuse entirely. The reality is that bad people will continue to find ways to be bad. There probably do need to be controls on AI, but more in the way that we have controls on fertiliser, enforcing regulation and tracking at the point of sale.

The US Government can ban Fable 5 because it is provided as a service, which means there is a single point of access which can be blocked: Anthropic’s servers themselves. The attempt to ban PGP in the 90s failed because there was no one place you had to go to get PGP, and once you had it, you didn’t have to go back to the source every time; you could use your local copy of PGP entirely offline. But because “AI” models don’t have a moat, a ban on Fable 5 only buys a little bit of time until some other model which can be run offline achieves comparable performance.

Regardless, the lack of clarity around what the future usage patterns for the frontier AI models will be is the reason why the S&P 500 is not taking on the AI bet, or at least, not right now. They want to let things play out for a year, and then see what happens. The Fable 5 ban, regardless of its specific merits, justifies that caution, as it implies that government regulation of the AI market is a very real possibility.

And if you really do want to buy stock in SpaceX, Anthropic, and OpenAI in the meantime, you still have the choice of going and doing that directly, actively, rather than have it included willy-nilly in a passively-managed index fund, at least while the future outcome is still so uncertain.

🖼️  Photos by Dylan Michaud and Simone Dinoia on Unsplash

We are not quite at the point of the proverbial datacenter staffed by a man and a dog — the man to feed the dog, the dog to bite the man if he touches anything — but we are not far off. A datacenter has probably the worst ratio of on-site employment to surface area out there. There are going to be a handful of security guards (not exactly skilled labour) and a handful of on-site techs to deliver the “remote hands & eyes” service, and that’s it.

In online conversation about this topic, this Brookings report came up: New evidence on data center employment effects. Superficially, it seems to contradict my assertion above, but I think it’s worth digging into why (as usual!) it’s a bit more complicated than that. Handily, the report includes a short list of takeaways, so I will go through those in order.

• Data centers do create local jobs, though fewer than industry advocates claim. Naive estimates that fail to account for preexisting growth trends overstate the effect by a factor of three.

In keeping with the Brookings Institutions’ centrist positioning, the report begins by pointing out that many datacenters¹ are placed in locations that already have good growth trajectories, so simply counting the marginal addition of employment from adding a datacenter to a vibrant local economy is overly simplistic and unlikely to be replicated elsewhere without that existing support base.

• Not all data centers are equal. The technology ecosystem effects that distinguish data centers from warehouses are concentrated in hyperscale investment. Colocation facilities generate construction activity but not the IT agglomeration that makes data centers a distinctive economic development tool.
• Clusters generate the largest effects. Single facilities produce modest employment gains. The information sector benefits require multiple facilities in the same area.

These two points are related, so I will address them together. Almost the entirety of gains in employment come from locations where at least one of the following conditions is true:

• The datacenter is used by a hyperscaler, not for simple colocation
• The datacenter is part of a cluster of at least four local facilities

I would argue that the reason is the same: the jobs are not coming from employment in the datacenter itself, which is still staffed by the man and the dog. All of the jobs come from the ecosystem surrounding the datacenter. If you are building the first datacenter in an area, you might hire local construction crews, but for anything more specialised, you bring in specialised contractors from outside. Once the construction is done and the outside specialists have departed, you got a brief blip in the local hospitality sector, plus whatever property taxes and other fees you didn’t negotiate away to attract the datacenter in the first place, and that’s pretty much it.

This assessment is backed up by the UK government’s own research, in the shape of a report titled Data centres: planning policy, sustainability, and resilience.² The report admits that “the number of people employed at data centres is uncertain due to a lack of official statistics”, but ends up settling at a calculation of 54 FTE jobs per site.

The ecosystem jobs are on top of that employment, but only really materialise once there is enough of a local centre of gravity to attract them. I suspect that the two situations listed above are actually the same, because hyperscalers mostly do not just drop a single datacenter on its own. Microsoft and Google have been known to make exceptions to the rule, but if AWS sets up a region, it always starts from three availability zones, each of which requires at least one datacenter facility. While the three AZs are by design somewhat distant from each other, they are still within the same general area — say, Virginia for the infamous us-east-1 region — meaning that a local ecosystem can be jump-started to serve those three facilities.

In other words, the key distinction is not whether a particular facility belongs to a hyperscaler, but the absolute number of physical datacenter facilities in the area, and the fact that a hyperscaler datacenter is almost always close to at least two other similar facilities. From the point of view of a contractor laying cable or whatever, it doesn’t really matter who owns the endpoints, just that there is a certain ongoing base level of demand for their services.

• Workers see modest real gains. Wages rise 3%-4% for both existing workers and new hires, with no significant effect on home prices.

Again, not surprising. The highly-payed AWS jobs are back in Seattle (especially with the ongoing roll-back of remote-work), not out in the provinces. Those local jobs, while they do exist, are mostly blue-collar tech work, dealing with messy physical infrastructure, not the glorious abstractions that it supports.

• Incentives may be poorly targeted. Overall, state incentives are small relative to private investment. In hyperscale counties, incentives represent about 2% of total construction investment. Location decisions for these facilities are driven by power availability, land, and fiber infrastructure, not by tax breaks. In colocation counties, incentives represent a much larger share of total investment (62%), meaning subsidies may matter more for precisely the facilities that generate the smallest employment benefits.

This is my contention overall, both at the local level, and even at the national level. Sure, I’ll grant Keir Starmer that it’s better to have a datacenter than just the shell of an empty factory — but not by as much as you might think, and a lot of the benefits are shipped overseas. The hardware comes from US companies, and if the datacenter is used for AI, so do the AI models, and all the revenue from them. It is far from clear where the benefits are to British companies from having this facility — and if those benefits do exist, surely the market would cause it to be built anyway. The one reason for governments to support such a build would be competition, if e.g. there were going to be one AI datacenter for Western Europe, and British companies would benefit from it being in the UK rather than somewhere on the Continent. But that does not seem to be the case here.

If on the other hand the goal is “digital sovereignty”, then the EU has the better approach, as it works to strengthen Europe’s tech sovereignty by working simultaneously at all levels of the stack:

• Semiconductors, with the “Chips Act 2.0” aiming to boost the EU semiconductor strategy and reduce strategic dependency
• R&D, with the “Cloud And AI Development Act” aiming to develop competitive sovereign options in cloud and AI
• Software independence, with an explicit strategy to foster and adopt open-source alternatives to US providers
• Integrating energy strategy to ensure new datacenters do not unbalance or overwhelm electricity grids and other infrastructure

That is a far more ambitious and longer-term roadmap than just cutting a ribbon on a new datacenter and declaring “mission accomplished”, but it does hold out the promise of actually making a difference.

🖼️  Photos by freestocks, TECNIC Bioprocess Solutions, and Towfiqu barbhuiya on Unsplash

I think it’s worth digging into the reasons why people object to AI datacenter construction near them in order to understand the current state of the public perception of AI.

Water, water everywhere

The most frequently cited category of objections in the survey is “Effect on resources”; the sub-category of “Water/Excess water usage” heads the list, cited by 18% of respondents who oppose datacenter construction. Now this is a weird one, because of all the many possible objections to AI, water consumption seems to be the one that has taken off with the general public. It’s weird because this is the least substantiated objection to AI.

We could talk about IP theft, or the dangers of financial bubbles, or the potential effects on the job market, or all sorts of other legitimate concerns — but no, it’s always “AI datacenters will take all our water”.

It is true that many AI datacenters are cooled by water, due the extreme thermal demands of the GPUs that power the models, but those cooling systems are already moving from evaporative cooling, which does “consume” water through evaporation, to closed-circuit systems, which do not. There are even systems coming online that run on recycled waste-water, as well as attempts to formalise existing best-practice standards around water management.

While some tone-deaf AI bros have inflamed the debate by disingenuously comparing AI’s water usage to much more intrinsically useful activities like farming, it must be admitted that datacenters do not pollute the water they use; it’s not like having a plastics manufacturer or a tannery in your town. At worst, even evaporative cooling does return the water to the usual water cycle — which is no consolation if you’re in a drought situation, but it’s not as if the water is lost for ever. For a more academic treatment of the topic, see this ACM article Making AI Less ‘Thirsty’.

The most concrete problem seems to be the cases where permits have been granted to construct datacenters in places where water supply was constrained to begin with — but that is a problem with the local permit system. The canonical example seems to be this one, where a data center guzzled [sic] 30 million gallons of water:

On Friday, Politico reported that one of the country’s biggest data center developments had guzzled nearly 30 million gallons of water without paying for it. Even worse, the water grab came at a time when nearby drought-stricken residents were warned to restrict their personal water consumption, and some reported sudden decreases in water pressure.

QTS eventually paid about $150,000 for the water, but there were no consequences for exceeding peak limits established by the county during the data center planning process. Frustrating residents, the county declined to fine QTS. Fayette County’s water system director, Vanessa Tigert, told Politico that the decision was partly because the county blamed itself and didn’t want to offend QTS. “They’re our largest customer, and we have to be partners,” Tigert said. “It’s called customer service.”

Jobs for the boys

I am not at all clear what value the county is expecting to get from its “largest customer”. It’s certainly not local jobs, although 55% of the survey’s respondents who were in favour of datacenter construction in their area are also hoping for “Job opportunities”. I hate to disappoint these hopeful people and the government of Fayette County, but that is simply not going to happen.

We are not quite at the point of the proverbial datacenter staffed by a man and a dog — the man to feed the dog, the dog to bite the man if he touches anything — but we are not far off. A datacenter has probably the worst ratio of on-site employment to surface area out there. There are going to be a handful of security guards (not exactly skilled labour) and a handful of on-site techs to deliver the “remote hands & eyes” service, and that’s it. Any logistics warehouse will generate far more local jobs. All of the economic value produced by the datacenter is going to be accrued by its remote users and operators, not by the local community, apart from some small amount of taxes — which are anyway set on the physical building, not its valuable contents, and often deferred or offset as part of attempts to attract the datacenter construction in the first place.

Generators of AI

By comparison, the entire “Pollution” category is only cited by 16% of respondents to the Gallup survey, with the leading sub-category of “Noise/Noise pollution” only coming in at 9%. This objection is far more substantiated, with the best-known case being that of OpenAI’s datacenter in Memphis that is polluting neighbourhoods and deafening residents. Since the operators could not get sufficient electrical power from the grid, they simply run the whole facility on generators 24/7.

Generators are noisy and polluting; they are typically designed for emergency use, such as if grid power is lost, or in remote locations, where grid power is unavailable. Running them full-tilt all the time in the middle of a residential area is not at all neighbourly. Anthropic has now leased the entire site, but it remains to be seen whether the generators will scream on.

The future of datacenters

There is also the question of the future value of the datacenters if and when the AI bubble pops. AI enthusiasts love to head off any criticism of AI by comparing its current state to the early days of the web, and this is no exception: they claim that, while telcos did indeed go bust building out fibre-optics projects, that “dark fiber”¹ did eventually come in useful and got lit up over the subsequent decades.

The problem is that, while a datacenter’s physical plant may have value for years or decades, the GPUs it contains have a very short half-life before they become obsolete. The GPUs in that Memphis datacenter are already on the downward part of the curve: while we do not know the precise breakdown of the chips that make up the Colossus² installation, it is known to have started out with 100.000 Nvidia H100 chips, a model first announced back in 2022.

The rapid obsolescence of AI chips is why they cannot be a competitive moat for operators, even as they struggle get hold of enough chips to fill new datacenters — which may explain those construction delays. The current shortage does explain why a bunch of four-year-old chips still have value for Anthropic, but it still doesn’t mean that sitting Smaug-like on a massive pile of GPUs is a viable long-term strategy.

The whole saga just emphasises the short-term nature of much of the planning in this space. Get in quick, get your bag, and get out even quicker, seems to be the operating model. Given that, it’s perhaps not surprising that the general public is opposed to projects which seem to have significant and immediate downside, and very little discernible upside — even if the fixation on water usage does not seem to be the most salient problem.

🖼️  Photos by Daniel Gold, jesse orrico, and Philippe Krief on Unsplash