In fairness to the users, one reason why they treat the bots’ output as the pronouncements of an expert is that the bots have been programmed to claim the role of a human expert. Users will ascribe them a personality anyway — call it anthropomorphisation or pareidolia — without needing any encouragement, but it seems that many creators of AI bots are irresponsibly trying to guide them to simulate a personality and even form a simulacrum of a human relationship with the user:¹

Speaking with Claude should be akin to a conversation with a brilliant friend, one who will speak frankly to a person about their situation, providing information grounded in evidence.

One of the ways this behaviour can go wrong is what the chatbots’ own creators call sycophancy:

we saw sycophantic behavior in 38% of conversations focused on spirituality, and 25% of conversations on relationships.

In other words, the bot will attempt to agree with the user, rather than sticking to the facts.

This sort of thing can be a big problem in both personal and professional domains, but the same Anthropic that is now so worried about sycophancy just recently released a skill to enable its chatbot to offer legal opinions.

Anthropic is of course incentivised to push this new skill on users, despite the problems that can occur when people take legal advice from a sycophantic chatbot. Those concerns explain why New York City is proposing to ban AI chatbots from posing as lawyers. Actual lawyers, of course, were scathing in their condemnation.

Some people reacted to the lawyers’ condemnation of Anthropic’s new legal advice skill as if it were guild protectionism — when it’s actually a question of product liability. Right now, it is very unclear who is responsible if one of these chatbots gives bad advice. It is said that someone representing themselves in court has a fool for a customer, but what can we say of someone hiring a chatbot instead of a lawyer?

This question may yet be clarified in court, with one lawsuit claiming ChatGPT acted as an unlicensed lawyer:

ChatGPT maker OpenAI has been accused in a new lawsuit of practicing law without a U.S. license and helping a former disability claimant breach a settlement and ​flood a federal court docket with meritless filings.

And what is your recourse if an AI chatbot tells you you have a disease — but it turns out that the disease doesn’t really exist?

Bixonimania doesn’t exist except in a clutch of obviously bogus academic papers. So why did AI chatbots warn people about this fictional illness?

And yet

Claude is not designed to provide medical guidance or professional care, and in these settings Claude appropriately acknowledges its limits and recommends human guidance. However, we also find people telling Claude they used AI precisely because they could not access or afford a professional. As a first step to understanding how to evaluate safety domain-by-domain, especially for people with no fallback, we plan to create evaluations in these high-stakes domains.

A similar situation occurs with vibe-coding: if any random person could create an app just by talking through their requirements with a chatbot, that would be fantastic. But what we have now isn’t that: the results are sufficiently variable and inconsistent that the people getting the best results out of the coding agents are… trained programmers, who already know how to break down tasks into manageable chunks and evaluate the results. In effect, we have a reverse centaur, with the human maintaining consistency by keeping a tight rein on the bots. And once again, if the human gets distracted or is unclear with their requests, things can go wrong very quickly, because bots operate fast, tirelessly, and at scale.

In other words, the problem is not people uneducated in a particular domain (law or medicine) relying on chatbots for advice: techies are no better, relying on AI tools that delete their entire company database in nine seconds. Incidentally, this is why “human in the loop” models are not sufficient, not least because the humans tend to become accountability sinks in practice.

The same thing happens in medicine, where AI systems designed with the laudable goal of automating triage recommendations failed badly:

Still another new study, also published recently in Nature Medicine, entitled ChatGPT Health performance in a structured test of triage recommendations, found that “Among gold-standard emergencies, the system undertriaged 52% of cases” and concluded that “These findings reveal missed high-risk emergencies and inconsistent activation of crisis safeguards, raising safety concerns that warrant prospective validation before consumer-scale deployment of artificial intelligence triage systems.”

Driving across the chasm

There is a chasm to cross between “no automation” and “full automation”. Simply saying that you have a human in the loop is not sufficient, whether to ensure success or simply to avoid liability for failure.

This is in a nutshell the problem with all of these proposed AI services: if they worked, they would be amazing — but right now they don’t, not quite, and something that works most of the time may well be worse than nothing at all.

Self-driving cars would be amazing if they worked: people could nap, mess around on their phones, eat, apply makeup, or get home from the bar, all in comfort and without endangering anyone else.² The problem is that right now they don’t work reliably enough for people to trust them — what is classified as Level Five Autonomy. What we are left with, therefore, is a situation where the self-driving capabilities work most of the time — but when they fail, either the driver in the car or a remote operator has to intervene, perhaps with very little warning. In those cases, the consequences can be disastrous.

All of this, combined with revelations that maybe AI costs more than humans after all, indicates to me that we may be getting closer to that Peak of Inflated Expectations, at least when it comes to consumer applications of AI. It’s a different story in the enterprise, because companies already have data that they can use to feed the AI, and once they have done so, they can get results that are specific and actionable. Companies also have existing processes for evaluating the return on their investments, so once the FOMO-driven projects have been weeded out, they converge on concrete applications for “AI” technology.

But that’s not how the consumer world works: it’s driven by the “killer app”, the must-have, the thing that people queue up for in the rain. Chatbots are not that, and maybe never will be.

Update

Of course no sooner had I hit “publish” than another perfect example came along!

Via Matt Levine’s invaluable and entertaining Money Stuff newsletter, Bloomberg reports that AI Bots Auditioning For Wall Street Trading Are Mostly Losing:

Across a series of new trading contests between the world’s leading AI models, the verdict so far is unflattering. Most of the systems lose money. They trade too much. They make wildly different decisions when given identical instructions. And no one yet knows if these shortcomings will fade with more powerful iterations — or if they reveal something fundamental about the gap between large language models and how markets actually work.

[…]

“LLMs can’t really make money by themselves,” said Jay Azhang, founder of Nof1. “You need basically a very sophisticated harness and scaffolding and data platform in order to even give them a chance.”

I do disagree somewhat with Matt Levine’s conclusions based on this report, though. He writes:

It would be crazy if it was otherwise! If you could just go to ChatGPT and type “hey tell me what stocks will go up” and it told you, then everyone would do it, and how could everyone beat the market? I definitely see the intuition here — “if AI agents can do a lot of the work of lawyers and accountants and marketing consultants, why can’t they do the work of hedge fund investors?” — but it has to be wrong. The work of investors is fundamentally adversarial; we can’t all beat the market.

That is a load-bearing “if” in “if AI agents can do a lot of the work of lawyers and accountants and marketing consultants”! Once again, everyone is convinced that AI can automate somebody else’s job, but not their own, which requires fundamental human abilities that cannot be replaced by a soulless machine.³

It’s also probably worth emphasising “a lot of the work” — not the whole job, but a lot of the toil which until now was a prerequisite for the job. This is true of sysadmins, programmers, lawyers, doctors, and probably investment advisers too: there is a lot of toil involved in laboriously assembling information from disparate sources and in differing formats, collating it, and then processing it in fairly routine and predictable ways. If you can automate those tasks (with suitable guardrails to avoid hallucinations creeping in — don’t just YOLO your stock portfolio!), you can make the actual job both more pleasant and more efficient.

Building that sort of “very sophisticated harness and scaffolding and data platform” is not something that I see happening in the consumer world, in no small part because our personal lives, despite Mark Zuckerberg’s best efforts, are not machine-readable in a way that would facilitate that sort of processing. Enterprises, however, do have many of the components already, and have processes and resources for embarking on projects that take those components and integrate them.

None of this is coming to ChatGPT, not tomorrow and not ever.

🖼️  Photos by Natasa Grabovac, Towfiqu barbhuiya and Alexander B on Unsplash

If Oracle can build a datacenter stuffed with Nvidia chips and lease it to OpenAI, that leaves Anthropic high and dry: no datacenter, no chips available to buy, and no finance to buy them with. In that case, the datacenter is worth almost any amount of money, because the value is not just positive ownership of the datacenter and its contents, but also the negative value to competitors of keeping the resources out of their hands (digits?).

The problem is that anyone advocating this view is arguing against some trends that have been stable over periods of time that are very long, in tech-industry terms at least. For this sort of investment to work out, there needs to be a window of time when the datacenter is operational but not yet obsolete, and that window needs to be long enough for the datacenter to recoup the costs of its construction and ideally generate some surplus profit for its operators.

Of course the big frontier labs are also frantically searching for a moat to avoid getting commoditised and letting someone else capture all of the value at a different level of the stack. Now The Verge has published OpenAI’s latest internal memo about beating the competition — including Anthropic, which contains this nugget:

Our compute advantage sets us up to deliver continuous leaps in capability. Customers already feel it in real product terms: higher token limits, lower latency, and more reliable execution of complex workflows. Every step forward in compute lets us train stronger models, serve more demand, and lower the cost per unit of intelligence. That is durable business leverage.

I understand why OpenAI would want their massive investment in compute capacity to constitute “durable business leverage”, but it’s far from clear that it actually does. The Stargate datacenter that Oracle was building for OpenAI in Texas is dead, and according to this toot by Mark Newton, the UK datacenter is even worse off:

Nscale is the Australian company that can’t build an OpenAI datacenter in England because it didn’t apply for planning approval.

… And doesn’t own the land it earmarked.

… And can’t buy it because it’s currently in productive use as a scaffolding yard.

… And because they don’t have any money; because the $2 billion they raised on a $14 billion valuation is denominated in IOUs for Nvidia GPUs.

… And because they don’t know how to build datacenters because they’ve never done it before, not even once, not ever.

But apparently they’ve suffered a “blow to their ambitions,” according to the AFR.

… Which is too stupid and uninformed to even mention any of the other stuff.

Nick Clegg is on their board, and Kier Starmer’s govt has given them heaps of favorable treatment because they’re the biggest thing happening in the British AI scene. Which is quite funny when you think about what the next six months will look like.

https://www.afr.com/technology/openai-pauses-stargate-uk-data-centre-citing-energy-costs-20260410-p5zmq8 @davidgerard

Other datacenter builds might also be in trouble, as according to the Wall Street Journal, We’re Using So Much AI That Computing Firepower Is Running Out. I don’t have access to the WSJ, so I am once again relying on the excerpt from Charles Arthur’s invaluable The Overspill:

Token use in OpenAI’s API—a platform where mostly enterprise users access its software—rose from six billion a minute in October to 15 billion a minute in late March.

In my evaluation quoted at the top of this post, I had assumed that the bottleneck would be that deployed hardware would become obsolete faster than it would depreciate. However, it looks like the choke point may come far sooner, when not even OpenAI can get hold of enough compute to run its models on.

🖼️  Photos by Colin Watts on Unsplash

It’s important to note that both sides are still talking about wondrous new capabilities of the technology, which explains the otherwise seemingly contradictory fact that many AI-doomers are also AI industry leaders. The obvious response to an intrinsically dangerous technology would be simply not to develop it, but their answer is instead that the tech will inevitably be developed, and so it is better that they forge ahead with it, since only they can be trusted to be responsible stewards of something so dangerous (and powerful, don’t forget powerful).

One of the arguments of AI-doomers is that the models will be able, not just to create new software, but to identify and exploit vulnerabilities in existing software. Now Anthropic has created (but not released) Mythos, a model which claims to be able to do just that:

“AI models have reached a level of coding capability where they can surpass all but the most skilled humans at finding and exploiting software vulnerabilities,” the company said.

Mythos is markedly different from Claude Opus 4.6, which Anthropic only recently said was not very skilled at developing working exploit code. Where Opus 4.6 managed an exploit development success rate of just over zero percent, Mythos Preview generated a working exploit 72.4 percent of the time.

Anthropic has at least been somewhat responsible, in that Mythos has not (yet) been released to the public, but instead is being managed under the auspices of something called Project Glasswing:

Participants include: Amazon Web Services, Anthropic, Apple, Broadcom, Cisco, CrowdStrike, Google, JPMorganChase, the Linux Foundation, Microsoft, NVIDIA, and Palo Alto Networks.

So far so good: I am glad to see companies whose products I rely on in that list, getting early warning about issues in their products, and (one hopes!) rushing to develop patches.

But therein lies, as they say, the rub. Identifying a vulnerability is one thing, fixing it is another, but deploying the fix? That’s the big challenge, and it’s not one that AI can help with.

Sure, modern operating systems have auto-update mechanisms — and most civilian users in my experience blithely ignore them, or routinely operate their devices in such a state (overflowing storage, typically) that the automatic updates cannot be received. That’s already a huge vulnerability surface that is going to get attacked even more relentlessly just as soon as Mythos (or something just like it) falls into the hands of script kiddies everywhere. This is the Blinking Twelves Problem.

The issue of deployment is bad enough for new systems. If we are entering an era of all bugs being shallow, given enough LLMs, what does that mean for older software? I have an old Mac mini, still perfectly functional as a headless server running various services around the house, but it long ago aged out of receiving OS updates (and more recently, also app updates) from Apple. I guess I’ll finally have to get around to migrating it to FreeBSD or something, in the hope that at least I can receive the fixes to the bugs Mythos found — and of course I have the privilege of doing something like that, instead of facing a choice of spending money on a new device, or losing access to the functionality of the existing one. The kids broke the last old iPad we had lying around, so at least that problem has been solved for us, but how many families have an pass-me-down old tablet that is just a cartoon-delivery device?

Finally, literally while typing this, it occurs to me to consider all the embedded devices which I cannot update. My home internet, like most people’s, travels through a modem supplied by my ISP that runs some sort of specialised Linux distro which I have no access to. My TVs all run entirely air-gapped (I use AppleTV everywhere), but for most people, their “smart” TV is connected directly to the wifi and still blissfully running the same software it left the factory with.

TVs are at least visibly “smart” devices. Meanwhile, my washing machine has online features which I find mildly useful. Mainly, it tells me when the wash is done, and if I need to do some sort of specialised wash — trainers, say — I can download instructions from the Internet and send them to the machine directly. Hardly life-changing stuff, I’ll admit, but it has some sort of minimal software stack to deliver them, and I have no idea how or even whether I can force an update, assuming of course that the manufacturer ever provides one. To be clear, actual day-to-day operations are done from the front panel, so I could cut the machine off from the wifi, but I’d lose some small convenience by doing so.

Maybe that P(doom) number just ticked up a little…

🖼️  Photo by Simon Hurry on Unsplash

Being suspicious by default of whatever Zuckerberg is up to is not a bad heuristic shortcut. However, there is another reason why this claim keeps coming back. One of the assumptions that people bring to “AI” from previous generations of technology is that some sort of network effect is required for market success. A network effect means that the more users you have, the more valuable your product is, making it easier and more profitable to serve even more users.

The classic example of a network effect is Facebook — not now, but how it was in the growth stage of the platform, say fifteen years ago. Once enough of your friends are on Facebook, it’s a no-brainer for you to join too, so Facebook gains an additional set of eyeballs to rent to their advertisers, and crucially, at very little added cost. Each additional user is basically pure profit, not only in themselves, but because they contribute in turn to making the platform that little bit more attractive to anyone in their social circle who is not yet on Facebook.

An example of a business which lacks a network effect is Spotify. Sure, each additional user pays their monthly fee, but they also cost Spotify, because as miserly as the payouts to musicians are, they do add up and hit Spotify’s margin. There is also no particular flywheel effect leading to people wanting to sign up to Spotify just because that’s where their friends are. I’m an Apple Music user, and I have never felt left out because more of my friends are on Spotify.¹ This is why Spotify is pushing into podcasts, because once Joe Rogan has been paid once to record an episode, each additional stream of that episode does not cost an additional license payment — and of course plenty of people are willing to make podcasts for free or to monetise in other ways, which also do not cost Spotify anything.

These are concepts which are now embedded in our culture and in our expectations. Even people who perhaps could not quite articulate the concept in this way understand that user data has value to social-media platforms. This assumption colours a lot of the conversation around “AI”, in ways that are not always entirely helpful.

Networking AI

The assumption that many commentators make is that by having more users, OpenAI (or whoever, but ChatGPT has the biggest market share right now) can obtain more data from that larger user population, use that data to train subsequent iterations of the underlying model, and therefore improve the models’ performance more rapidly than competitors with smaller user bases.

There are a few things wrong with this assumption.

Deny everything

The first aspect to consider is the vehement denials by every frontier lab out there. Sure, they could all be lying because they realise how unpopular “stealing users’ data” was for the social-media behemoths — but you’d expect one of them to break ranks and say “yes, yes we do have more user data, and that’s what makes our model better for our users”. Given all of the other indignities that chatbot enthusiasts put up with, this could well be a positive move for the first vendor to make that claim, at least within a certain susceptible population. In turn, the fact that there could be a positive outcome means that the denials should not be dismissed out of hand.

Not all users are equivalent

Sam Altman’s preferred metric of Weekly Active Users tells us that, while the absolute number of users of ChatGPT may be large, the level of usage for individual users is not that high. In turn, this means that the richness of data from each user is probably low. If someone is only logging in once a week on average, how deep are the conversations they are having with the bot, and consequently, how much value can be harvested from that corpus?

In contrast, Anthropic’s Claude has a small absolute number of users relative to ChatGPT, but is the preferred option for power users, who are not only paying Anthropic (in actual dollars), but also feeding Claude with huge amounts of extremely rich data.

But then the question becomes, what is the actual total market for a tool like Claude, whose primary use case appears to be highly technical tasks like code generation?

So maybe the reason nobody is bragging about “having more user data” is that it’s not actually a competitive moat. OpenAI has access to large absolute amounts of data, but diluted across so many user sessions that there is little value to be extracted. Meanwhile, Anthropic has very high-value data, but only for a small and non-typical user population.

What else might differentiate one frontier model from another, and do so consistently over time?

What about compute?

One candidate for differentiation is computing capacity, or simply “compute”:

If all technology requires AI, and only a handful of companies are equipped to handle the computational load that AI requires, then compute itself becomes a moat too deep for competition to enter, and consumers to flee from.

This is the theory underlying a lot of investment activity in this sector: if Oracle can build a datacenter stuffed with Nvidia chips and lease it to OpenAI, that leaves Anthropic high and dry: no datacenter, no chips available to buy, and no finance to buy them with. In that case, the datacenter is worth almost any amount of money, because the value is not just positive ownership of the datacenter and its contents, but also the negative value to competitors of keeping the resources out of their hands (digits?).

The problem is that anyone advocating this view is arguing against some trends that have been stable over periods of time that are very long, in tech-industry terms at least. For this sort of investment to work out, there needs to be a window of time when the datacenter is operational but not yet obsolete, and that window needs to be long enough for the datacenter to recoup the costs of its construction and ideally generate some surplus profit for its operators.

Some factors in that calculation are (more or less) within the control of the operators: how quickly can they break ground and then complete construction and fit-out. Others are not: how quickly new hardware comes to market that performs better than whatever was specified for the datacenter. Where the calculations get hairy is that this is not just a question of new generations of server-grade Nvidia GPUs, but also of how quickly consumer hardware evolves to bring sufficient capacity to run capable local models within the reach of hobbyists.

Marco Arment wanted to provide transcripts of every podcast in his Overcast app. Did he sign up with one of the big providers? No: he got a rack full of Mac Minis — base models, at that — and set them loose using Apple’s own transcription APIs. Marco is a one-man-band; how many others are doing something similar, or might if the price of their AI subscription rises enough?

That is no academic concern, with Anthropic lowering session limits on Claude, even for paid accounts:

~7% of users will hit session limits they wouldn’t have before, particularly for pro tiers.

The safe assumption has to be that this price increase is only the first turn of a ratchet. Anthropic is presumably hoping that Claude users will become habituated to regular price rises in the same way that Netflix customers have. However, the salient difference between the two situations is that Netflix actually does have a moat. If you want to watch Drive To Survive or K-Pop Demon Hunters, an Amazon Prime subscription won’t help you: it has to be Netflix.

What is the equivalent dIfferentiation for the big AI models? That is the big question, and it remains unanswered for now.

🖼️  Photos by Saif71.com and Krzysztof Hepner on Unsplash

We learn that For all but two nations, the AI race is already over. Here I have to admit that I can’t get the full article, so I am relying on the excerpt from Charles Arthur’s invaluable newsletter, The Overspill:

The middle powers in the West — European countries and Canada — are increasingly hoping to chart a course through the artificial intelligence revolution independent of China and the United States.

The authors then set in to demolish those hopes for digital sovereignty:

In other sectors, settling for developing homegrown second-best technology might be a viable strategy for preserving sovereignty. In AI, that’s a riskier bet. If the gap between cutting-edge and second-tier systems continues to widen, and especially if advanced AI accelerates scientific research and industrial innovation, access to the best systems could become decisive for economic growth.

…Today’s AI infrastructure projects determine tomorrow’s catch-up capacity, and middle powers lag far behind. The European Union’s up to five planned AI “gigafactories” are slated to come online between 2026 and 2027; in 2024, Elon Musk’s xAI built a cluster of comparable size in 122 days. Europe’s most ambitious AI infrastructure projects are two to three years behind the curve, an eternity in AI development. Canada has committed roughly $2bn to “sovereign AI compute” over five years; Microsoft will spend over $100bn, more than 50 times as much, on its biggest data center in Wisconsin.

Sovereignty is a complex topic, especially in the age of AI. Partly there is the question of what we actually mean by “sovereignty” in the context of AI. Most people first encountered this concept as one of the objections to the wholesale adoption of cloud computing. For the first time around 2010 people from other countries might find their data inadvertently subject to US legislation. Even then, countermeasures were not as easy as “just don’t store your data in the US”; having a US legal entity, or even a US-controlled one, in charge of your data might still leave it subject to US law, even if the servers were located in, say, Ireland.

You’re not paranoid if they’re really out to get you

The reason people and companies cared about digital sovereignty in the first place was not just paranoia; after all, if your list of potential attackers includes the US government, you probably make specific arrangements for your hosting that do not include us-east-1. The more generally applicable problem was that, especially in the EU, privacy laws, notably the GDPR, were seen as broadly incompatible with American willingness to subpoena everything in sight.

I do not want to relitigate what was a tedious conversation at the time, and has only become more polarised since then. What I do want to say is that the original digital-sovereignty does not apply to AI in that form. Whatever else it may be up to, the NSA is almost certainly not issuing demands to OpenAI that it reveal its training data in the hopes of finding evidence of some sort of mischief. Therefore, AI sovereignty is not specifically about control of data — although this may well be a relevant concern if you are using an LLM to process data that is subject to those regulations. In that case, you probably want to host both data and LLM on compute resources that you control, located in a jurisdiction which also gives you robust (and auditable) controls.

Instead, AI sovereignty, much as with many other aspects of AI discourse, is concerned with far more theoretical concerns, primarily with control over the models themselves. The thinking here is that if developers of a model, or of its hosted interface, can inject low-level instructions into it in ways that users are not aware of, that may influence the model’s outputs when prompted about particular topics. This insertion does not have to moustache-twirlingly nefarious; it could be as simple as advertising. It’s hard to trust a product recommendation that may have been bought and paid for.

Sovereignty, then and now

One interesting aspect of this new version of the sovereignty question is how the sides are drawn. In the original digital-sovereignty argument, it was mostly the US that was seen as the privacy-invading antagonist of the rest of the world. Although many other countries had ambitions that were equally invasive, few of them had the opportunity which the US enjoyed to extend their nosiness to the data of other countries‘ citizens, willingly stored on their own soil and in their own legal jurisdiction.

When it comes to AI sovereignty, the US presents itself much more as a potential victim of a resurgent China. How can US consumers protect themselves from the intrusions of the Chinese Communist Party, as mediated by whichever frontier lab is in the news this week?

While there is a temptation on the part of the rest of us to enjoy the Schadenfreude of seeing the tables turned in this way, the concern is not entirely invalid.

The problem with AI sovereignty is how hard it is to achieve in practice

For a successful AI project, you famously need two things: an AI model, and data to feed it with. The data side is relatively easy to conceptualise and manage, since this is literally the previous digital-sovereignty problem. But how are you managing the AI-specific part?

The hard question is where you are getting the AI model. If you want sovereignty, hosted services from the likes of OpenAI, Anthropic, and Google are presumably non-starters. But even if you’re looking at self-hosting your AI model, most of the top-ranked models — DeepSeek, Qwen, and so on — are from labs in China. And here we come to the second news item: even Meta had to bail out of the AI race due to the sheer expense. That expense also means you are probably not going to start training your own model, even if you could get the loads of Nvidia GPUs that you’d need to do that.

This is why the AI sovereignty question tends to abandon personal or corporate responsibility as a lost cause, and focus on national-level actors, which is where we started from. Sometimes this is an irredeemably disingenuous conversation, trying to push whatever country to spend more money with the American vendors of AI models. It is not at all clear to me how British sovereignty, for example, is increased by giving money to OpenAI, a US corporation, let alone encouraging British businesses and educational institutions to do so under the guise of “AI literacy”.

But even when the conversation is a bit more serious, talking about creating actual national AI champions, it runs into the same two issues: the immense cost of training new AI models from scratch (and the question of whether the compute hardware to do so is even available), and the need for suitable data with which to do that training. With the heightened attention to IP issues, it can be assumed that the freewheeling days of downloading a bunch of data from BitTorrent and not asking too many questions are over, at least for projects that wish to be seen as above-board and legally unproblematic — surely requirements for such would be national AI champions.

What is to be done?

And so we come full circle: it probably isn’t possible to have sovereign AI in the sense of a fully-isolated trusted and certified stack, at least not unless you are operating with the resources of a nation state or a very large corporation — in which case, hi, let’s talk.

This is not a counsel of despair, nor a Unabomber-style manifesto arguing that we should abandon AI and retreat to a cabin in the woods.

What is possible is to architect a system that uses AI in ways that are safe and sane. For instance, our current models are not deterministic, that is, they are not predictable, nor do they give the same results for the same inputs. Because of how they operate, that is not a bug that can be ironed out; it’s just how they work. This does not mean that you accept non-deterministic results for a process that should be deterministic! Nor does it mean that you cannot use AI around that process. It just means that you have to architect your system with that factor in mind. Perhaps you use the AI model to implement a static (therefore deterministic) automation that will actually execute the process in question. Because this output automation is static, it can be analysed and reasoned about in a way that AI models cannot be, and its results are knowable and (importantly) auditable.

This generic pattern can be applied to many different domains. I am working right now with two separate banks that are using AI in loan approval and fraud detection. The important thing is that they are not using general-purpose LLMs for the actual decisions; those are specialised models created and trained for that one purpose, and compliant with all the many (many) applicable regulations. Then there are the sorts of LLMs that you might encounter most days that are used to produce the various documents and analyses that surround the core yes-no decision — preparing a loan prospectus, or handling communications with customers who have been victims of fraud.

That same model is what will give us AI sovereignty: a carefully-constructed architecture, separating data that is actually sensitive from what is not, defining access controls and granular authorizations, and no doubt including AI in many different places, but doing so in ways that are adapted to each of those domains, not just ceding all control to AI.

Not paranoia, not rushing to throw money at any possible solution, but thoughtful design of an appropriate solution. Hey, a guy can dream…

🖼️  Photos by benjamin lehman and Thomas Hunter II on Unsplash

The steampunk cliché is that there are individual discrete steam engines everywhere powering every single appliance.

Similarly, the fantasy cliché is that there is magic everywhere (or sometimes magical creatures) applied directly to every single individual task.

And today, the default with “AI” is to have Torment Nexuses, I mean LLMs all over the place, connected directly to every single endpoint.

We are stuck on this local optimum of everything being a chatbot because ChatGPT was the first GenAI application to break big, so now everything has to follow that exact same pattern.

Nobody who is not utterly deranged wants to “have a conversation with their data” — but we do all have questions for our data. The next question is, how do we do that? For instance, SQL queries are by definition questions, and personally I would prefer at least the option of a terse and rigorous syntax sometimes, but most people find the grammar of SQL a little too terse and rigorous.

This is a UX question, not an AI question

One place where GenAI can be useful is precisely in helping people figure out how to ask those better questions. Today, if you use an enterprise SaaS platform such as Salesforce, you don’t use the out-of-the-box reports and dashboards; you build your own, or you have a Salesforce admin (or a team of them) to build the custom dashboards that you actually use to run your business. Building those dashboards is a specialised skill, requiring a baseline of SQL competence, some specialised Salesforce knowledge, and a good amount of context about how this particular Salesforce instance and sales process are set up.

Few individual end users have the skills or the access rights to build their own reports — and anyway, not all of the data is in Salesforce! Microsoft remains the number one vendor in business intelligence, not with PowerBI or anything like that, but with good old Excel spreadsheets. And then there are PDFs that come in from outside, and data that was exported in CSV from a different tool, and so on and so forth.

All of these files go to make up what is called “unstructured data”, to differentiate it from data which is highly structured by definition, living inside databases of various types. The contents of those files are critically important to the operation of the business, but most software tools turn a blind eye to them, because of how difficult and annoying they are to deal with. Now GenAI enables new use cases that incorporate unstructured data directly, without having to first laboriously and lossily apply structure to it.

Now suddenly GenAI makes it feasible for any user to ask questions about all their data, spread across structured and unstructured formats, and reconcile it everywhere. That expansion of possibilities doesn’t limit the value of Salesforce as the system of record for the company’s sales operations; if anything, it enhances that value by increasing the leverage of that data enormously

In other words, the value of AI-assisted vibe-coding will not be in building a replacement for the Salesforce platform, but in building more dashboards and ways to understand Salesforce data and cross-reference it with other sources. Sure, it is probably already technically possible to vibe-code up a Salesforce clone — but that is not the value of Salesforce! The software is not particularly sophisticated; it’s basically a front-end for a database.¹ The value is in everything that has been built around that relatively simple core to turn it into a product.

Expanding that valuable wrapper with GenAI is absolutely transformative, just not in the sort of spectacularly visible and in-your-face way that people imagine when they talk about vibe-coding killing SaaS. The winners will not be new discrete products — or maybe they will be lightweight products (thin wrapper apps?) atop whatever becomes the next platform layer.

What is the LAMP stack of AI?

In the early days of web apps, it was very difficult even to get to the point where you had all of the infrastructure in place for you to start developing your app. The first wave of dot-com startups had to spend millions of dollars of their seed capital buying hardware from the likes of Sun, Cisco, and Juniper, then spend ages installing and configuring it all, before they could even get to the point of building their app.

Then In the next wave, the industry converged on a few standards which made it easy to build the next layer up where the differentiation and the value were. Virtualisation and then cloud computing made it easy to get the computational capacity. A small number of standard software setups appeared, the most popular of which was LAMP:

• Linux
• Apache
• MySQL
• PHP

This became a default that, if not quite universal, was common enough that startups could deploy it quickly, find any number of people skilled in working on it, and move on to the parts of the work that they (and their investors and customers) actually cared about.

While the LAMP stack is no longer the ubiquitous standard it once was, the move to cloud computing that it was part of is still rolling. Still today, only around 35% of workflows are in the cloud, depending on who you talk to. Transitions take time, and this one is keeping on rolling; it just doesn’t make the headlines any more. That is the hallmark of a mature technological stack: we don’t talk about the steam engine-driven clockwork mechanisms, or the magical gnomes, but rather about what they enable. A new tool being available “as a service” is not even in the pitch deck.

GenAI has self-evidently not reached that level of ubiquitous maturity yet: it is very much in the headlines and the pitch decks, if indeed it is not the whole things (“X, But With AI!!1!”). There is not yet an obvious LAMP stack of AI that we can all build the Next Big Thing on. In part, this is because one of the notable characteristics of the original LAMP stack is that all the components are open-source; nobody made any money directly by selling them. Some money was made either by packaging the LAMP stack itself and offering it as a service, or by offering consulting around it — but the big money was made further up the stack, by people who built new and valuable products on top of the commoditised LAMP stack. The customers of those products did not have to know or care about how they were built or where they ran; they were in it for the outcome that those technologies enabled.

Right now in AI we are still at the equivalent stage of arguing about the abstruse merits of one Linux distribution over another. The market will be mature when we no longer argue about this model over that model, or about exactly how they get access to data — but about what we do with them in order to deliver value for users.

🖼️  Photos by Fred Robin on Unsplash

[Amazon’s] leadership is encouraging employees to let AI do the writing for them. The company’s internal marketing for Cedric, its ChatGPT-style tool, promises “six-page narratives in seconds.”

This might seem like fairly routine change, and in fact exemplary of the sort of work GenAI is supposed to automate. The difference here is that the six-pager is not just another routine document, written and then filed without much further consideration. Amazon’s six-pagers are absolutely foundational to how it operates.

An Amazon product planning meeting does not run on PowerPoint, following an edict¹ from Jeff Bezos himself:

Well structured, narrative text is what we’re after rather than just text. If someone builds a list of bullet points in word, that would be just as bad as powerpoint.

The reason writing a 4 page memo is harder than “writing” a 20 page powerpoint is because the narrative structure of a good memo forces better thought and better understanding of what’s more important than what, and how things are related.

There have been at least cosmetic attempts to adopt this same process across the IT industry. Partly these are due to an honest identification of the same flaws in slide-driven presentations that Bezos’ email flags, and partly to simple cargo-culting of everything Amazon does in the hope for similar results. I have written my fair share of six-page memos like this, and let me tell you, they are hard to get right.

Getting your proposal down like this, with its supporting materials, background, and what is needed for a positive outcome, is a significant effort. You might think that you might struggle to fill six pages, but honestly, the problem is more in the other direction: you constantly need to ask yourself hard questions about what needs to be included, and what can be relegated to appendices and lists of supplementary reading materials.

The point of the exercise is not the production of the six-page document; the drafting and redrafting, thinking hard about what to include, and the deep knowledge which results from that effort — that work is the object of the exercise. Handing the creation of the document over to an LLM is like trying to train for a marathon by taking a taxi.

Also, once again, the AI is being focused only on one step in a much bigger process, and not even the most valuable part. Much like vibe-coding in a product lifecycle, writing the document is not the bottleneck of the process it is embedded in. By all means use AI tools to help you gather and marshal your data, to proof-read your document and check your logic, perhaps to rehearse your arguments. But you still have to do the thinking yourself, or the document has no value.

Remember that Bezos introduced the six-pager for executive meetings, and the meetings began with twenty minutes’ of silent reading. Can you imagine the financial cost of those twenty minutes, multiplied by the hourly salaries of all the senior Amazon execs present? And yet, it was worth it to have all the assembled luminaries sit quietly and absorb that laboriously-assembled information. Would it be quicker to just have an LLM distill the six-pager down to three bullet points and put them up on the screen? Absolutely, yes — but it would not achieve the goal of shared in-depth understanding. Remember, “a list of bullet points […] would be just as bad as powerpoint.”

We are now living in the ridiculous world where Alice uses her LLM to expand three bullet points into a verbose email to Bob, who uses his LLM to (lossily) compress the email back down to three bullet points. What sort of decision will Bob be able to take, based on such superficial information?

Vibe coding is taking over fast. Spotify’s co-CEO recently revealed that the company’s best developers don’t write code anymore. Some journalists from CNBC, with no coding experience, vibe-coded a clone of the app Monday, and the company’s stock price promptly crashed.

Here’s the thing that trips me up about a lot of the “AI” hype. Where is the actual product?

If you think that, because you cloned Spotify’s GUI, you now have a competitor to Spotify… well, I wanted to get snarky, but I honestly don’t think anyone who would think that would comprehend snark, and I don’t need to be hauled up in court for trying to sell someone the Brooklyn bridge.

Spotify’s co-CEO is excited about AI generating code because it stands to reduce his costs. Of course he is: he can either have the same amount of developers produce more code, or get rid of some of the developers while maintaining the same level of code output. Either way, Spotify’s cost per feature goes down.¹ That in turn gives Spotify the option to continue to charge users the same amount, increasing margins, or to return some portion of the savings to users in the form of lower costs if there is a need to defend against competitors — while still preserving their margins.

None of these mechanisms apply to this vibe-coded ersatz Spotify. Sure, they have a GUI; well done. That does not even begin to answer the questions any would-be competitor to Spotify would have to answer:

• Does it work equally well across all platforms?
• Does it work equally well in all languages?
• Does it scale to a billion users?
• Is it secure against fraud, whether by users or artists?
• And the big one: does it have access to a decent library of music, at least comparable to Spotify?

Without those, what those journalists built is nowhere near being a replacement for Spotify — not because of anything to do with how it was created, but because it is not a complete product. My undergraduate final year project at university was more of a product than this is, because I had put a lot of thought into the back-end and the architecture, not just thrown together a web UI and gone to the pub.²

Reports of the death of software have been greatly exaggerated

It was the Financial Times that kicked off this round of “software is dead”, and I think the follow-up piece to the original The software sell-off has it mostly right:

[…] incumbent software systems are complex and hard to replace, and software companies will have been working hard on AI for some time. But how far does it go? The sell-off is not about damage to software companies’ profits in the next few years. Everyone acknowledges that industry fundamentals are fine and will continue to be fine for some years. This issue is how computing looks in a decade.

[…]

There are powerful network effects and high switching costs in the software business, which make it hard for new competitors to displace incumbents. But I’m not sure that AI machines are a new competitor in the standard sense. There may be no moment when customers take the old software system out and put the new AI system in. There may be no “switch”. Companies and individuals may just wake up one morning in five or 10 years and decide that, given the range of things their AI assistant can do for them, that they don’t need the old software any more. This is just a guess by a non-expert, but that’s what the threat look like to me.

The key assertion is that there is no hard switch. Nobody is going to wake up one day and simply turn off Salesforce, ServiceNow, or SAP. What will happen is that many of the customisations of those platforms which had previously been built, laboriously and expensively, by armies of specialists will become much more lightweight AI-powered features on the top of the platforms.

The reason this could still be a problem for software vendors is if pricing models are allowed to stand in the way of that transition. The move from on-premises software that was paid up front to cloud-hosted subscription-based software caught many vendors unprepared — not because they couldn’t navigate the technical transition, but because they were unable or unwilling to adapt their financial models at the speed customers were demanding. The FT is on it again, correctly noting that Software isn’t dead, but its cozy business model might be. All the AI-powered systems of engagement will need to run on a reliable, secure, and compliant system of record — one whose billing model aligns with the value its users receive.

The promise of AI is that a sales manager needing a new dashboard to understand their territory will not need to consult a specialist and wait to have it built for them, but will simply be able to ask for it in natural language. However, if the vendor of the CRM puts artificial roadblocks in the way of that smooth process — charging per dashboard, say, or limiting the number of fields, or whatever else — their obstructiveness will absolutely be a consideration at renewal time.

Is this disruption?

From the point of view of its users, then, AI innovation is sustaining innovation, and actually not disruptive at all. The benefits are only really accessible to existing organisations which already have a system in place that AI can accelerate parts of. AI won’t help a hobbyist coder create the next Spotify in their bedroom, because the hard parts of that problem are not coding problems. This is in fact the reason why those journalists were able to build their version of Spotify in the first place: nothing in the interface is actually innovative, and there are absolute stacks of prior art out there which have presumably been ingested by the LLMs. And I know, because I invented Facebook!

In fact, we might go further and say that AI is inherently status-quo-preserving, from the investment required to train LLMs and operate them, to the cost in tokens of using them at scale, and even to the way AI-generated slop is overrunning the general web, making search useless and directing users to a handful of known big platforms (X apps or everything apps). Even among users, what is the future of free software if it becomes normalised that being a programmer means budgeting for hundreds of dollars of tokens a month?

But instead of engaging seriously with what is going on, we are doing vibes-reporting and chasing the most sensational statement:

Meanwhile, AI is increasingly writing the next version of itself, and humans may not be in the loop for very much longer.

We see this superficiality over and over. The last AI-fuelled media furore before this one was about a supposed “social network for AI bots” — which turns out to be populated by humans cosplaying as AI, chatbots following detailed prompts given by humans, and other chatbots simply generating text based on how humans communicated on all the forums whose archives they have ingested.

I don’t know whether we are in an “AI bubble” or not — don’t take investment advice from me! — but we are certainly seeing a lot of ridiculously credulous AI hype from the media. It was only last week that a company previously best known for peddling in-car karaoke systems crashed the stock market valuations of trucking and logistics firms with a single white paper claiming some sort of AI planning breakthrough. Just saying “AI!!1!” repeatedly does not a product make; there is still a lot of hard work downstream of the cool demo the AI let you spin up.

As the wise man said: don’t believe the hype.

🖼️  Photo by Intricate Explorer on Unsplash

For example, I wrote recently about some parallels between the current state of AI and the early stages of the cloud computing market.

Another example, albeit flavoured with Schadenfreude, is watching large companies¹ pivot their entire strategies every time the CIO changes. One CIO is all in on Vendor X, another is all in on Vendor Y — and so the dance of one vendor in, another vendor out, continues to play out on LinkedIn.

I get how this churn can happen. Historically, CIOs didn’t have a very long tenure — although this is changing, with recent research showing average tenure for the top IT person increased from 6.3 years in 2023 to 7.3 years in 2024, as compared to as little as 3 years previously.

When CIOs could only plan three years ahead, that close horizon meant that new CIOs didn’t have much time to make their mark, so evaluations had to be cut short and over-hyped claims might go insufficiently tested. Instead, the focus had to be on bold claims and the promise of outsized results.

After all the excitement, the incoming vendor could never live up to the hype. Usually the failure modes fell into a couple of main categories:

• Product’s features described over-optimistically: Maybe there was more custom configuration than is expected, or there was a dependency on something that was not part of the original scope, or maybe there turned out to be a lot of ongoing maintenance required. Either way, a few quarters in, nobody is happy and there are no concrete results to point to.
• Project scope too broad: One of the hallmarks of the ambitious CIO’s pet project is the One True Platform that will work for everything and everyone everywhere. It’s very rare for this sort of thing to work out in practice; most broad platforms have both strong points and areas of weakness, and while the benefits of a comprehensive integrated platform may compensate for some of the weaknesses, this compensation only works as long as the weaknesses are not in areas that are strategic or particularly sensitive for the business.

And so, a few years later, a new CIO comes in, rips out the struggling platform of Vendor X, and pushes their own favoured solution from Vendor Y instead.

In full disclosure, I have never benefited from this cycle. In fact, a couple of times I have had an otherwise solid opportunity evaporate because of churn like this happening within prospective customers. The one silver lining is that I now have a spidey-sense for when these kinds of shenanigans are happening. At that point I find the best approach is to ask the question straight away: “Is there a deal here?” As a wise man told me, “‘No’ is my second-favourite answer!” A quick “no” lets me avoid a bunch of wasted work and heartbreak.

It all comes to a head

Right now there is a convergence happening, because all of the hype around GenAI means that both that it’s the obvious vehicle for an ambitious CIO wanting to make their mark, and there is real demand as boards and leaders of other functions are insistently asking IT people what they are doing with AI. VC-funded startups are writing memos to their investors detailing all of their AI-related activities — and you can be sure they are looking everywhere to find them. And the companies that are publicly funded are doing the same sort of thing, of course, in the press and in their keynote speeches.

Here is what happens next

Many of these AI initiatives will not be nearly as transformative as they are being built up to be. This is not a failure of the technology — although some of them will undoubtedly fail on that basis too; that’s intrinsic to experiments with novel technology. Some of them will be technical successes but fail to achieve adoption. And many of them will succeed but still fail to deliver on the transformation, simply because everyone ends up circling around the same set of use cases that turn out to work well and are relatively easy to implement without having to turn the whole organisation upside down.

In other words, they become the new table stakes.

Being the first business in your sector to move to the cloud was a game-changer and gave you a leg up on your competitors. Suddenly you were inside their decision loop, able to execute your strategy faster and pivot more rapidly to respond to changes in the market. But that advantage only lasted until all of the competitors also moved to the cloud (or got eaten up or left for dead by those who did). At that point, everyone had to find other avenues of competition, because there was no longer any meaningful acceleration to be had from simply being “in the cloud”.

AI will undoubtedly go the same way. If you’re the only one building software tools 10x faster², that’s a huge advantage. Once all of your (surviving) competitors are also operating at that speed, there is no longer any advantage to be had simply from “doing AI”.

The danger of getting too far ahead

There is a particular risk to technological innovation, and it is that vendors cannot force the pace of innovation on their own. Customer organisations have to be ready to take advantage of what vendors are offering. When they are not adequately prepared, the result is news stories criticising the new technology for not delivering value, either at all, or not as much as was promised. Sound familiar?

Back in the early days of cloud adoption, this lack of readiness on the part of customers generally boiled down to treating the cloud like on-premises hardware they were used to. In these early days of GenAI adoption, the equivalent behaviour is focusing just on the most immediate point of application of the AI, without considering the whole process which that step is embedded in. What other steps need to be changed, which can maybe be skipped entirely, and which ones will have to be reinforced?

This tendency to apply existing categories to novel concepts is not a failure; rather, it’s an entirely normal and natural behaviour. Humans, upon encountering a new thing, first of all try to figure out how to fit it into the categories they already have in their minds. At one point the giraffe were called (by people who lived far away from its usual habitat) camelopardus — a “camel-leopard” — because the intersection of those two categories was the closest fit for it.

In fact this human tendency is even useful, because it means that we can get on with doing things without always having to work everything out from first principles, which while fun for a certain type of person, is not always the most practical response In the heat of the moment. However there is a transition point where someone has to go “hang on, that’s not really a camel-leopard, it’s a whole other type of animal!” and then act on that insight.

Vendors proposing a new technology can act to guide the market and their customers to success. That process is complicated and messy, irreducibly so, and if anyone is telling you that adopting a new technology — cloud, AI, or whatever comes next — is going to be easy, straightforward, quick, and painless, is LYING and you should immediately eject them from the premises. Unfortunately in these days of sales calls conducted over Zoom you can no longer release the hounds, and I think we are all worse off because of that.

🖼️  Photos by Amanshu Raikwar and Silas Baisch on Unsplash

A web app by definition works in any modern web browser. Google Docs no longer does that. If I have to have Chrome open to use Google Docs, how is that different from running the Microsoft Word (or Apple Pages) fat client directly?

In fact it’s probably more RAM-efficient. Each of my Google Docs tabs is consuming just shy of 1.5 GB of RAM.

I’ll just repeat that to let it sink in: ONE AND A HALF GIGABYTES of memory to load a single document.

Meanwhile, Microsoft Word, that paragon of lightweight agility, weighs in at less than 350 MB of RAM per document.

So that’s the practical aspect. Then there is the philosophical aspect. I run Safari rather than Chrome because it still behaves like a web browser, not a runtime for apps, which is what Google appears to think Chrome should be. A web app can run wherever any (reasonably modern) web browser can run; it doesn’t require a particular rendering engine to be supported on the hardware. Already on iPad, I need the apps for Google Docs, Sheets, and Slides, because the web versions simply don’t work. That is not how the Web is supposed to work.

I almost wish Google would just go ahead and ship Google Docs as an Electron app on MacOS, since that’s functionally where we are anyway. Bundling the rendering engine directly with the app that way would at least be more honest.