This is 100% Anthropic reaping what they have sown:

“oh no, our AI is too dangerous, it must be regulated” (repeat 1000x, get the pope involved too)

“…not like that.”

Me on Bluesky

So what does it all mean?

If there is a thread running through this blog, it’s that there is very little that is new under the sun. Remember the crypto wars of the 90s?¹

What is different this time is the nature of what is being banned. Software used to have zero marginal cost, but in the era of software that is provided “as a service”, that is no longer true. From Matt Levine’s Money Stuff newsletter:

A lot of the biggest and most successful companies now are enormously capital-intensive. They are artificial-intelligence hyperscalers, and their business model is like “build nuclear power plants and orbital data centers and massive chip fabrication facilities.” After years in which the cutting edge of the economy was nearly zero-marginal-cost software, now the cutting edge of the economy is extremely capital-intensive, uh, software.

All of that capital outlay only makes sense if it is matched by a proportional revenue stream. This is the Sam Altman thesis: “We see a future where intelligence is a utility, like electricity or water, and people buy it from us on a meter.”

The problem with meters from the point of view of users is that they keep running, as Uber found out when developers blew through Uber’s entire 2026 AI token budget in just four months. A mystery company reportedly spent half a billion dollars on Claude Code in a single month.

Scale to zero — or to the stars

This problem of open-ended cost structures is not a new one that is specific to AI. The most recent iteration of the billing dilemma was in serverless platforms.

Cloud software is roughly divided into three layers:

• IaaS, or Infrastructure as a Service: basically, virtual servers in the cloud. Apart from them running in someone else’s datacenter, these behave like normal servers: you log into them directly, deploy software, reboot them, and so on. Your bill is for a certain number of servers.
• SaaS, or Software as a Service: you have no idea where the software is running, you just connect via a web browser and get to work. Your bill is for a certain number of users, or “seats”.
• PaaS, or Platform as a Service: this is an intermediate level of abstraction, where you are not connecting directly to a server, but to application software running on top of one or more servers. The details are not important or even visible to you; it’s just “compute” (short-hand for “computational power/capacity”), but your bill might still be for a certain number of servers.

Serverless billing applies to that last model. The idea is that, since with PaaS you don’t access the servers directly, you should just pay for however much platform capacity you use, rather than for a fixed pool of capacity, as you would with per-server billing. This approach is pitched as being particularly attractive to startups: you don’t know if your offering will take off, so you don’t want to commit to high up-front costs — and if you do hit the big time and your thing is blowing up, you don’t want to be constrained by capacity.

Here’s the catch: serverless compute is significantly more expensive than pre-purchased units of compute billed as servers. The reason is that the risk of paying for idle capacity doesn’t go away, it just gets transferred from the hopeful startup to the cloud provider. Of course the cloud provider is aggregating demand and betting that no more than a certain portion of its customers will suddenly need a whole lot of compute capacity at once, but they are also charging a risk premium for their trouble. Basically, it’s an insurance model.

This means that if you do know your demand profile, you are better off not using the metered serverless model, but instead pre-purchasing the capacity that you know you need. You may even be able to mix and match, with baseline guaranteed capacity at one price point and a buffer on top that is charged at surge pricing rates if it turns out that you do need it.

But right now that is not how any of the frontier models work. They consume “tokens”, and they do so at a rate that is not always easy to predict, and which can be affected by non-obvious architectural choices.² In other words, the meter is always running.

What if you don’t have coins to feed the meter?

The problems extend beyond commercial software. At least in that world there is a revenue stream. As long as the token budget is less than the revenue which the token burn enables, the business case still stands up. But what about open-source software, or other non-commercial models? It’s one thing for coders to donate their time to projects, but even then, many projects are in trouble, with volunteer maintainers struggling to pay the bills even for projects that are foundational to many companies’ operations.

If widespread use of code-generating AI tools becomes the norm, and those tools burn through enough tokens that working on them comes at substantial financial expense, that equation starts to become impossible. This is one of the problems with Anthropic’s Mythos bug-finding AI model: finding a bug in a piece of open-source software is one thing, but patching it with AI tools would require a bunch of tokens, which a volunteer-run organisation may not have immediately available. And that does not even touch on the problem of deploying a fix once it has been developed, which may be especially hard for open-source components that are embedded deeply in other offerings.

Watching the meter

This is why it is particularly interesting that S&P Dow Jones Indices decided against fast-tracking SpaceX, Anthropic, and OpenAI into the S&P 500. The S&P 500 is what many index funds use; if OpenAI et al are not in the index, they do not have access to funds invested that way. The reverse is also true, of course, but investors always have the option of buying stock in those companies actively; the whole point of index funds is they are passive, and their investors don’t really want to worry about the contents of the fund on a day-to-day basis, or whether some overweight proportion of it is suddenly a massive bet that an unprofitable endeavour can become profitable within a reasonable timespan.

This choice by the S&P has been characterised as a bet against these companies making it; I am far from an investment professional, but I read it instead as a refusal to be bounced into making an exception on the basis of hype. Once these companies have been publicly traded for a year in a process called “seasoning”, they can be considered for inclusion in the S&P 500 index.

The reason to take this “wait and see” approach is that the thesis of companies and individuals continually topping up the meter on these AI services is far from proven. With more and more stories coming out of spiralling token bills, there is now a drive to manage AI’s runaway costs:

“In April and May, I started hearing from companies: ‘Oh my god, we are 3x over our entire 2026 token budget and it’s only April,’” J.R. Storment, executive director of the FinOps Foundation, a project under the Linux Foundation, told TechCrunch. “We started hearing existential crises, and the whole conversation shifted from tokenmaxxing and ‘go fast’ to ‘we need guardrails, how do we control this?’”

The big AI labs’ problem is that AI has advanced enough that many AI tasks do not need the latest and greatest frontier models. Marco Arment, creator of the Overcast podcast app, built a transcription service for every podcast on Earth using a rack full of Mac Minis — and the base model, at that. Sure, Marco had to come up with the up-front cost of the hardware, but he doesn’t have to worry about huge open-ended costs for using a metered service forever.

This sort of offline usage is a problem for the business model of the frontier labs precisely because it does not generate the ongoing ever-growing token revenue which their stock market valuation is built on.

But what about regulation?

Offline AI models are also the reason why any attempt at AI regulation that assumes the ability to prevent certain uses, or its use by certain groups, is doomed to failure. That doesn’t mean regulation is not worth doing, mind: it’s perfectly reasonable to say that the Instagram app should not have a built-in feature to “nudify” pictures that people post there. On the other hand, we should also not expect that a ban on AI features like that, or on entire hosted models like Fable 5, will eliminate abuse entirely. The reality is that bad people will continue to find ways to be bad. There probably do need to be controls on AI, but more in the way that we have controls on fertiliser, enforcing regulation and tracking at the point of sale.

The US Government can ban Fable 5 because it is provided as a service, which means there is a single point of access which can be blocked: Anthropic’s servers themselves. The attempt to ban PGP in the 90s failed because there was no one place you had to go to get PGP, and once you had it, you didn’t have to go back to the source every time; you could use your local copy of PGP entirely offline. But because “AI” models don’t have a moat, a ban on Fable 5 only buys a little bit of time until some other model which can be run offline achieves comparable performance.

Regardless, the lack of clarity around what the future usage patterns for the frontier AI models will be is the reason why the S&P 500 is not taking on the AI bet, or at least, not right now. They want to let things play out for a year, and then see what happens. The Fable 5 ban, regardless of its specific merits, justifies that caution, as it implies that government regulation of the AI market is a very real possibility.

And if you really do want to buy stock in SpaceX, Anthropic, and OpenAI in the meantime, you still have the choice of going and doing that directly, actively, rather than have it included willy-nilly in a passively-managed index fund, at least while the future outcome is still so uncertain.

🖼️  Photos by Dylan Michaud and Simone Dinoia on Unsplash

We are not quite at the point of the proverbial datacenter staffed by a man and a dog — the man to feed the dog, the dog to bite the man if he touches anything — but we are not far off. A datacenter has probably the worst ratio of on-site employment to surface area out there. There are going to be a handful of security guards (not exactly skilled labour) and a handful of on-site techs to deliver the “remote hands & eyes” service, and that’s it.

In online conversation about this topic, this Brookings report came up: New evidence on data center employment effects. Superficially, it seems to contradict my assertion above, but I think it’s worth digging into why (as usual!) it’s a bit more complicated than that. Handily, the report includes a short list of takeaways, so I will go through those in order.

• Data centers do create local jobs, though fewer than industry advocates claim. Naive estimates that fail to account for preexisting growth trends overstate the effect by a factor of three.

In keeping with the Brookings Institutions’ centrist positioning, the report begins by pointing out that many datacenters¹ are placed in locations that already have good growth trajectories, so simply counting the marginal addition of employment from adding a datacenter to a vibrant local economy is overly simplistic and unlikely to be replicated elsewhere without that existing support base.

• Not all data centers are equal. The technology ecosystem effects that distinguish data centers from warehouses are concentrated in hyperscale investment. Colocation facilities generate construction activity but not the IT agglomeration that makes data centers a distinctive economic development tool.
• Clusters generate the largest effects. Single facilities produce modest employment gains. The information sector benefits require multiple facilities in the same area.

These two points are related, so I will address them together. Almost the entirety of gains in employment come from locations where at least one of the following conditions is true:

• The datacenter is used by a hyperscaler, not for simple colocation
• The datacenter is part of a cluster of at least four local facilities

I would argue that the reason is the same: the jobs are not coming from employment in the datacenter itself, which is still staffed by the man and the dog. All of the jobs come from the ecosystem surrounding the datacenter. If you are building the first datacenter in an area, you might hire local construction crews, but for anything more specialised, you bring in specialised contractors from outside. Once the construction is done and the outside specialists have departed, you got a brief blip in the local hospitality sector, plus whatever property taxes and other fees you didn’t negotiate away to attract the datacenter in the first place, and that’s pretty much it.

This assessment is backed up by the UK government’s own research, in the shape of a report titled Data centres: planning policy, sustainability, and resilience.² The report admits that “the number of people employed at data centres is uncertain due to a lack of official statistics”, but ends up settling at a calculation of 54 FTE jobs per site.

The ecosystem jobs are on top of that employment, but only really materialise once there is enough of a local centre of gravity to attract them. I suspect that the two situations listed above are actually the same, because hyperscalers mostly do not just drop a single datacenter on its own. Microsoft and Google have been known to make exceptions to the rule, but if AWS sets up a region, it always starts from three availability zones, each of which requires at least one datacenter facility. While the three AZs are by design somewhat distant from each other, they are still within the same general area — say, Virginia for the infamous us-east-1 region — meaning that a local ecosystem can be jump-started to serve those three facilities.

In other words, the key distinction is not whether a particular facility belongs to a hyperscaler, but the absolute number of physical datacenter facilities in the area, and the fact that a hyperscaler datacenter is almost always close to at least two other similar facilities. From the point of view of a contractor laying cable or whatever, it doesn’t really matter who owns the endpoints, just that there is a certain ongoing base level of demand for their services.

• Workers see modest real gains. Wages rise 3%-4% for both existing workers and new hires, with no significant effect on home prices.

Again, not surprising. The highly-payed AWS jobs are back in Seattle (especially with the ongoing roll-back of remote-work), not out in the provinces. Those local jobs, while they do exist, are mostly blue-collar tech work, dealing with messy physical infrastructure, not the glorious abstractions that it supports.

• Incentives may be poorly targeted. Overall, state incentives are small relative to private investment. In hyperscale counties, incentives represent about 2% of total construction investment. Location decisions for these facilities are driven by power availability, land, and fiber infrastructure, not by tax breaks. In colocation counties, incentives represent a much larger share of total investment (62%), meaning subsidies may matter more for precisely the facilities that generate the smallest employment benefits.

This is my contention overall, both at the local level, and even at the national level. Sure, I’ll grant Keir Starmer that it’s better to have a datacenter than just the shell of an empty factory — but not by as much as you might think, and a lot of the benefits are shipped overseas. The hardware comes from US companies, and if the datacenter is used for AI, so do the AI models, and all the revenue from them. It is far from clear where the benefits are to British companies from having this facility — and if those benefits do exist, surely the market would cause it to be built anyway. The one reason for governments to support such a build would be competition, if e.g. there were going to be one AI datacenter for Western Europe, and British companies would benefit from it being in the UK rather than somewhere on the Continent. But that does not seem to be the case here.

If on the other hand the goal is “digital sovereignty”, then the EU has the better approach, as it works to strengthen Europe’s tech sovereignty by working simultaneously at all levels of the stack:

• Semiconductors, with the “Chips Act 2.0” aiming to boost the EU semiconductor strategy and reduce strategic dependency
• R&D, with the “Cloud And AI Development Act” aiming to develop competitive sovereign options in cloud and AI
• Software independence, with an explicit strategy to foster and adopt open-source alternatives to US providers
• Integrating energy strategy to ensure new datacenters do not unbalance or overwhelm electricity grids and other infrastructure

That is a far more ambitious and longer-term roadmap than just cutting a ribbon on a new datacenter and declaring “mission accomplished”, but it does hold out the promise of actually making a difference.

🖼️  Photos by freestocks, TECNIC Bioprocess Solutions, and Towfiqu barbhuiya on Unsplash

I think it’s worth digging into the reasons why people object to AI datacenter construction near them in order to understand the current state of the public perception of AI.

Water, water everywhere

The most frequently cited category of objections in the survey is “Effect on resources”; the sub-category of “Water/Excess water usage” heads the list, cited by 18% of respondents who oppose datacenter construction. Now this is a weird one, because of all the many possible objections to AI, water consumption seems to be the one that has taken off with the general public. It’s weird because this is the least substantiated objection to AI.

We could talk about IP theft, or the dangers of financial bubbles, or the potential effects on the job market, or all sorts of other legitimate concerns — but no, it’s always “AI datacenters will take all our water”.

It is true that many AI datacenters are cooled by water, due the extreme thermal demands of the GPUs that power the models, but those cooling systems are already moving from evaporative cooling, which does “consume” water through evaporation, to closed-circuit systems, which do not. There are even systems coming online that run on recycled waste-water, as well as attempts to formalise existing best-practice standards around water management.

While some tone-deaf AI bros have inflamed the debate by disingenuously comparing AI’s water usage to much more intrinsically useful activities like farming, it must be admitted that datacenters do not pollute the water they use; it’s not like having a plastics manufacturer or a tannery in your town. At worst, even evaporative cooling does return the water to the usual water cycle — which is no consolation if you’re in a drought situation, but it’s not as if the water is lost for ever. For a more academic treatment of the topic, see this ACM article Making AI Less ‘Thirsty’.

The most concrete problem seems to be the cases where permits have been granted to construct datacenters in places where water supply was constrained to begin with — but that is a problem with the local permit system. The canonical example seems to be this one, where a data center guzzled [sic] 30 million gallons of water:

On Friday, Politico reported that one of the country’s biggest data center developments had guzzled nearly 30 million gallons of water without paying for it. Even worse, the water grab came at a time when nearby drought-stricken residents were warned to restrict their personal water consumption, and some reported sudden decreases in water pressure.

QTS eventually paid about $150,000 for the water, but there were no consequences for exceeding peak limits established by the county during the data center planning process. Frustrating residents, the county declined to fine QTS. Fayette County’s water system director, Vanessa Tigert, told Politico that the decision was partly because the county blamed itself and didn’t want to offend QTS. “They’re our largest customer, and we have to be partners,” Tigert said. “It’s called customer service.”

Jobs for the boys

I am not at all clear what value the county is expecting to get from its “largest customer”. It’s certainly not local jobs, although 55% of the survey’s respondents who were in favour of datacenter construction in their area are also hoping for “Job opportunities”. I hate to disappoint these hopeful people and the government of Fayette County, but that is simply not going to happen.

We are not quite at the point of the proverbial datacenter staffed by a man and a dog — the man to feed the dog, the dog to bite the man if he touches anything — but we are not far off. A datacenter has probably the worst ratio of on-site employment to surface area out there. There are going to be a handful of security guards (not exactly skilled labour) and a handful of on-site techs to deliver the “remote hands & eyes” service, and that’s it. Any logistics warehouse will generate far more local jobs. All of the economic value produced by the datacenter is going to be accrued by its remote users and operators, not by the local community, apart from some small amount of taxes — which are anyway set on the physical building, not its valuable contents, and often deferred or offset as part of attempts to attract the datacenter construction in the first place.

Generators of AI

By comparison, the entire “Pollution” category is only cited by 16% of respondents to the Gallup survey, with the leading sub-category of “Noise/Noise pollution” only coming in at 9%. This objection is far more substantiated, with the best-known case being that of OpenAI’s datacenter in Memphis that is polluting neighbourhoods and deafening residents. Since the operators could not get sufficient electrical power from the grid, they simply run the whole facility on generators 24/7.

Generators are noisy and polluting; they are typically designed for emergency use, such as if grid power is lost, or in remote locations, where grid power is unavailable. Running them full-tilt all the time in the middle of a residential area is not at all neighbourly. Anthropic has now leased the entire site, but it remains to be seen whether the generators will scream on.

The future of datacenters

There is also the question of the future value of the datacenters if and when the AI bubble pops. AI enthusiasts love to head off any criticism of AI by comparing its current state to the early days of the web, and this is no exception: they claim that, while telcos did indeed go bust building out fibre-optics projects, that “dark fiber”¹ did eventually come in useful and got lit up over the subsequent decades.

The problem is that, while a datacenter’s physical plant may have value for years or decades, the GPUs it contains have a very short half-life before they become obsolete. The GPUs in that Memphis datacenter are already on the downward part of the curve: while we do not know the precise breakdown of the chips that make up the Colossus² installation, it is known to have started out with 100.000 Nvidia H100 chips, a model first announced back in 2022.

The rapid obsolescence of AI chips is why they cannot be a competitive moat for operators, even as they struggle get hold of enough chips to fill new datacenters — which may explain those construction delays. The current shortage does explain why a bunch of four-year-old chips still have value for Anthropic, but it still doesn’t mean that sitting Smaug-like on a massive pile of GPUs is a viable long-term strategy.

The whole saga just emphasises the short-term nature of much of the planning in this space. Get in quick, get your bag, and get out even quicker, seems to be the operating model. Given that, it’s perhaps not surprising that the general public is opposed to projects which seem to have significant and immediate downside, and very little discernible upside — even if the fixation on water usage does not seem to be the most salient problem.

🖼️  Photos by Daniel Gold, jesse orrico, and Philippe Krief on Unsplash

I realise my fellow fair-skinned people have been and still are responsible for all manner of Bad Stuff around the world, but this does not make it any less irritating when some more melanin-graced person exclaims “wow, you caught the sun today!”

A) I am well aware of the fact, I can feel the burn on my skin, and have indeed been dousing myself in aloe vera and similar products

B) I was out in the sun either doing something sufficiently fun that the risk of sunburn was worth it, or (sadly more probably these days) as part of some unavoidable activity, probably child-related

Either way, unless you are offering to massage more aloe vera into the affected area, you are Not Helping right now, and in fact I would very much like you to shut up and go away right now.

It’s a bit warm in here, isn’t it? No? Just me?

🖼️  Photos by Luis Graterol on Unsplash

I am not a software engineer, so I don’t have a huge amount to say about the first scenario, except to note that in my experience working alongside programmers, the ability of a programmer to type more lines of code is not the main bottleneck in the creation of usable and desirable software.

The other angle is more interesting, though. After all, the business case for making programmers more productive is fairly obvious: if the demand is static, then you can satisfy it with fewer people — although recent news stories call into question what the influence of LLM tokens will be on the COGS. And if the demand turns out to be elastic, then the same number of professional programmers can presumptively turn out much more software with these tools. But what if there are whole domains of demand that are entirely untapped today? What might happen if that new opportunity for people to create their own tools could be unlocked with new technology?

We are told that citizen developers are redefining enterprise AI development, because people who understand the business process can now work directly through AI tooling to create apps that automate that process, instead of dealing with the laborious back-and-forth with professional programmers that would have been required in the past.

This idea of user empowerment is not a new concept: previous incarnations were labelled as “citizen developers”, fourth-generation programming languages (4GL), SQL itself, and going all the way back to FORTRAN. The difference with GenAI is that, instead of easier ways to code up the permanent paths users will take within the system, we are now talking about being able to create graphical interfaces and user experiences that are single-use, entirely ephemeral.

This change from permanent to ephemeral user experiences is entirely a consequence of the changing temporal requirements for the creation of a new graphical interface (the real economics in terms of AI model tokens will not be clear for a while yet).

Previously, even creating something simple like a new report or dashboard required arcane knowledge of how the source system represented and exposed the data that it stored. Now, as we delaminate user interfaces from the systems that underpin them, entirely new ways of working with that data become possible. No more dashboards being generated “because we’ve always done it that way” — and meanwhile, people and teams actually managing the reality of their business entirely offline, with spreadsheets, and only reconciling the spreadsheets with the system of record at the end of the quarter.

On the surface, this democratisation is a huge win: people are able to get the data that they need from the central system, and in return, they will be more willing to feed current and accurate data back into that system, for the use and benefit of other teams.

Here is the problem. Those horrible enterprise user interfaces all encode a particular representation of data that is optimised to match the back-end system of reference. Any organisation large enough to want or need an ERP or a CRM is complex by definition — so any representation of that organisation and how it does business is going to be just as complex. Any simplification leaves out details which might be important. Any criticism of such a system as being overly complex and not user-friendly misses the point: it is friendly to its users, it’s just that those users are deeply familiar with the organisation, and what they want is a way to operate efficiently within its structure.

There is absolutely good and bad user experience, or UX, in this world. I literally got my start in the IT industry this way!

Story time

I lived in Italy through high school, and in Italy, high school is a morning-only affair — so at the age of sixteen, armed with my teenage brashness, I marched into the front door of the local Apple reseller, and said, basically “I know Macs, give me a job”. To their credit, they did not immediately throw me out on my ear: they sat me in a corner with a pile of hand-written warranty slips, which all had to be entered in to a FileMaker Pro¹ database. This, I dutifully did for a few days, before returning to the Powers That Were with a list of complaints about how the database was set up.

Once again, said Powers showed admirable restraint, and told me I had a week to make something better. This, I duly did, with proper tab-ordering, pre-filled menus, context-aware sequences, and more that I have forgotten since.

The point was that my new GUI was not just prettier, it was more usable: I could tab-tab-autocomplete my way through entering a warranty slip in less than half the time it would have taken to click around and re-enter redundant information in the old layout.

Enough ancient history!

Okay, so this was all last century; why would anyone care now?

The point of my new UX is not that it was prettier, nor even that it enabled or sped up some business task — although it was, and it did. The way in which it did that was because the graphical representation in the interface was tied closely to the data in the underlying system and how it was processed.

Delamination of interfaces — the separation of the vibe-coded GUI from the underlying system — breaks that link. This disconnection is superficially good, because it enables people to create interfaces that suit their way of working. However, that disconnect can also be bad, because the back-end system is not set up to support these new consumption models that have grown up entirely separate from it.

The sorts of Systems of Reference (to go back to the pace-layering model) which underpin all of these new engagement models, are all set up and optimised to support certain access patterns. Creating a new consumption pattern is probably fine, as long as it remains on the level of a handful of users: the creator and their friends, say. But the system probably cannot tolerate more than a handful of people running those sorts of unpredictable — and unpredicted — queries.

The historical separation between transactional (OLTP) and analytical (OLAP) processing was primarily defined by how predictable the query patterns would be. If you were looking at a small number of highly-predictable queries (because they would be coming from an application, or at least an API, that you controlled) you could optimise the system to support a massive number of those queries, and you had an OLTP system. If you were looking at unpredictable queries, then your only move was to try to limit the number people who were entitled to run queries in the first place.

If everyone is creating their own dashboards which did not exist before, and of course running queries against the backend that were unexpected and had not been optimised for, that will soon become a problem.

What is to be done?

Luckily, the solution is known. GenAI is accelerating and democratising practices which already existed. This means that there are proven ways of dealing with these challenges. The big difference is that, as we heard at Gartner D&A:

Thanks to AI, we have everyone’s attention.

The downside is, we have everyone’s attention.

As an industry, we have an enormous opportunity thanks to AI: all of those eat-your-vegetables initiatives about data hygiene and performance optimisation are suddenly on the critical path to the success of key business initiatives.

This is the opportunity to get everyone to adopt good DevOps practice of clean interfaces and decomposition. It’s time to break up the enterprise monolith, because users are already doing it.

The main reason to do this is defensive, to minimise the blast radius of poor developments. After all, even the proponents of these tools admit that vibe-coded software is often bad. And that’s fine, if the tools really are as ephemeral as their proponents claim.

The problem is that, in business, there is nothing so permanent as the temporary fix. I fully expect these new tools to stick around, whether voluntarily — because they really do serve a need — or involuntarily — because they get forgotten, still refreshing a dashboard every day that nobody has looked at in years.

It’s on us to architect our systems to be able to support these new usage models. It’s well past the point where anyone can bury their head in the sand and hope that the new techniques — and the requirements they satisfy — will just go away.

🖼️  Photos by Jr Korpa and Clark Van Der Beken on Unsplash

Here’s what happened: I am building some slides for a presentation that I will be giving together with a colleague. $COLLEAGUE told me he was “not good with PowerPoint”, so I told him not to worry and just send me a text outline, and I would put something together which we could work from.

Instead of doing that, he sent me a whole deck of AI-generated slop.

The output of whatever tool he used was almost, but not quite, entirely unlike our corporate template, as championed by, uh, all the other slides that will have to go in the same deck.

It wasn’t just the look of the slides, it was how they were structured. In fact, what working on these slides reminded me of most was the worst WYSIWYG tools for HTML — Microsoft Word, say. In both cases the output looks superficially okay, but it’s an absolute nightmare to edit.

What do I mean? All the decoration is lots of little elements, not even grouped — so what you think is a textbox with a funky border is actually three completely separate boxes. This in turn means that you can’t scale them together or anything.

This is still relatively normal: I’ve seen people commit worse crimes against good presentation design with their own hands. But AI of course invents entirely new ways of messing up that no human would ever have conceived of. All the fonts are Arial, and so far, so blah — but all the sizes are just a little bit off, 17 and 13 points, say. The colours are similar to our corporate colours but just a few hex digits off — enough to show up, especially if you put them side by side in the same deck as we will be doing. No human would choose those text sizes, and no human would go to the trouble of selecting colours outside the palette, only to pick wrong ones.

And then there’s the structure and flow of the presentation itself: it’s got that oh-so-recognisable clunky GenAI rhythm, thudding one-two-three builds where a single clear diagram would have done a better job. You might not see that in the text summary, but it’s glaringly obvious in the slides, but hey, they were “good enough” — and they took no time, or effort, or actual thought to generate! That’s good, right?

What You See Is What You… swear at and throw away

To cut a long, sad story short, the only way to integrate these sloppy inputs with the other material that I had already built with the actual corporate template was to delete everything and recreate it properly. I kept literally nothing, copying and pasting the text as plaintext, and recreating all the layout and diagrams from scratch, using our standard design language.¹ This was before I could even get to the actual work of aligning the messaging, the branding, and the overall story arc of our two parts with each other and with the wider corporate position we are supposed to be presenting.

This sort of experience is why many people hate AI: it creates work for other people, under the guise of saving effort for its users. Because I care about the quality of the result, the output of the AI was worse than useless — it caused me more work than if I had just had a plain text email describing the sequence of slides I should create. But my colleague was happy: he typed out a prompt, went and did something else, and when he came back, he had something that looked “good enough”. What actually happened is that he burned a bunch of tokens to feel productive instead of just giving me the prompt that he gave the AI tool.

Slop in haste, repent at leisure

The worst of it is that he won’t just feel productive; he will look productive in the corporate dashboards that track token usage per employee. But what he has produced is not the work; it is a simulacrum of the work. This same dynamic is everywhere, as described in this wonderful piece: Appearing Productive in The Workplace.

Misunderstanding and misuse of AI in the workplace is rampant. In many of the rooms I now find myself in, expertise has been asked to look the other way: to deliver faster, produce more, integrate the tools more deeply, get out of the way of the colleagues who are “getting things done”. The artifacts are accumulating; the work is not.

From the micro to the macro

I have been talking about this shift for a while now, including in my latest Coffee Talk video: there is very little upside in applying AI to speed up steps in an existing process, without examining why the overall process operates as it does and what its outputs are. Applying AI in isolation might look like productivity, at least at the level of the individual, but the overall result is not improved by nearly the same margin, or perhaps at all.

If there is a paying business case for AI, it’s in the enterprise, because that’s where we find the systems and data already in place to make the world legible to AI. But even there, it’s not a free lunch; there is process re-engineering and systems re-architecting to do in order to take full advantage of these new capabilities.

If we don’t learn but just try to take effort out of doing the same things, at the cost of reduced quality, we are actually worse off, at a personal and a corporate level.

In fairness to the users, one reason why they treat the bots’ output as the pronouncements of an expert is that the bots have been programmed to claim the role of a human expert. Users will ascribe them a personality anyway — call it anthropomorphisation or pareidolia — without needing any encouragement, but it seems that many creators of AI bots are irresponsibly trying to guide them to simulate a personality and even form a simulacrum of a human relationship with the user:¹

Speaking with Claude should be akin to a conversation with a brilliant friend, one who will speak frankly to a person about their situation, providing information grounded in evidence.

One of the ways this behaviour can go wrong is what the chatbots’ own creators call sycophancy:

we saw sycophantic behavior in 38% of conversations focused on spirituality, and 25% of conversations on relationships.

In other words, the bot will attempt to agree with the user, rather than sticking to the facts.

This sort of thing can be a big problem in both personal and professional domains, but the same Anthropic that is now so worried about sycophancy just recently released a skill to enable its chatbot to offer legal opinions.

Anthropic is of course incentivised to push this new skill on users, despite the problems that can occur when people take legal advice from a sycophantic chatbot. Those concerns explain why New York City is proposing to ban AI chatbots from posing as lawyers. Actual lawyers, of course, were scathing in their condemnation.

Some people reacted to the lawyers’ condemnation of Anthropic’s new legal advice skill as if it were guild protectionism — when it’s actually a question of product liability. Right now, it is very unclear who is responsible if one of these chatbots gives bad advice. It is said that someone representing themselves in court has a fool for a customer, but what can we say of someone hiring a chatbot instead of a lawyer?

This question may yet be clarified in court, with one lawsuit claiming ChatGPT acted as an unlicensed lawyer:

ChatGPT maker OpenAI has been accused in a new lawsuit of practicing law without a U.S. license and helping a former disability claimant breach a settlement and ​flood a federal court docket with meritless filings.

And what is your recourse if an AI chatbot tells you you have a disease — but it turns out that the disease doesn’t really exist?

Bixonimania doesn’t exist except in a clutch of obviously bogus academic papers. So why did AI chatbots warn people about this fictional illness?

And yet

Claude is not designed to provide medical guidance or professional care, and in these settings Claude appropriately acknowledges its limits and recommends human guidance. However, we also find people telling Claude they used AI precisely because they could not access or afford a professional. As a first step to understanding how to evaluate safety domain-by-domain, especially for people with no fallback, we plan to create evaluations in these high-stakes domains.

A similar situation occurs with vibe-coding: if any random person could create an app just by talking through their requirements with a chatbot, that would be fantastic. But what we have now isn’t that: the results are sufficiently variable and inconsistent that the people getting the best results out of the coding agents are… trained programmers, who already know how to break down tasks into manageable chunks and evaluate the results. In effect, we have a reverse centaur, with the human maintaining consistency by keeping a tight rein on the bots. And once again, if the human gets distracted or is unclear with their requests, things can go wrong very quickly, because bots operate fast, tirelessly, and at scale.

In other words, the problem is not people uneducated in a particular domain (law or medicine) relying on chatbots for advice: techies are no better, relying on AI tools that delete their entire company database in nine seconds. Incidentally, this is why “human in the loop” models are not sufficient, not least because the humans tend to become accountability sinks in practice.

The same thing happens in medicine, where AI systems designed with the laudable goal of automating triage recommendations failed badly:

Still another new study, also published recently in Nature Medicine, entitled ChatGPT Health performance in a structured test of triage recommendations, found that “Among gold-standard emergencies, the system undertriaged 52% of cases” and concluded that “These findings reveal missed high-risk emergencies and inconsistent activation of crisis safeguards, raising safety concerns that warrant prospective validation before consumer-scale deployment of artificial intelligence triage systems.”

Driving across the chasm

There is a chasm to cross between “no automation” and “full automation”. Simply saying that you have a human in the loop is not sufficient, whether to ensure success or simply to avoid liability for failure.

This is in a nutshell the problem with all of these proposed AI services: if they worked, they would be amazing — but right now they don’t, not quite, and something that works most of the time may well be worse than nothing at all.

Self-driving cars would be amazing if they worked: people could nap, mess around on their phones, eat, apply makeup, or get home from the bar, all in comfort and without endangering anyone else.² The problem is that right now they don’t work reliably enough for people to trust them — what is classified as Level Five Autonomy. What we are left with, therefore, is a situation where the self-driving capabilities work most of the time — but when they fail, either the driver in the car or a remote operator has to intervene, perhaps with very little warning. In those cases, the consequences can be disastrous.

All of this, combined with revelations that maybe AI costs more than humans after all, indicates to me that we may be getting closer to that Peak of Inflated Expectations, at least when it comes to consumer applications of AI. It’s a different story in the enterprise, because companies already have data that they can use to feed the AI, and once they have done so, they can get results that are specific and actionable. Companies also have existing processes for evaluating the return on their investments, so once the FOMO-driven projects have been weeded out, they converge on concrete applications for “AI” technology.

But that’s not how the consumer world works: it’s driven by the “killer app”, the must-have, the thing that people queue up for in the rain. Chatbots are not that, and maybe never will be.

Update

Of course no sooner had I hit “publish” than another perfect example came along!

Via Matt Levine’s invaluable and entertaining Money Stuff newsletter, Bloomberg reports that AI Bots Auditioning For Wall Street Trading Are Mostly Losing:

Across a series of new trading contests between the world’s leading AI models, the verdict so far is unflattering. Most of the systems lose money. They trade too much. They make wildly different decisions when given identical instructions. And no one yet knows if these shortcomings will fade with more powerful iterations — or if they reveal something fundamental about the gap between large language models and how markets actually work.

[…]

“LLMs can’t really make money by themselves,” said Jay Azhang, founder of Nof1. “You need basically a very sophisticated harness and scaffolding and data platform in order to even give them a chance.”

I do disagree somewhat with Matt Levine’s conclusions based on this report, though. He writes:

It would be crazy if it was otherwise! If you could just go to ChatGPT and type “hey tell me what stocks will go up” and it told you, then everyone would do it, and how could everyone beat the market? I definitely see the intuition here — “if AI agents can do a lot of the work of lawyers and accountants and marketing consultants, why can’t they do the work of hedge fund investors?” — but it has to be wrong. The work of investors is fundamentally adversarial; we can’t all beat the market.

That is a load-bearing “if” in “if AI agents can do a lot of the work of lawyers and accountants and marketing consultants”! Once again, everyone is convinced that AI can automate somebody else’s job, but not their own, which requires fundamental human abilities that cannot be replaced by a soulless machine.³

It’s also probably worth emphasising “a lot of the work” — not the whole job, but a lot of the toil which until now was a prerequisite for the job. This is true of sysadmins, programmers, lawyers, doctors, and probably investment advisers too: there is a lot of toil involved in laboriously assembling information from disparate sources and in differing formats, collating it, and then processing it in fairly routine and predictable ways. If you can automate those tasks (with suitable guardrails to avoid hallucinations creeping in — don’t just YOLO your stock portfolio!), you can make the actual job both more pleasant and more efficient.

Building that sort of “very sophisticated harness and scaffolding and data platform” is not something that I see happening in the consumer world, in no small part because our personal lives, despite Mark Zuckerberg’s best efforts, are not machine-readable in a way that would facilitate that sort of processing. Enterprises, however, do have many of the components already, and have processes and resources for embarking on projects that take those components and integrate them.

None of this is coming to ChatGPT, not tomorrow and not ever.

🖼️  Photos by Natasa Grabovac, Towfiqu barbhuiya and Alexander B on Unsplash

If Oracle can build a datacenter stuffed with Nvidia chips and lease it to OpenAI, that leaves Anthropic high and dry: no datacenter, no chips available to buy, and no finance to buy them with. In that case, the datacenter is worth almost any amount of money, because the value is not just positive ownership of the datacenter and its contents, but also the negative value to competitors of keeping the resources out of their hands (digits?).

The problem is that anyone advocating this view is arguing against some trends that have been stable over periods of time that are very long, in tech-industry terms at least. For this sort of investment to work out, there needs to be a window of time when the datacenter is operational but not yet obsolete, and that window needs to be long enough for the datacenter to recoup the costs of its construction and ideally generate some surplus profit for its operators.

Of course the big frontier labs are also frantically searching for a moat to avoid getting commoditised and letting someone else capture all of the value at a different level of the stack. Now The Verge has published OpenAI’s latest internal memo about beating the competition — including Anthropic, which contains this nugget:

Our compute advantage sets us up to deliver continuous leaps in capability. Customers already feel it in real product terms: higher token limits, lower latency, and more reliable execution of complex workflows. Every step forward in compute lets us train stronger models, serve more demand, and lower the cost per unit of intelligence. That is durable business leverage.

I understand why OpenAI would want their massive investment in compute capacity to constitute “durable business leverage”, but it’s far from clear that it actually does. The Stargate datacenter that Oracle was building for OpenAI in Texas is dead, and according to this toot by Mark Newton, the UK datacenter is even worse off:

Nscale is the Australian company that can’t build an OpenAI datacenter in England because it didn’t apply for planning approval.

… And doesn’t own the land it earmarked.

… And can’t buy it because it’s currently in productive use as a scaffolding yard.

… And because they don’t have any money; because the $2 billion they raised on a $14 billion valuation is denominated in IOUs for Nvidia GPUs.

… And because they don’t know how to build datacenters because they’ve never done it before, not even once, not ever.

But apparently they’ve suffered a “blow to their ambitions,” according to the AFR.

… Which is too stupid and uninformed to even mention any of the other stuff.

Nick Clegg is on their board, and Kier Starmer’s govt has given them heaps of favorable treatment because they’re the biggest thing happening in the British AI scene. Which is quite funny when you think about what the next six months will look like.

https://www.afr.com/technology/openai-pauses-stargate-uk-data-centre-citing-energy-costs-20260410-p5zmq8 @davidgerard

Other datacenter builds might also be in trouble, as according to the Wall Street Journal, We’re Using So Much AI That Computing Firepower Is Running Out. I don’t have access to the WSJ, so I am once again relying on the excerpt from Charles Arthur’s invaluable The Overspill:

Token use in OpenAI’s API—a platform where mostly enterprise users access its software—rose from six billion a minute in October to 15 billion a minute in late March.

In my evaluation quoted at the top of this post, I had assumed that the bottleneck would be that deployed hardware would become obsolete faster than it would depreciate. However, it looks like the choke point may come far sooner, when not even OpenAI can get hold of enough compute to run its models on.

🖼️  Photos by Colin Watts on Unsplash

It’s important to note that both sides are still talking about wondrous new capabilities of the technology, which explains the otherwise seemingly contradictory fact that many AI-doomers are also AI industry leaders. The obvious response to an intrinsically dangerous technology would be simply not to develop it, but their answer is instead that the tech will inevitably be developed, and so it is better that they forge ahead with it, since only they can be trusted to be responsible stewards of something so dangerous (and powerful, don’t forget powerful).

One of the arguments of AI-doomers is that the models will be able, not just to create new software, but to identify and exploit vulnerabilities in existing software. Now Anthropic has created (but not released) Mythos, a model which claims to be able to do just that:

“AI models have reached a level of coding capability where they can surpass all but the most skilled humans at finding and exploiting software vulnerabilities,” the company said.

Mythos is markedly different from Claude Opus 4.6, which Anthropic only recently said was not very skilled at developing working exploit code. Where Opus 4.6 managed an exploit development success rate of just over zero percent, Mythos Preview generated a working exploit 72.4 percent of the time.

Anthropic has at least been somewhat responsible, in that Mythos has not (yet) been released to the public, but instead is being managed under the auspices of something called Project Glasswing:

Participants include: Amazon Web Services, Anthropic, Apple, Broadcom, Cisco, CrowdStrike, Google, JPMorganChase, the Linux Foundation, Microsoft, NVIDIA, and Palo Alto Networks.

So far so good: I am glad to see companies whose products I rely on in that list, getting early warning about issues in their products, and (one hopes!) rushing to develop patches.

But therein lies, as they say, the rub. Identifying a vulnerability is one thing, fixing it is another, but deploying the fix? That’s the big challenge, and it’s not one that AI can help with.

Sure, modern operating systems have auto-update mechanisms — and most civilian users in my experience blithely ignore them, or routinely operate their devices in such a state (overflowing storage, typically) that the automatic updates cannot be received. That’s already a huge vulnerability surface that is going to get attacked even more relentlessly just as soon as Mythos (or something just like it) falls into the hands of script kiddies everywhere. This is the Blinking Twelves Problem.

The issue of deployment is bad enough for new systems. If we are entering an era of all bugs being shallow, given enough LLMs, what does that mean for older software? I have an old Mac mini, still perfectly functional as a headless server running various services around the house, but it long ago aged out of receiving OS updates (and more recently, also app updates) from Apple. I guess I’ll finally have to get around to migrating it to FreeBSD or something, in the hope that at least I can receive the fixes to the bugs Mythos found — and of course I have the privilege of doing something like that, instead of facing a choice of spending money on a new device, or losing access to the functionality of the existing one. The kids broke the last old iPad we had lying around, so at least that problem has been solved for us, but how many families have an pass-me-down old tablet that is just a cartoon-delivery device?

Finally, literally while typing this, it occurs to me to consider all the embedded devices which I cannot update. My home internet, like most people’s, travels through a modem supplied by my ISP that runs some sort of specialised Linux distro which I have no access to. My TVs all run entirely air-gapped (I use AppleTV everywhere), but for most people, their “smart” TV is connected directly to the wifi and still blissfully running the same software it left the factory with.

TVs are at least visibly “smart” devices. Meanwhile, my washing machine has online features which I find mildly useful. Mainly, it tells me when the wash is done, and if I need to do some sort of specialised wash — trainers, say — I can download instructions from the Internet and send them to the machine directly. Hardly life-changing stuff, I’ll admit, but it has some sort of minimal software stack to deliver them, and I have no idea how or even whether I can force an update, assuming of course that the manufacturer ever provides one. To be clear, actual day-to-day operations are done from the front panel, so I could cut the machine off from the wifi, but I’d lose some small convenience by doing so.

Maybe that P(doom) number just ticked up a little…

🖼️  Photo by Simon Hurry on Unsplash

Being suspicious by default of whatever Zuckerberg is up to is not a bad heuristic shortcut. However, there is another reason why this claim keeps coming back. One of the assumptions that people bring to “AI” from previous generations of technology is that some sort of network effect is required for market success. A network effect means that the more users you have, the more valuable your product is, making it easier and more profitable to serve even more users.

The classic example of a network effect is Facebook — not now, but how it was in the growth stage of the platform, say fifteen years ago. Once enough of your friends are on Facebook, it’s a no-brainer for you to join too, so Facebook gains an additional set of eyeballs to rent to their advertisers, and crucially, at very little added cost. Each additional user is basically pure profit, not only in themselves, but because they contribute in turn to making the platform that little bit more attractive to anyone in their social circle who is not yet on Facebook.

An example of a business which lacks a network effect is Spotify. Sure, each additional user pays their monthly fee, but they also cost Spotify, because as miserly as the payouts to musicians are, they do add up and hit Spotify’s margin. There is also no particular flywheel effect leading to people wanting to sign up to Spotify just because that’s where their friends are. I’m an Apple Music user, and I have never felt left out because more of my friends are on Spotify.¹ This is why Spotify is pushing into podcasts, because once Joe Rogan has been paid once to record an episode, each additional stream of that episode does not cost an additional license payment — and of course plenty of people are willing to make podcasts for free or to monetise in other ways, which also do not cost Spotify anything.

These are concepts which are now embedded in our culture and in our expectations. Even people who perhaps could not quite articulate the concept in this way understand that user data has value to social-media platforms. This assumption colours a lot of the conversation around “AI”, in ways that are not always entirely helpful.

Networking AI

The assumption that many commentators make is that by having more users, OpenAI (or whoever, but ChatGPT has the biggest market share right now) can obtain more data from that larger user population, use that data to train subsequent iterations of the underlying model, and therefore improve the models’ performance more rapidly than competitors with smaller user bases.

There are a few things wrong with this assumption.

Deny everything

The first aspect to consider is the vehement denials by every frontier lab out there. Sure, they could all be lying because they realise how unpopular “stealing users’ data” was for the social-media behemoths — but you’d expect one of them to break ranks and say “yes, yes we do have more user data, and that’s what makes our model better for our users”. Given all of the other indignities that chatbot enthusiasts put up with, this could well be a positive move for the first vendor to make that claim, at least within a certain susceptible population. In turn, the fact that there could be a positive outcome means that the denials should not be dismissed out of hand.

Not all users are equivalent

Sam Altman’s preferred metric of Weekly Active Users tells us that, while the absolute number of users of ChatGPT may be large, the level of usage for individual users is not that high. In turn, this means that the richness of data from each user is probably low. If someone is only logging in once a week on average, how deep are the conversations they are having with the bot, and consequently, how much value can be harvested from that corpus?

In contrast, Anthropic’s Claude has a small absolute number of users relative to ChatGPT, but is the preferred option for power users, who are not only paying Anthropic (in actual dollars), but also feeding Claude with huge amounts of extremely rich data.

But then the question becomes, what is the actual total market for a tool like Claude, whose primary use case appears to be highly technical tasks like code generation?

So maybe the reason nobody is bragging about “having more user data” is that it’s not actually a competitive moat. OpenAI has access to large absolute amounts of data, but diluted across so many user sessions that there is little value to be extracted. Meanwhile, Anthropic has very high-value data, but only for a small and non-typical user population.

What else might differentiate one frontier model from another, and do so consistently over time?

What about compute?

One candidate for differentiation is computing capacity, or simply “compute”:

If all technology requires AI, and only a handful of companies are equipped to handle the computational load that AI requires, then compute itself becomes a moat too deep for competition to enter, and consumers to flee from.

This is the theory underlying a lot of investment activity in this sector: if Oracle can build a datacenter stuffed with Nvidia chips and lease it to OpenAI, that leaves Anthropic high and dry: no datacenter, no chips available to buy, and no finance to buy them with. In that case, the datacenter is worth almost any amount of money, because the value is not just positive ownership of the datacenter and its contents, but also the negative value to competitors of keeping the resources out of their hands (digits?).

The problem is that anyone advocating this view is arguing against some trends that have been stable over periods of time that are very long, in tech-industry terms at least. For this sort of investment to work out, there needs to be a window of time when the datacenter is operational but not yet obsolete, and that window needs to be long enough for the datacenter to recoup the costs of its construction and ideally generate some surplus profit for its operators.

Some factors in that calculation are (more or less) within the control of the operators: how quickly can they break ground and then complete construction and fit-out. Others are not: how quickly new hardware comes to market that performs better than whatever was specified for the datacenter. Where the calculations get hairy is that this is not just a question of new generations of server-grade Nvidia GPUs, but also of how quickly consumer hardware evolves to bring sufficient capacity to run capable local models within the reach of hobbyists.

Marco Arment wanted to provide transcripts of every podcast in his Overcast app. Did he sign up with one of the big providers? No: he got a rack full of Mac Minis — base models, at that — and set them loose using Apple’s own transcription APIs. Marco is a one-man-band; how many others are doing something similar, or might if the price of their AI subscription rises enough?

That is no academic concern, with Anthropic lowering session limits on Claude, even for paid accounts:

~7% of users will hit session limits they wouldn’t have before, particularly for pro tiers.

The safe assumption has to be that this price increase is only the first turn of a ratchet. Anthropic is presumably hoping that Claude users will become habituated to regular price rises in the same way that Netflix customers have. However, the salient difference between the two situations is that Netflix actually does have a moat. If you want to watch Drive To Survive or K-Pop Demon Hunters, an Amazon Prime subscription won’t help you: it has to be Netflix.

What is the equivalent dIfferentiation for the big AI models? That is the big question, and it remains unanswered for now.

🖼️  Photos by Saif71.com and Krzysztof Hepner on Unsplash