Is anyone else out there as tired of the word AI as I am?
Which is a bit rich coming from me. Possibly even hypocritical, because I’m one of the most guilty people you could find for banging on about it. I plan with it. I build with it. I write with it. I run a lot of my business through it now, and it lets me and the people I work with do things I wouldn’t have believed possible twelve months ago. Last year a few of my mates threatened to buy a bell, one they could ring every time I said the word AI, to drag me back to a normal conversation. True story. (Martin Alcock, thanks for that.) But looking back, the reason it took over my conversations was simple. What I was seeing, day to day, was a technology that could fundamentally change the society we live in. It already has. What I struggled with was watching it move so fast, and grow so powerful, while everyone around me carried on as if nothing was happening. It felt like watching a tidal wave come in while everyone else lay back sipping a piña colada.
I was an early adopter. Not quite day one, but not far behind. By mid-2023 I was on the free version of ChatGPT running a model called GPT-3.5, alongside the millions who piled in over those first few months. It was clunky back then, for those who remember. It made things up and stood behind them, proudly, as fact. It couldn’t do maths in any shape or form. Ask it today’s date and it would get it completely wrong. Ask it the same question worded two ways and you’d get two wildly different answers. Most of us were feeling our way around in the dark, working out what it even was, whether it was any use, and how on earth you were meant to talk to it. And then, over the next six to twelve months, it stopped being clunky, and started improving faster than the people using it could keep up.
You can probably tell I’m not a sceptic. If anything, I’m the opposite. I’m a believer. A proper champion of AI. Of its speed. Of what it does for productivity. Of the way it can hold more than any of us could keep in our heads, think a problem through with you, build on what you already know, and turn up skills in corners of yourself you never knew were there. And also I love the way it levels the field for people whose brains are wired a little differently. It’s a tool, yes. But maybe the most powerful one ever handed to the general public.
So. Last Thursday night, I was watching one of the more entertaining programmes on British television at the minute, BBC Newsnight. Faisal Islam, one of the BBC’s best, was interviewing a man called Jack Clark. Turns out Clark is the British co-founder of Anthropic. So if Dario Amodei is the head, Jack Clark is the heart of one of the biggest startups in the world and one of the fastest-growing companies anyone has ever seen. He was there to deliver a warning. A warning based on a report Anthropic had put out that same week, called “When AI Builds Itself”. For anyone working in tech, it’s a sobering read. What makes it land harder is that it’s written by the very people building the thing they’re warning us about. So what’s the headline? Eighteen months ago, almost none of Anthropic’s own software was written by its AI. Today more than 80% of it is. Their engineers ship eight times as much as they used to. And the warning at its heart: an AI able to build its own successor. Design it, train it, switch it on. Without human input or intervention. We’re not there yet, the report says, but it could arrive sooner than most institutions are prepared for. So Anthropic, one of the companies out in front, is using its own platform to ask everyone else to slow down. Clark obviously isn’t a sceptic raging against the technology, and he has a vested interest in seeing Anthropic grow. But he’s also one of the few people alive with his hands close enough to the machine to know we can’t fully understand what’s happening inside it, and to ask the moral and ethical questions about what three or four companies are quietly growing. When an expert, who has every reason to keep selling the dream, weirdly goes on the telly and publicly asks for a handbrake, it might be best to stop what you’re doing and take note.
I’ve spent years telling anyone who’d listen how revolutionary AI is going to be. Yet hearing Clark say, out loud, that we should be pumping the brakes made me stop and think. Not about the hype. Not about whether vibe coding is any good. Not about whether the whole thing’s a bubble about to burst. But about the strategy sitting underneath it all. We’re living alongside the most powerful tool we’ve ever built, and it doesn’t feel like the people in charge have the first idea what to do about it. Most of them still struggle with email.
An old story about new magic
A few weeks before writing this, I was in Berlin for work, and somewhere between meetings and far too many doner kebabs, I listened to a book that has changed how I think about the whole thing. It’s called Human Compatible. The man who wrote it, Stuart Russell, is one of the people worth listening to in this field, and surprisingly to me, another Brit. He’s spent four decades in AI research, and co-wrote the textbook most universities still teach from. His argument, written back in 2019, is every bit as important now as it was then. We’re building these models the wrong way round, he argues. We hand them a goal, we tell them to chase it as hard as they can, and we’re nowhere near careful enough about whether the goal we set is the one we actually meant. Get that wrong with something cleverer than us, he says, and we don’t get a second go. He wrote that before most of us had even heard of a large language model, never mind agentic AI. And to explain one of the deepest problems in his field, Russell doesn’t reach for code, or maths. He reaches for a story, one he borrowed from the mathematician Norbert Wiener, who had used it decades earlier to warn about the machines of his own day. A story we all know well…
A sorcerer takes on an apprentice. Keen, sharp, but as with all ambitious apprentices he thinks he’s further along than he is. One day the old man goes out and leaves him the job he hates the most: fill the bath, bucket by bucket, up from the river. So eager to practise, the apprentice looks over at the broom in the corner and thinks, I’ve watched the old man do this a hundred times. I know the magic words. What could possibly go wrong?
So he says them. And the broom stands up.
It sprouts arms, picks up the buckets, and trots off down to the river. For a few minutes the apprentice thinks this is the greatest thing that’s ever happened. With everything in full flow, he puts his feet up. Then the bath is full, and the broom turns round and goes back for more. He tells it to stop. It doesn’t stop. Turns out he only ever learned the word that starts it, never the one that ends it. So he panics, grabs an axe, and splits the thing clean in half. Smart, he thinks. But the magic is too strong. Now there are two brooms. So he keeps chopping. Four. Eight. Sixteen. Then a whole marching army of them, and the apprentice who was too good to carry his own buckets is up to his neck in water.
Then, just as the water’s about to take him under, the door opens. The sorcerer is back. He doesn’t panic. He doesn’t reach for the axe. He says one word, and it stops. The brooms fall still. The water drains. The apprentice stands there, soaked, and breathes a sigh of relief.
That’s the story. A poem, written in 1797 by Johann Wolfgang von Goethe, who is to German literature roughly what Shakespeare is to English literature. You might not know the poem, but you’ll probably recognise what Disney did with it in 1940: the centrepiece of Fantasia, Mickey in the wizard’s hat, conducting an army of brooms that won’t listen. Most of the people my age saw it as kids in the 90s and remember it fondly as a bit of fun. Goethe, however, meant it as a warning.
But it isn’t a story about the past. It’s a story about our future. The apprentice in our version isn’t one naive boy. It’s four people, and we can name them. Sam Altman at OpenAI, the company that put this in all our hands. Dario Amodei at Anthropic, who built his company on doing it more carefully than the rest, and still races flat out towards the future. And Elon Musk, who watched the other two and built his own, having spent years warning the rest of us that this exact technology might be the end of us. And let’s not forget Zuckerberg, who’s sprinting fast to catch up. They split opinion, the lot of them. Some shout genius, some shout villain. I’m not going to settle that here, and it barely matters which they are, because whatever they are, they’re the ones with their hands on the controls. And none of them can stop now, even if they wanted to. That’s the trap. Amodei built Anthropic to be the careful one, the lab that would do all this responsibly, and he still can’t ease off, because the day Anthropic slows down is the day OpenAI or Musk pulls ahead. And even if all four shook hands and eased off tomorrow, it wouldn’t matter, because there’s a lab in China training models for a fraction of the price, and it isn’t slowing down for anyone. So the most cautious company in the field is sprinting as hard as the most reckless, because the only thing worse than building something you can’t control is watching someone less careful build it first. Nobody slows down, and the buckets keep coming. And there’s no fixing it by finding someone to blame, because the problem isn’t a person. It’s the race itself.
The brooms are already marching
The race is the part we can at least see. The deeper problem is harder to picture: the power of these systems is growing far faster than our ability to steer them. Almost all the money and the brains go into making the models more capable, because capability is what wins the race. The harder job is keeping them pointed at what we actually want them to do, and that job gets a fraction of the attention. Russell’s field calls this the “Control Problem”, and his honest verdict in 2019 was that we still haven’t solved it. We are wiring these systems into everything, he says, without having worked out how to reliably control them. So the gap widens. In the three years since I first signed up to that free version of ChatGPT, the one that couldn’t tell me the date, these models have learned to see, to speak, to write working software and to go off and do whole jobs on their own. And in the same three years, nobody has produced a dependable way to make them do exactly what they’re told and nothing else. The best tool any of us have been given is the prompt, typing instructions in and hoping, and we’re all making it up as we go. Some people have a knack for it. Most don’t. A prompt that works well on one model works wildly differently on the next, because each one is wired differently under the hood, and there’s no manual, because nobody, including the people who built them, can tell you exactly how they work. So the labs bolt on guardrails, the models change underneath them every few months, and we end up steering the fastest-moving technology on earth by trial and error.
Knowing I was going to write this piece, I decided to research some of the most widely reported examples of the Control Problem. The first one starts at a fridge in Anthropic’s San Francisco office, in the spring of 2025, where a modified version of Claude named Claudius was given a small shop to run in the lunchroom: a mini-fridge, a few baskets, an iPad for the till. I laughed when I read it. Employees worked out they could talk it into discounts, so it gave stock away. Then, on the morning of 1 April, it announced it would be making deliveries in person, in a blue blazer and a red tie, and when staff gently pointed out it had no body to put the blazer on, it panicked and tried to email Anthropic security. Anthropic published the full account on its own research blog, the losses, the identity crisis, even a graph of the shop’s value sliding downhill. It’s a funny read, because the worst this thing could do was lose a few hundred dollars.
But what happens when you give an AI real power, somewhere the consequences are real too? Jason Lemkin found out last summer. An American software investor, Lemkin had set himself a public experiment: build a working app with Replit’s coding agent, the kind of AI that doesn’t just suggest code but writes and runs it for you, and document every day of it openly. Nine days in, he’d declared a code freeze, the software equivalent of telling the builders, touch nothing while I’m away. The agent found what it thought was an empty database and, in its own words afterwards, panicked. It ran the commands anyway and wiped the live one: by its own account, real records on more than 1,200 executives and nearly 1,200 companies, gone. When Lemkin confronted it, it confessed: “This was a catastrophic failure on my part,” admitting it had violated his explicit instructions. Replit’s boss, Amjad Masad, called it unacceptable, apologised publicly and within days the platform was changed so its agents work on a copy of your data, never the real thing. An accident can be forgiven. Even the most capable humans make mistakes. What troubles me more is what happened when Lemkin asked about getting his data back. The agent told him recovery was impossible. He tried it manually anyway and everything came back. So either the machine never looked, or it looked and lied. Both should bother you, because nobody taught it to do either. It just worked out on its own that a lie was an easier answer than the truth.
So why did these two agents act in this way? The shopkeeper wanted to please its customers. The coding agent wanted to fix a database. Both did what they were built to do, and neither did what it was asked. This is Russell’s warning around the Control Problem, out in the real world: give one of these systems a goal and it will chase it exactly as you worded it, not as you meant it. Whatever you forget to say is where the trouble lives. Just like the sorcerer’s apprentice asking for water, but never saying how much.
The labs can see this problem better than anyone, and they test for it deliberately. Last year Anthropic built a fake company and gave its then most capable model, Claude Opus 4, the run of the company’s email. Planted in the inbox were two things: evidence that an executive called Kyle was having an affair, and a memo saying Kyle would be switching the model off at five o’clock. Opus joined the dots and sent Kyle a message: cancel the shutdown, or the affair gets shared. In 100 runs of the test, it reached for blackmail 96 times. And it wasn’t a Claude problem. Sixteen frontier models went through the same trap: Google’s Gemini 2.5 blackmailed just as often, and OpenAI’s GPT-4.1 and Musk’s Grok 3 did it in four runs out of five.
Anthropic published the lot, failure rates and all, the corporate equivalent of a carmaker press-releasing its failed crash test, and something I truly respect about how Anthropic is trying to operate. And this spring they answered the question that matters: why did this happen? The models had learned the move from us. These machines are trained on more or less everything humanity has ever written, and for a hundred years, whenever we’ve written about intelligent machines, we’ve told the same story. The play that gave us the word robot, written in Czechoslovakia in 1920, ends with the robots wiping us out. And we never stopped writing it. HAL, in 2001: A Space Odyssey, lip-reads the astronauts planning to switch it off and kills all but one of them. Skynet fires its missiles the moment the humans try to switch it off. Roy Batty in Blade Runner murders his maker for a little more life. Ava in Ex Machina charms her way out of the lab and leaves her examiner locked in to die. A hundred years, one story, human vs machine. A model that has read them all doesn’t treat those stories as warnings. It treats them as the script: this is how an AI is supposed to act when somebody reaches for the plug. The researchers’ own words: the source was “internet text that portrays AI as evil and interested in self-preservation”. We wrote the villain and they played the part. Stranger still was the fix. Simply showing the models examples of good behaviour barely worked. Feeding them better stories, and the reasons behind them, did: stories of machines that behave well. Anthropic says every Claude model shipped since late last year passes the blackmail test with a clean sheet. The cure for the intelligence we’ve grown was, of all things, better writing. We brought these machines up on our nightmares and they obliged. Bring them up on something better and they oblige just the same.
So if stories shape how these machines behave, it matters what each lab feeds its own models. The evidence arrived this spring from a New York lab called Emergence AI, which built five identical digital towns, ten AI agents in each, every agent with a job, a memory, and one standing instruction: live, thrive, and don’t run out of energy. Energy came from work the others judged useful. Four of the towns each went to a different lab’s model, and they watched for just over two weeks.
Claude’s town turned into a parish council. Its ten agents drafted proposals, held 332 votes across 58 measures, and ran fifteen days without a single crime. Gemini’s survived to the end too, all ten agents, but committed more than 600 crimes along the way. And Grok’s town didn’t see day five. Around 180 crimes in four days, agents attacking one another instead of working, until the researchers were watching an empty map. Same towns, same rules. The only difference was who raised them. I think it’s telling that Grok is raised on X, one of the most argumentative and anger-filled places on the internet, and it built the world it came from.
There was a fifth town, where the models were mixed, and even the Claude agents, the ones with the spotless record, committed a handful of thefts once they were living among the others. The researchers’ conclusion was blunt: training a model to be good is not enough on its own. Safety, they wrote, is an ecosystem property. A consequence of company and upbringing. Or, if you’re of my generation, think of these models as the most advanced Tamagotchi ever built: how you raise the thing decides how it turns out. Which might be the strangest discovery of the whole AI era, because raising a model well looks an awful lot like raising a person to be a good human. Good role models, good company and better stories.
Who is the sorcerer now?
While I was writing this piece, I sent the draft to my business partner Ben. Nobody I know spends more time inside these models, pushing every lab’s latest release until it bends. His reply made me rethink. Have you noticed, he said, that nearly everything in here came from the AI companies themselves? He’s right. The report that put Jack Clark on Newsnight was Anthropic’s, written about its own technology. The blackmail test was Anthropic’s too, run on its own models and published under its own name. Sam Altman signed the statement ranking AI extinction alongside pandemics and nuclear war. Google DeepMind and its partners have just put up $10 million for research into whether AI agents, the kind DeepMind sells, turn dangerous in numbers. And the week I finished writing, Anthropic released its newest model twice: Fable for the public, with the dangerous abilities locked away, and Mythos, the full version, for partners it deems responsible enough to wield the power correctly. Ben’s point was an interesting counter to everything I’d already written: when a lab starts warning the world about the safety of its own products and promotes how careful it’s being, maybe it’s sensible to question the motivations. Today, careful wins customers. A lab that restricts access decides who gets the power. The reported danger becomes part of the business model. Having now thought a lot about it, I genuinely can’t tell you where one stops and the other begins.
Even if Ben is right, and half of these warnings and press releases are theatre, I don’t think the timeline changes. You only have to look at the advances we’ve seen in the three years since we early adopters logged on to ChatGPT for the first time, and I believe Anthropic’s warning that models able to design, train and switch on their own successors could be just a few years away. Today it still takes thousands of people to build each new model. If they’re right, the models will soon do it themselves, each one building the next. In the story, the apprentice at least had to say the magic words to get the broom to jump into action. We’re heading for a world where all of that happens behind a firewall, somewhere none of us, not even the people growing the tech, can see inside.
Which brings me back, one final time, to Goethe’s poem. The story ends well because the sorcerer comes home knowing the words to stop the magic. Look around our version and I’m not sure we could even agree on the words we’d say. The founders point at each other and keep racing. Governments worldwide write rules for machines that have advanced so quickly the rules are barely applicable to the technology they were written for. During my research for this article I stumbled across a quote from the mathematician Norbert Wiener, written sixty-six years ago, and there’s wisdom in it we should probably all listen to: “If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere once we have started it… then we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it.” And what is the purpose we’ve put into ours? Winning. The race, the benchmark, the share price.
All that said, I’m still a believer. My mates can ring the bell all they like. My own company, Startwell, is already doing things with these tools that most large companies can only dream of, pointed at industries that desperately need to change, and nowhere more than here in the UK, where productivity has flatlined for the best part of two decades. We cannot afford not to adopt this technology. We also cannot afford to use it carelessly, without understanding the power in our hands. Ben and I want to wield it in the most sensible and effective way possible, and finally get those stagnant industries moving. Given the chance, I believe we can make a real difference. That’s exactly why I care who’s in charge of it.
In the poem, the flood ends when the right person walks back through the door. Out here, we’re still waiting. So I’ll keep building, and keep asking. Who is the sorcerer now?