Every day I'll go onto LinkedIn and slurp down some AI agent thought leaderslop, like the hog that I am. I need to know what brainrot these guys are talking about, you see. I am very healthy.

In this process I'll occasionally stumble across something that gives me a moment of inspiration. I don't have the original LinkedIn post at hand, but it was during the brief government-imposed Fable 5 outage window.

Essentially, the user had generated an entire app from scratch with Fable 5 in the short period it was available. When they got downgraded to Opus 4.7, their agent could no longer understand the codebase. It would get stuck in loops and fail to make progress.

That specific failure is rather idiosyncratic, but I think there's a general point here.

The future, maybe

Do you think that humans should stop writing software?

Like it or not, this is where a lot of the industry is running headlong towards. I interviewed with a company trying to generate specs from running enterprise applications, and then use them to generate full rewrites using massively parallel agentic workflows. The expectation is that each attempt could take weeks to months.

Maybe that'll cost millions: but that's still cheaper than spending tens or hundreds of millions to have a 50% shot of the project failing anyways, which is the reality of many large-scale software development projects. So personally, I think it's a bad future but also that we will still live in it.

It's a bit of a digression, but I encourage reading a bit about the the "Phoenix Architecture" which I've seen circulated in some of these AI-agents-for-everything crowds.

I hate everything about this idea, to be clear: I think it is just incorrect, not well reasoned, and not useful for thinking about actual problems. I believe it will always be true that it is less work for an agent to understand and then modify a codebase, than to understand specs and regenerate a working codebase again from scratch, because a codebase already is a more precise kind of specification.

Back to the point

But let's think through the implications of a codebase that has essentially no human involvement in writing code. Especially if it's generated solely from human-language specifications, as a sort of "AI compiler".

Like many machines, it will be designed around the premise that your fingers must be kept out of it. Under those conditions there's no guarantee that your codebase can be extended, or even understood, without AI agents. But there's also no guarantee that you'll continue to have access to those agents, or that they'll work.

There are the obvious failure modes: you could just run out of money to buy tokens. Or providers could go bust in ways that destroy available capacity. Or your country could have access to advanced AI tools cut off - either by the US government or by your own.

And then there are the esoteric cases. What if newer and more advanced agents fail on "simpler" problems, like Opus progressively getting worse at prose?1 What if the codebase grows beyond single-agent context, or starts to encroach on safety guardrails?2 What if ad hoc guardrails get added to confront new problems, and your app happens to be collateral damage? What if it Just Stops Working on your codebase - due to some undocumented tweaks in weights or harness - and there's no way to get support from a human at a megacorporation?3

Keep your fingers in that machine

I've followed the right to repair movement for a while. In the physical world this can mean access to parts or manuals. But as more devices have become controlled by computers, it's also been extended to include access to software.

I think this is a good starting point. What would it look like if we required agentic codebase providers to guarantee that their generated codebases can be serviced and extended, including by other providers?

I don't actually think this looks like requiring that their coding models be kept available. These software stacks have a lot of properties that make them hard to reproduce:

  • non-deterministic implementations

  • hardware dependence

  • proprietary training sets

  • behavior drift as we get further from knowledge cutoff dates

  • fragile economics

  • evolving regulation

...and so many more!

In fact I'd wager there is no amount of money that could guarantee a reproducible Opus 4.7, with current behavior and cost and user expectations, over the next 5 years. It's just not happening.

I also don't think this looks like everyone self-hosting their code generation models. A lot of the long-term reproducibility issues still apply, and there's a lot of benefits to centralization. We're gonna have hyperscale vendors until we change our entire model of economic understanding and decision-making.

What I think it would look like is instead designing agents and harnesses that generate codebases that look and feel like human codebases. (Just a lot cheaper and faster.)

That means well-maintained and well-documented codebases, with working example code, with operations scripts and observability tooling. Not just the tools that the agent uses, but ones that would help humans understand and get up to speed. Even down to expecting agents to produce human-understandable development history4.

That's not very exciting to thought leaders. It doesn't mean new agent-first programming languages, or conlangs for agent-to-agent communication, or regenerating your whole codebase from scratch from Markdown. It maybe means it's just what we've got today, but a lot more of it.

And that's why I like it! Mature technologies should be boring. Boring things are easier to understand, to repair, to migrate. I am hopeful that we can make coding agents into something boring, ideally before they take over too many portions of the global economy.