There's no accounting for AI

Lately I've been thinking a lot about the present of AI: not how we got here, not on where we'll be in a year or a decade, but on the bottom line impact at companies today.

Despite almost every tech discussion looping back to AI use (and especially coding agents), it's been surprisingly hard to prove that any of this works when rolled out at company scale. There have been some astounding results, but many companies have found themselves moderating their previously "all in" stance: most notably Uber instituting per-employee caps, or Amazon ditching their AI use leaderboard.

Anecdotally, many engineers love working with AI agents, while others report that quality is down and incidents are up. Is morale tanking because of a shaky job market and the prospect of being automated away, or is it tanking because AI creates bad working conditions? Ask different teams and you'll get very different answers.

Frankly, it's weird out there right now!

Last post, I talked about why it's hard to even find "real" cost effectiveness info due to a mix of ignorance, ulterior motives, and outright bullshit. This post is about why even relatively "honest" data can be very hard to interpret.

Doing better for yourself

So let's say you're interested in the cost effectiveness of AI, how to budget for it in an organization, and how it should impact your hiring. How can you sort through the noise and find a decent answer?

First off, don't trust anyone who doesn't give you their own usage context. What works at Amazon isn't appropriate for your 20 person startup. You should always be skeptical of advice about problems from people who sell solutions! And you shouldn't expect a person who tinkers with AI on the weekend to know how to build a corporate adoption program, not any more than you'd expect an AI company's CEO to be fluent in training local-first models.

Knowing where someone is coming from is crucial, and I'm writing this post from my own limited experience:

I'm an IC on a platform team at an AI neolab (~300 employees, many of whom are researchers and not traditional software engineers). I'm one of the admins for our Copilot instance, and I've been working on and off to adjust our use of AI and our cost model around it, including with the legal and finance teams. I don't decide our org-wide strategy, but I am an early adopter, and I'm one of the first people who'll get Slacked when anything breaks.

Previously I've worked at several other AI/ML companies. I also am actively interviewing with many companies and hearing about how they're doing it, and always talking to industry peers. So I've got some insight into what other actors are dealing with as well.

The view from the ground

With that context, here are some random but interesting notes from the past year of adoption within my own org (that is, what I've directly seen, not from the broader software industry):

There is more bottom-up pressure for AI adoption than top-down.

Many non-technical users are using AI to contribute in ways that would've required help from a dedicated software engineering asset. And they're generally happy about this.

AI has had no visible impact on our hiring plans

Most users use Copilot at least weekly, and over 50% use it daily

Over 50% of lines of code are written by Copilot, and it's common for individual users to write almost all of their code with Copilot assistance.

We have a Pareto distribution with AI credit spend:

1 user uses 25% of our token budget

A few users use the next 25% of our token budget

Half of our users spend "a cup of coffee or less" on tokens per month

90% or more of tokens go to Claude models, usually Opus 4.7 and 4.8 (we have Fable turned off). There was a huge push to ask for these models after Claude Code was ruled out - nobody's advocated that hard for any other AI editors or tools.

The most common failure modes are self-imposed (budget limit hit or org-wide config rollout error), followed by "user error" related to complexity (Copilot has a confusing licensing model and features), followed by the inference provider having issues (typically GitHub but sometimes Anthropic)

We do have some uses of...

Custom instructions for repos

Skills

RAG pipelines (idiosyncratic, self-built tools)

Almost no one is using...

Agent swarms

Headless agents

Self-improving harnesses

Self-hosting or open-weights agents

MCP servers (other than trivially)

Personal or heavily customized agents

Many individuals are vastly more productive (lines of code, PRs shipped)

Overall productivity (impactful releases, milestones, functionality) isn't really going up that much

A relay isn't a sprint isn't a marathon

The last point probably raises some eyebrows - how could we have over 50% of code being written by agents, at greatly increased speed, yet not be able to clearly show that we're more productive?

Well, there's actually a lot of reasons for this. Let's tackle them in rough order of the software development lifecycle:

Design

We're a research institute! We don't typically rely on raw "productivity" measures like PRs closed, because the link is even more tenuous than it is for software engineering teams.

Like many companies in tech, we're going through a lot of departures and restructuring. This disruption and decline in capacity means that company-wide productivity staying the same could actually be a win. We might never have a "stable baseline".

Developers are now "only limited by ideas" - but in a lot of situations (like research!), having good ideas is actually a major bottleneck.

Development

People will tend to go after the most promising ideas, and they tend to have decent intuition. There are many circumstances where exploring a lot of ideas in parallel still means that only one pans out (and it's the one you would've picked before AI anyways).

People tend to experiment with AI starting from minor projects, or greenfield efforts that previously weren't important enough to justify building. This means that AI ships will trend towards less impactful, especially at first.

Writing good specs hasn't gotten any faster. Plus, most companies aren't hiring software engineers for their skills at reading and writing - and this is far harder to pick up than a new framework or IDE.

Accountability and promotion still runs through JIRA and team meetings. Getting your team to agree on what's ready to move into implementation hasn't gotten any faster.

You'll find a massive literature on how context switching is the ultimate evil for developers.1 Juggling multiple tickets is still context switches, even if it's autonomous agent swarms rather than individual PRs.

Deployment

A lot of agent code simply gets thrown away without being merged - because another agent did it first, because it's incorrect, or because it's just not interesting once finished. This heavily skews the raw "percent of commits authored by agents" metric.

Agents can also be extremely verbose in terms of size of commit, number of commits, and number of PRs. For instance, I used an agent to automate a dozen PRs across hundreds of modules - but they were all low value Terraform refactor tasks.

Review processes usually still depend on human-in-the-loop, and these often exist for non-technical reasons that are far slower to change.

The most interesting features tend to require new infrastructure, new budget, legal review, and other things that are entirely outside of the SDLC so automated reviews won't even fix them.

Getting any of these processes changed might require executive approval at a minimum. In some cases, it requires legislative or standard bodies changes/clarifications!

Most teams don't have the technical chops to have deploy, observability, or rollback infrastructure that can 2x (much less 10x or 100x) deployments without more-than-2xing manual work.

Maintenance

Even if you fix review and deployment, you still want to know what other people are releasing - not just at the level of code, but at the level of user facing features (ensuring they all make sense together). Someone speeding up will always tend to slightly slow down everyone else.

Shipping features into a multi-feature application may bring in new traffic, but it definitely will cannibalize traffic from other features. Exponential growth of features does not lead to exponential adoption.

To a software developer, keeping up with the latest and greatest AI tooling might seem like a huge priority, but to an organization as a whole it might not even crack the Top 10. Even the most software-heavy companies do not actually share the interests and needs of the software developers that work at them. Many of those challenges are structural, require consensus and buy-in, are ego-driven: they are company culture and therefore one of the hardest parts to change mid-flight.

Somehow, cloud cost returned.

So far we've talked about why one half of the "cost effectiveness" equation is hard to nail down. But what about cost?

Of course at the end of the day, you do get a bill, so you've at least got a hard number. But if you want to model cost scenarios or do internal usage accounting, then what you get from AI providers turns out to be very hard to work with:

Are you looking at tokens, dollars, or both? Neither is the same from model to model, or even over time for the same models.

We've entered an era where new models are constantly shipped and their prices are constantly changing. You may even have multiple billing models in flight at the same time at the same provider.

Oh, are you multi-provider? They all bill differently: even when they're serving the same models.

Are you self-hosting? Welcome to the wonderful world of amortization and capacity planning.

There is no way to determine in advance how much work a query will produce. Even simple queries can occasionally become pathological.

The same query may change dramatically with new agents, or even new behind-the-scenes harness changes that are totally invisible to the company.

There is no way to submit a prompt with a "max token spend", and if that existed it wouldn't be helpful for most users because tokens at execution time do not map to human comprehension.

Input tokens are billed, and models will happily do things like slurp in raw data files, or the entirety of your installed venv to "read their documentation".

Output tokens are billed, and models will happily do things like generate entire PDFs or DOCXs for you, multiple times per session.

Any kind of sandboxing around AIs is very "iffy". Agents do not get stable or even traceable ephemeral identifiers, and there isn't a comparable line item entity to a "cloud account". We're in the bad old world where extremely unrelated work all goes into the same billing bucket.

There's typically no way to understand related costs, like sandboxes, tool use, CI jobs, or resources provisioned by AI. These may show up under entirely different line items, often under the user's own account. AI adoption can increase all of your other costs too.

The best billing granularity you're getting out of GitHub is "per-user", "per-model", or "per-repo" top daily charts. Even though they "own" your PRs, they don't attribute billing to them. Or even to "isolated API keys/workspaces/VSCode instances".

People have tried creating leaderboards to better surface cost and in a few widely reported cases this has backfired.

And none of this even gets into shadow IT, or the users building on "free" tiers (very appealing to developers, and very dangerous to a business).

AI inference providers have essentially "unlearned" every accounting lesson that cloud providers (and their associated practitioners) bolted onto their platforms over the past decade. And that's part of what makes adoption a hard sell to corporate decisionmakers: not the total cost, but the fact that it's a generic "AI Credits" line item with almost no metadata or rationale!

Proving your case

All of that might seem a little bleak, but consider that most of it used to be true of cloud migrations - and, well, we still mostly got migrated! While there's still a lot of room for improvement, nothing I've said so far prevents your company from moving ahead with AI spend today.

There are a few interesting reasons for that.

First, even the most egregious uses of AI tools are rarely the company's largest expenses. An employee spending $2,000 in a month on "developer tools" might raise an alarm bell when those are usually $50-100/seat; but the same employee's salary is higher yet than that, and they might not even have to ask anyone to spend staggeringly more money on GPU compute for training. A single rented H100 is on the order of $100/day2 after all!

Second, even if some employees incur high costs, most users won't even try most tools you offer them. Especially at smaller companies it's often easier to just give a blanket "yes" and then intervene for the handful of "expensive" or "problematic" users. Having individual conversations actually does scale when you're small enough to know everyone who works there.

Third, you can get corporate approval to spend a lot of money, even if it's hard to show immediate benefit or how much it will cost. Actually, companies already buy a lot of things that do not have immediately verifiable cost effectiveness! Consider: marketing budgets, new hires (always a risk!), company swag, and even your office snacks.

Fourth, there are always non-financial reasons for corporations to adopt technologies, which don't fit into cost effectiveness because they're hard to quantify or downright intangible. Startups make cost-inefficient choices, like hiring consultants or using cloud, because they also provide flexibility. Teams get delegated authority to make their own choices for reasons of morale and autonomy. VCs have expectations of their portfolio companies. The tools you choose impact recruiting. These are all considerations as real as the bottom line!

Remember, the job of your finance team is not to just rubber stamp anything that "seems cost effective". Much of their job actually centers on maintaining control of money flow and optimizing how money is spent. Yes, they'll ask for justifications - but that's to help them do their job, not to prevent you from doing yours.

The ask

So if effectiveness is fake and cost is impossible, what do you actually have to talk to a finance team about, anyways?

When you're considering an AI adoption effort at your company, you still need to come up with a pitch - it's just that it's rarely about an exact "dollars earned" or "hours saved". In fact, you might look a little foolish if that's what you lead with!

Especially at a startup, it's expected that you won't immediately know whether you made the right call. What you want is reasonable evidence that it could work, a way to tell if it isn't working, a plan to either ramp up or back out based on early indicators, and an idea of the time horizons involved. And in a lot of cases, it's better to spend more money and get better outcomes than to just prove efficiency.

If you're pitching a multi-quarter company-wide AI project, you should hopefully be talking to other stakeholders at this level of analysis:

What's our best effort at determining cost effectiveness (and reducing waste), and what are the known limitations?

How much will this spend be, and how predictable is it? Teams need this info to structure downstream workflows, like which card will get billed or when to switch to invoicing instead.

Does this get billed back to teams, or does it all fall under a single team or line item?

Would this come from room in existing budgets, or be a new expense? Not all budget lines are interchangeable.

Who should oversee spending trends, and who should approve overages?

Is this going to be a perpetual month over month increase, or will it eventually saturate? (And if it does: based on our headcount, on our customer count, or something else?)

Does this substitute costs in other areas? For AI tools, this could reasonably include other developer tools, continuing education, certain types of cloud resources, and potentially even head count.

What safeguards are there on cost? You should never be in a position where an agent can cause a company-ending event, but it's not useful to pick fights over a specific user's bills. Have sensible limits to prevent overruns but err on the side of quickly "bumping" them when a user is blocked, with any "usage conversation" coming after.

But probably the most important success factor is ensuring that everyone in the room knows that whether you promote AI adoption is not a purely financial decision. What you're pitching is a change to company culture that could impact any cost associated with software development going forward: recruiting, salaries, CI spend or cloud resource usage, you name it. A lot of smart people think it'll be worth it, but it's not a sure thing yet, and definitely isn't easy to prove today.

It's important that everyone understands that because you don't ever want to start your culture change project, and then cancel it due to cost - that kind of churn kills companies. Even if you "experiment" and then change direction later, it's possible to cause lasting negative effects (on morale, recruiting, tech debt, you name it). Even incidentally firing a team at the same time as adopting AI and you might tank morale for a long time! But it's not possible to just opt out, either - if you're too restrictive around AI usage you may lose employees from that!

If you're trying to engage with corporate AI adoption, you need to think of it as an emerging culture of working and not just a dollar cost on a spreadsheet. Yet similar transformations take quarters to years even at small companies... and large companies aren't even done with their "cloud transformations" yet.

So that means you've already started the conversation... right?

I've always thought this was a bit overstated. In my experience, the most effective developers often have the most context switches and the best handling of them!