Intelligence is bounded by its systems
When people talk about large models these days, they start with the leaderboard. Whose new model took first place, how many points it gained over the last one — there is news like this almost every month.
In my years on Microsoft’s AI platform, though, the thing that kept me up at night was rarely the model. Far more often it was a 3 a.m. alert: a node had dropped, a training job had died, and the questions were when the last checkpoint was written and how many GPU-hours we had just lost. The model decides what is possible. The system decides how much of it you actually get. I am increasingly convinced the second is becoming the one that matters more.
The leaderboard is flattening
Start with a few numbers. Stanford’s 2026 AI Index reports that as of March 2026, the best models from Anthropic, xAI, Google and OpenAI sat at 1503, 1495, 1494 and 1481 Elo on the Arena leaderboard — 22 points from first to fourth. The gap between the best American and best Chinese model was 2.7%.
What does 22 Elo points mean? By definition, it means the first-place model beats the fourth in a blind head-to-head only about 53% of the time — not much better than a coin toss. The report’s own conclusion is blunt: competitive pressure is shifting toward cost, reliability and domain-specific performance.
Models matter, of course; frontier breakthroughs still set the ceiling. But for most teams trying to put AI to real use, the models they can buy are now close enough that the difference lies elsewhere: in whether they can run them reliably, cheaply and continuously.
Training: sixteen thousand GPUs, a failure every three hours
In its Llama 3 paper, Meta published something rare — a failure ledger. The 405B model was pre-trained on up to 16,384 H100 GPUs. Over one 54-day window the job was interrupted 466 times: 47 planned maintenance events and 419 unexpected failures. About 78% of the unexpected ones were confirmed or suspected hardware problems, and GPU issues alone accounted for 58.7%.
That works out to an unexpected interruption roughly every three hours.
Worse, synchronous training has almost no tolerance for failure. When one GPU goes, the whole job usually stops and restarts from the most recent checkpoint. At that failure rate Meta still achieved more than 90% effective training time, and only three incidents in those 54 days needed significant manual intervention; automation handled the rest. The methods the paper describes are not exotic: cut job-startup and checkpoint time, and build tools that diagnose problems fast.
This is exactly where I spent a great deal of time on Nebula. A checkpoint looks like “just save to disk,” but at the scale of thousands of GPUs it is a delicate trade-off. Checkpoint too often and you spend your time writing while the GPUs wait. Checkpoint too rarely and every failure throws away all the computation since the last save.
There is an old rule of thumb for this, from Young’s 1974 paper: the optimal checkpoint interval is roughly the square root of 2 × checkpoint cost × mean time between failures. Run the numbers. With a failure every 3 hours and a 10-minute blocking checkpoint, the optimal interval is about 60 minutes, and checkpointing plus lost work eat close to a third of all time. Cut checkpoint cost by a factor of fifty, to about 12 seconds, and the optimal interval drops to about eight and a half minutes while total overhead falls below 5%.
It is only a first-order approximation — it ignores restart and reload time — but the order of magnitude is right. The three-tier checkpoint storage we built for Nebula rests on one idea: let the training process get rid of the checkpoint as quickly as possible. Write first to the fastest tier, closest to the GPU, resume training immediately, and let background processes push the data down, tier by tier, to durable storage. In the end checkpoint saving became about 50× more efficient.
To researchers this is almost invisible. Yet it decides how much of a given compute budget actually turns into model capability.
Inference: cheap tokens, expensive systems
On the inference side the story runs the other way, but the lesson is the same.
Individual tokens are becoming extraordinarily cheap. Stanford’s 2025 AI Index found that the price of a model performing at GPT-3.5 level (64.8 on MMLU) fell from $20 per million tokens in November 2022 to $0.07 in October 2024 — more than 280-fold in 18 months.
Yet nearly every team I know has a growing AI bill. The reason is simple: prices fall, but usage grows faster, and an agent completing one task calls the model many times over, consuming far more tokens than a single conversation. The unit price is the model vendor’s problem. The total bill is the system’s.
When I worked on the MaaS runtime and rate limiting, the lesson that stuck with me was that inference traffic is unreasonable. It has tides, bursts, and the occasional customer who saturates their quota in an instant. For backends such as OpenAI and Azure OpenAI, the Rate Limit Service we designed was, put plainly, doing three things: protecting the backends from being overwhelmed, sharing limited capacity fairly among tenants, and degrading in an orderly way when capacity ran out rather than letting everything fall over together. None of that depends on how smart the model is. All of it decides whether the “intelligence” users experience is actually usable.
Building MoiAI, I met the same problem in a different form. Under high concurrency and high tokens-per-minute, decisions such as which model a request is routed to, which requests can be batched, which context can be cached and which steps can go to a smaller model often matter more than switching to a “stronger” model.
Does the system learn?
Beyond training and inference there is a part people overlook: the data loop.
A model stops learning the moment it ships. A good AI system does not. Where users give up, which tasks always need a retry, which answers get rewritten by hand — collected and analyzed systematically, these signals become the most valuable input to the next iteration. I owned telemetry for a large organization on Microsoft’s AI platform, and at MoiAI we treated it as part of the architecture from day one. Without it, a team’s judgment about its product is largely guesswork.
Two products built on the same base model can feel worlds apart six months later. The gap is usually here.
A closing thought
If model capability is the ceiling, the system is the floor. The ceiling sets how high you could go in theory. The floor sets where you actually stand every day.
For the past few years the industry has focused on raising the ceiling, which is entirely understandable. But as everyone’s ceilings converge, the differences move to the floor: whether training runs to completion, whether inference holds up at peak, whether costs stay under control, whether the system learns from use.
This work is not glamorous and rarely appears in a keynote. I am increasingly sure it is where the real moats of the next few years will be built.
References
- Stanford HAI, AI Index Report 2026, Technical Performance
- Stanford HAI, AI Index Report 2025
- Meta, The Llama 3 Herd of Models, 2024
- J. W. Young, “A First Order Approximation to the Optimum Checkpoint Interval”, Communications of the ACM, 1974