Software is becoming a colleague
The first time I really felt the change was while debugging a long task in MoiAI. I handed it a piece of work and went to a meeting. When I came back it had done most of it and stopped at a point that needed my sign-off, with a short note: what it had done, why it stopped, and what it suggested next.
It was a strange feeling. I wasn’t “using” software. It felt more like taking a handover from a colleague.
For decades the basic shape of software barely changed: a person gives an instruction, the software executes, the result comes straight back. From the command line to graphical interfaces to phone apps, interaction became friendlier, but the relationship stayed the same — a person operating a tool. What agents change is that people start stating goals instead of issuing step-by-step operations. It is a shift from operating to delegating.
Humans have been thinking about delegation for thousands of years. In the end it tests whether you dare to let go; how clever the other party is comes second.
Being able to do it is not the same as being trusted to
METR runs an ongoing study that measures AI agents by task length: for a task that takes a human expert a given amount of time, how reliably can an AI complete it on its own?
The results are striking. At a 50% success rate, GPT-4 in March 2023 could handle tasks equivalent to about four minutes of human work. By February 2026, Claude Opus 4.6 had reached roughly twelve hours. On METR’s data the measure has doubled about every 188 days over the long run, and faster since 2023 — about every 129 days.
But the same data has a second line that rarely gets mentioned. Raise the bar to an 80% success rate, and the task length Claude Opus 4.6 can handle drops to about 70 minutes — a full order of magnitude less.
That gap is the distance between “can do it” and “can be trusted to do it.” You would not hand twelve hours of work to a colleague who fails half the time and then leave for vacation.
The arithmetic of long tasks
Why does the manageable task length collapse when you ask for more reliability? Behind it is simple multiplication.
Suppose a job takes 20 consecutive steps and each step succeeds 95% of the time. That sounds dependable, yet the whole job succeeds only 0.95 to the 20th power — about 36% of the time. Raise each step to 99% and the whole job still succeeds only about 82% of the time. The longer the task, the harsher the demand on every step, and it grows harsher exponentially.
That was our biggest lesson when we re-architected MoiAI’s execution loop. At first we assumed the bottleneck was that the model wasn’t smart enough. We found instead that most failures were not “couldn’t do it” but “didn’t know it had got it wrong”: a tool call returned something abnormal, the agent didn’t notice, and it carried the error a dozen more steps. So across the loop of planning, tool use, execution and feedback, a good share of our effort went into feedback. Every step must be verifiable, errors must be detectable, and once detected the agent must be able to roll back to a clean state.
It is the same principle as checkpointing in large-model training. No system never fails; reliability comes from recovering cheaply when it does.
You think it made you faster
There is a subtler problem too: our sense of how well delegation works may not be reliable.
In July 2025 METR published a randomized controlled trial. Sixteen experienced open-source developers completed 246 real tasks in large codebases they had maintained for years, each task randomly assigned to allow or forbid AI tools. Beforehand they expected AI to make them 24% faster. Afterwards they estimated it had made them 20% faster. Measured, they were 19% slower.
The study has limits. The sample is small, the tools were from early 2025, and the setting was code the developers knew intimately; METR itself cautions against over-generalizing. But the problem it reveals is real: delegation has a cost. You have to explain the work, wait, check and redo. If checking a piece of work costs nearly as much as doing it yourself, delegation does not pay, however fast the other party is.
Beyond doing good work, a good colleague has one quality people overlook: their work is cheap to check.
What a good colleague looks like
So when I think about agent products now, my questions sound more and more like interviewing a new hire than benchmarking a model.
They know when to stop and ask. The most dangerous thing is often pressing on when unsure. Whether an agent can recognize “I’m not confident about this step,” and hand the question back in a form a person can decide quickly, matters more than whether it can take a few more steps.
Their work leaves a trail. What was done at each step, on what basis, and what changed should all be reviewable. Traceability is the precondition for trust, and for locating problems quickly when they happen.
They work within boundaries. Permissions, data and the systems they may touch all need clear limits. It is one reason MoiAI insists on running locally and privately: when an agent handles your files and workflows, keeping the data on your own machine is often what lets a professional let go.
They learn how you work. Every team has its own conventions, and that is what a skill framework is for: capturing how a kind of work is done so the agent does it more the way you want next time, instead of guessing from scratch each time.
The engineer’s new problem
Looking back, the core question for this generation of engineers has moved from “how do we make software able to do it” to “how do we make software trustworthy doing it.”
It is really a management problem with a new kind of report. Most new managers know the bind: hand work off and you worry; keep it and you can’t get it all done. Perfect people are not on offer; the way out is to design the work so it can be delegated — a clear goal, a clear definition of done, checkpoints that can be inspected, and a way back when things go wrong.
Agents are no different. Their capability curve will keep rising; I have no doubt about that. Whether they truly become colleagues depends on whether we make reliability, verifiability and boundaries part of the software itself.
References
- METR, Time Horizon 1.1 data (metr.org/time-horizons)
- METR, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity”, July 2025