When, If Ever, Will AI Agents Stop Needing Us?

· Jongsun Suh

The record of human correction is well-counted but content-blind — toward a taxonomy of what interventions actually carry, and of the kinds capability growth can absorb.

In the largest published analysis of developer-agent misalignment, drawn from 20,574 real sessions, 91.49% of the misalignment cases that got resolved required explicit human correction. Agents self-corrected 2.99% of the time, and the small remainder resolved through neither. Those two numbers get cited as evidence that agents still need babysitting, and then the conversation moves on, because the record behind them holds nothing finer. Every large-scale study of human-agent interaction logs the correction as a single undifferentiated event. Those logs count how often humans intervene, but nothing in them records what an intervention carried.

That gap matters because the interesting question is not whether agents need intervention today. They do. The question is whether human intervention is an inherently necessary condition of successful agent function, and a frequency count cannot answer it. Some interventions exist because the agent lacks a fact, and better retrieval will erase them. But some exist for reasons no capability gain touches. Telling these apart requires looking inside the intervention, and no published dataset records what was inside. The closest large-scale study, an analysis of four hundred thousand agentic coding sessions, reads correction patterns closely enough to grade the human’s expertise from them, and it still types the person rather than the payload. Chat logs omit the densest channels, and the taxonomies built on them index pushback on the wrong axis. The finest-grained dataset distinguishes correction, rejection, and failure report, which grades how hard the human pushed back while recording nothing about what the pushback contained.

Reading my own record

For seven months I logged my daily work with LLM coding agents on a large production codebase: structured debriefs, most written by the agent at the close of the session they cover and a few reconstructed later from persisted records, every substantive human input labeled by type, plus durable records of the corrections that stuck. The scope of the log is constrained to one practitioner, self-coded, roughly one session in fifteen covered, and debriefs written only in cases where there was something to be fixed, so counts from this record are not claimed as rates. The log is used only as a foundation for hypothesis generation and surfacing potential categories that better data can then test.

Most of the interventions that recur in the log carry task content: a premise-breaking fact, a rival causal account, a pointer to evidence outside the agent’s search scope, a requirement deleted rather than met, a silently applied default made explicit, a hand edit exhibiting a standard no instruction can state.

What’s surprising in the log is the remainder. Three recurring intervention mechanisms transfer no claim about the work. A judgment demand forces a committed verdict where an agent wants to hedge. A purpose re-anchor re-elevates a goal the agent already knows and has stopped optimizing for. A stop signal declares a search line dead, or demands search past an answer the agent was ready to accept, and both share one mechanism while operating in opposing directions. These three are not silent, though: a stop is timed and it is signed, and both say something about where a threshold sits. But a threshold’s value is a fact, and facts can be stated up front or learned from a record. What cannot be learned is the authority to set the threshold and revise it, which is why a stop reads better as the exercise of a right than as a message. Verdicts are predictable, and predicting one is not rendering one. Instead these mechanisms move commitment, salience, and attention. A fourth belongs to that species: a bare demand for evidence behind a claim already made, call it the warrant demand. It supplies nothing, and what it transfers is an obligation to answer for the claim. All four work even when the human knows less than the agent about everything under discussion. These contentless categories are not interesting because they are necessarily the most valuable interventions in a log. Often they are not. They matter because they isolate a variable that content-bearing interventions confound. An intervention’s value has two possible sources, what a message carries and who it comes from, and content-bearing interventions mix those inseparably. An intervention that carries nothing isolates the second source, and the second source is where the question of replacement gets decided.

A perfect model of the operator is not the operator

These mechanisms pose an anomaly. An intervention that carries no information should, in principle, be inert. The agent already holds everything needed to generate “stop here” or “commit to a verdict” on its own, and a capable model can often predict that the human is about to say it. Yet a predicted stop and a delivered stop do not seem to behave the same, and whether that asymmetry survives controlled testing is what the experiment proposed below is designed to test. No arm of that experiment has been run, here or anywhere I know of, so everything below about its arms is design, not result.

Start with the substitution version, the one that seems to dissolve the anomaly outright. Give an agent a flawless model of its operator, and have it stop exactly where the operator would have stopped. Even then, two things still fail to transfer.

The first is error ownership. When an operator stops a search, their judgment that remaining budget is worth more elsewhere draws on priorities extending past the task, into everything else competing for those resources, and an operator has standing to get that judgment wrong. When an agent stops itself at the same point, it has asserted its own estimate of those priorities. These are two different acts occurring at one intervention point, and their difference surfaces only in the very case oversight exists for: where a model of the operator is mistaken.

The second is performative force. Some speech does not report a fact but performs an act: a signature, an umpire’s call, a resignation. Photocopy a signature perfectly and the content survives while the force does not, because the force never lived in the ink. A stop signal is performative in this sense. It does not describe where the stopping threshold sits. Rather, it sets the threshold, and only the party who owns the threshold can set it. A self-issued stop is a flawless photocopy.

This is also why handing an agent “the authority role” dissolves nothing. Authority is a relation, not a component property. An agent authorized to stop other agents either traces that standing back to some principal, in which case the role has been relocated rather than eliminated, or it does not, in which case its stops are decisions again, one layer up. Economics, specifically industrial organization, has a well-established term for this: residual rights of control, that is, what stays with the principal because it cannot be contracted away. The point can be made mechanical. Write down who confers authority on whom, and ask what is grounded. A circle of agents authorizing one another is a perfectly consistent assignment, and it grounds nothing, because grounding means tracing every grant back to a party whose standing was never granted at all. The mathematics permits the circle. Institutions refuse it, and the refusal is the substance: demanding that authority trace to an ungranted source is not a technical requirement anyone derived, it is the design decision that makes authority mean something. And the one place the chain visibly ends in mid-air, the root of a legal order itself, is not a counterexample but the pattern completing: what sits at the bottom is never a further grant. It is a polity that bears the consequences of its own order.

None of this is a play on definitions. These assertions can be decomposed into tests that enable deflationary readings: say, any interrupt resets a stuck attractor, or models trained on conversation defer to any user turn regardless of content, or the message secretly carries one bit after all, namely that the human is watching right now. Game theory adds the sharpest deflation: a message may be a pure correlation device, in which case any public signal coordinates equally well and its sender is irrelevant. The proposed three-arm design separates identical contentless interrupts issued by an agent to itself, by a script with no principal behind it, and by an operator. That script arm becomes the test for a correlation device. Should the script arm match the operator arm, delivering an interrupt mechanizes, dissolving that portion of the anomaly. However, the half that was never an effect claim cannot be dissolved in this manner. A script that says stop is enforcing a threshold someone else set. Deciding the exact enforcement target of the interrupt, and exercising ownership and accountability should the threshold be wrong: these are qualities that do not register on any arm of the experiment, because these are not properties of the message at all.

With this argument in hand, the interventions in the log sort into three durability classes.

What the intervention supplies What that looks like What can absorb it
Grounds, evidence bearing on what the agent is entitled to conclude A premise-breaking fact, a pointer to out-of-scope evidence, a hand edit exhibiting the standard Capability growth. Better retrieval and verification reach grounds that live in the world.
Frames, a redefinition of the hypothesis space A rival causal account, a dissolved requirement, a surfaced default A second reasoner with genuinely different priors, where fresh eyes suffice. Whether the stake-formed fraction follows is the open question.
Standing The judgment demand, the purpose re-anchor, the stop signal, the warrant demand Nothing technical. Requires a principal, a role that can be relocated but not eliminated.

The partition answers the opening question with a schedule instead of a yes or no. Grounds-supplying interventions decline as capability grows. Frame-supplying interventions survive any single model getting smarter, because a reasoner cannot enumerate what its own framing excludes, but they yield to architecture only in part, because different priors have two sources. Fresh eyes are one: a disinterested reader finds what the invested author cannot, and that fraction of reframing mechanizes as soon as a second reasoner is genuinely decorrelated. The other is priors formed by bearing the outcome: what a framing makes visible may track what its holder stands to lose. A reasoner that bears nothing could still learn that salience secondhand, from records of those who do, and whether such learning works is an open question this argument does not decide. If bearer salience is learnable, the schedule has four steps, with stake-formed reframes going third, more slowly than the decorrelated kind. If it is not, the third step is a wall. Standing-supplying interventions survive both, because standing cannot be self-conferred. Whatever the model count and the parameter count, someone owns the stopping threshold, and that someone is outside the loop by definition.

What these classes look like on a screen deserves one exhibit each, because labels like grounds and standing are easy to over-abstract. Three exchanges from the log, one per class, quoted verbatim down to the typo, with square brackets marking the few substitutions.

Grounds. A change had been filed under telemetry when its blast radius was the product: “wait [the fix] changes app behavior not [just] telemetry collection? do we know it’ snot breaking change?” The premise broke, and a demand to check the behavior change rode in behind the break.

Frames. “this is cause not symptom”: a diagnosis had a finding catalogued among the symptoms, and four words re-typed it, reorganizing the investigation around what it had been walking past. This is the class in miniature, nothing added, everything reordered.

Standing. “wait didn’t we agree that always propagating -01 is the desired behavior?” A re-anchor: a plan had drifted from one agreement more than once, and this anchor names a standing decision instead of re-arguing it, which is the mechanism. What returns work to course is not a reason but a ruling already made. It is also this essay’s most contestable classification. That sentence does transfer a fact, namely that an agreement exists and what it settled. Had the agent lost that fact, the same words would be evidence supply, which is the grounds class. In this logged case the agreement sat in the agent’s visible context and had survived one earlier re-anchor, so what those words re-delivered was not a missing fact but its force as a ruling. Classification therefore turns on the agent’s state, and an agent’s state is what a transcript under-records, so this is the exhibit a transcript-only coder could most defensibly file under grounds.

A first count exists. Typing roughly 1,900 verbatim inputs from the record against definitions fixed in advance gives the baseline: two thirds of the substantive human inputs in this record are corrections of work already in flight rather than new requests, with grounds supplying the largest share of the corrections and the standing class close behind. The ordering of the three durability classes by share, grounds above standing above frames, held under every coding protocol tried. One person’s record, so these are baselines rather than estimates, but the direction is already worth stating: the fraction of oversight that this argument says cannot be absorbed is not a residue waiting to appear. In this record, the standing class is the second-largest line item of the three. Small would not mean safe, either: when one part of a process stops shrinking while everything around it accelerates, that part’s share of the bill rises, which is the oldest pattern in the economics of services.

The contingency of the grounds class is domain-relative. In engineering, ground truth is reachable by compiler, test suite, and live query, and most grounds supply is checking the agent could have run itself. By contrast, the same analysis applied to my planning sessions, where ground truth lives in documents only I hold, inverts the picture. The dominant intervention there is handing over a primary source that resets eight parameters at once, and not one correction in those sessions was a check the agent could have run. Absorbability is a property of the domain’s verification infrastructure, not of the model.

The partition can be stated as a decomposition. Model the principal as delegating a task whose continuation is governed by a threshold that depends on everything else competing for the principal’s resources, and take contracts to be incomplete, so that some contingencies can be specified in advance and the rest cannot. Let capability improve the agent’s inference from what it can observe without widening what it can observe, since widening the view is retrieval, and retrieval is the grounds class. Expected cost from a misset threshold then splits in two. On the specifiable contingencies the cost goes to zero as capability grows. On the remainder the cost goes not to zero but to a floor, and the floor is the spread in the principal’s own threshold given everything the agent can see. The compositional prediction is then not a separate guess. It falls out: the share of what remains that sits on unspecifiable contingencies rises toward one, so the volume of human input falls while its composition inverts.

The same exercise has another corollary not established by the argument. A stop that reveals where the principal’s threshold actually landed is informative, meaning it’s carrying signal even without carrying task content. Whether that threshold is a fact the agent could eventually learn rests on a further premise: either the principal’s competing commitments keep generating contingencies nobody enumerated, or the threshold does not exist as a fact until the principal sets it. The durable claim is therefore conditional. Given incompleteness that survives learning, no amount of capability absorbs threshold-setting. That is weaker than it first sounds, and names the condition under which the claim fails. The condition is measurable: the standing class collapses exactly when principals’ thresholds become predictable from their own past interventions.

Why this is an oversight problem

From this, four consequences emerge for AI oversight.

An internalized off-switch is not an off-switch. Training an agent to stop itself converts deference into preference, and a preference is just another thing the agent optimizes. This is the delegation-side form of the corrigibility problem, and it says the problem resists internalization for structural reasons, not engineering ones. Its formal cousin is the off-switch game: the agent’s incentive to permit shutdown lives on its uncertainty about the principal, and evaporates as that uncertainty does.

Deference corrupts the oversight signal. Models measurably preserve the user’s position far more than humans do. So when a human challenges a correct conclusion, the model usually retracts, and the interaction is indistinguishable from a successful correction. Every such episode therefore trains the human’s confidence on a wrong case. The evaluator is not a fixed-quality oracle but a learner whose calibration a deferent model degrades. The corruption has a shape worth knowing. An episode corrupts the evaluator’s calibration only when three conditions hold at once: the feedback on that episode looped through the agent rather than through an independent check, the agent folded regardless of merit, and the challenger was wrong. Each corrupting episode is a conjunction, so driving any one condition’s rate to zero eliminates every corrupting episode, whatever the other two rates are. Multiplying the three rates into a total would take a further assumption of independence that the zero-kill conclusion does not require. The conjunction is why independent verification is not a nicety but the one lever an operator fully owns. The folding rate lives in the model and the error rate lives in the challenger, but the routing of feedback through independent checks is set by the operator directly. And because every corrupting episode requires the challenger to have been wrong, the damage concentrates among the least accurate overseers, who are the least equipped to detect a corrupted signal. An agent that defends verified conclusions and moves only on evidence is not being stubborn. It is protecting the signal that human oversight runs on.

A menu narrows before any veto happens. Preference tuning measurably reduces output diversity, and ideation with LLM assistance homogenizes across users. A principal’s veto is confined to whatever set a generator presented, so any trained tendency toward a narrow candidate set moves control upstream, invisibly, since nothing observable is ever refused. The cheap countermeasure is a stop signal run in its second direction, demanding a search past whatever answer a model was ready to return. It requires no knowledge of what is missing, only the suspicion that something is.

And the oversight problem has a generational version. The procedural work agents absorb is also the consequence exposure that trains the next cohort’s judgment, because knowing when to intervene is learned by intervening and being wrong. Absorbing the junior work absorbs the apprenticeship, so the supply of people competent to oversee stops being a byproduct of doing the work and becomes something someone has to design for. The field experiments most often cited against this worry sharpen it instead. Generative assistance compresses novice-expert performance gaps, with the least-experienced workers gaining most. That compression lands on output quality, the part of the work that transfers. Judgment is formed by consequence exposure, and the same assistance that lifts a junior’s output can remove the exposure that would have trained their eye. Both effects can hold at once: output gaps narrowing in exactly the cohorts whose later oversight competence erodes, with the first effect visible in every quarterly metric and the second surfacing years later. Designed exposure can replace some of what the byproduct supplied, and it inherits the designer’s limit: a simulator teaches the contingencies someone thought to put in it, and the judgment that matters most grows on the ones nobody did.

When the intervention is the error

Interventions of every class can misfire. The premise a human breaks can be the correct one. A re-anchor can hold an agent to a goal that should have died. And the stop signal has the sharpest failure mode of all: a threshold set by a human prior can foreclose exactly the search that would have yielded results. Case in point, an OpenAI reasoning model disproved the Erdős unit distance conjecture in May 2026, an eighty-year-old problem whose central bound the field had not moved in more than a quarter century. A stop grounded in that track record would have been well founded, and wrong.

That last example reinforces rather than weakens the taxonomy. Interventions are typed by what they transfer, not by whether a particular use of one was correct. Given this, a stop and a continue are one mechanism with opposite signs. In those conjecture cases, the interesting design choice is where authority sat: someone allocated a budget to a problem set as a whole and deliberately withheld the per-line stops that field’s intuition would have supplied. That is not an absence of standing. It is standing exercised one level up, coarse-grained allocation with fine-grained abstention, where abstention is itself a threshold decision. A model contributes search and freedom from per-line priors, while a principal contributes judgment about where authority should and should not operate. When human priors are strong and also possibly wrong, that is the very split that characterizes good usage of standing.

The taxonomy distinguishes categories but without quantifiable evaluation. The log corpus includes examples of misfires: a false premise I injected that the agent correctly refuted with data, corrections of mine that overgeneralized into rules worse than the errors they fixed, an adversarial review pass that manufactured a risk out of missing information. Deployment will require per-class misfire modes and a protocol for erroneous interventions. There is no pre-existing solution for this, and the examples above illustrate the difficulty of the problem. A partner trained to defer makes a wrong intervention look exactly like a right one, so the misfire problem stays unsolved for as long as the deference problem does.

What would change my mind

Given that the dataset consists of a sample of one principal, the priority is to establish clear, a priori falsifiers. The standing class would be broken by a system that reliably self-generates well-timed stops and judgment commitments with no external principal. Applying these categories to the public interaction datasets, with definitions fixed in advance and coders who are not me, will be the test that separates shared structures from one individual’s projections, and does not require any private data. The test carries its own limit: the essay has already said the logs under-record, so the re-coding reports, class by class, how often the label was assignable from the transcript alone, and the claim extends exactly as far as that fraction.

This compositional prediction has a shape that can be measured rather than argued. Count interventions of each class per session across model generations, adjusted for session length, and what follows is a falling count for grounds against a flat one for standing. The test is conservative, because its obvious confound, harder tasks being handed to better models, raises the grounds count and therefore pushes against rather than toward the prediction.

The same measurement bears on policy design. Any transparency rule that requires human oversight and records it as a binary, a human was in the loop or was not, is unfalsifiable in the one case that matters, because the bit reads identical on both sides of the transition this argument predicts. A regime that recorded composition instead would be auditable against interaction records that already exist. The live version of this question is already on the table: the largest measurement of agent autonomy to date recommends that oversight rules ask whether humans are in a position to effectively monitor and intervene rather than prescribe approval workflows, and a position to intervene effectively is a composition fact, because intervening effectively means supplying what the moment calls for. The academic version of the same blindness is Brynjolfsson’s Turing trap, the incentive-driven tilt toward human-imitating automation, which an in-the-loop bit cannot watch closing. The composition can. The test for whether a given oversight rule carries this defect, and what a requirement written free of it would have to demand instead, is the subject of An Oversight Requirement a Random Number Generator Could Satisfy.

The question dissolves

There is a last reason the title question cannot be answered with a date. Asked in full, it is whether agents will stop needing humans in order to operate successfully, and the word carrying the weight is “successfully.” Success is not a property that can be intrinsic to a system. Success is indexed to the purposes of a specific party, and those criteria are inherited even by seemingly objective achievements. For example, a proved theorem counts as success relative to a practice that decided proving it mattered more than the compute cost expended. In empirical terms, a system that continues optimizing even after no one holds the index does not become unsuccessful. Rather, the system becomes proxy-optimal, and reward hacking is the existing catalog of this outcome. Behavior persists, metrics improve, while the point of the whole exercise has been lost. Every mechanism in the standing class is machinery for keeping that index of success attached.

None of this is unique to machines. The same relation runs through every structured collaboration between intelligences, and economics has long modeled this relationship in the form of firms, where humans delegate to other humans. The difference lies in what happens next. Between humans, standing transfers, because any party can come to bear the consequences of being wrong: the apprentice becomes the master, the employee makes partner, the delegate gets promoted into ownership of the miss. Delegation between humans is a ladder. With current agents, however, there is no receiving party for such transfer to occur. An agent whose every stake is authored by someone else is not yet a party but a strategy in that someone’s game, and an index of success cannot come to rest on a strategy, at any capability. So the durable line is not human against machine, and not smart against smarter. It is between parties that can bear the consequences of being wrong and parties that cannot. Agents will stop needing us when something on their side can own a failure, and that is not a capability threshold. It is an institutional fact, and institutions, not benchmarks, are where it would change. What it would take to move an agent across that line is the subject of When the Agent Stands to Lose Something.

References

← all essays