When does human oversight of AI stop being real control?

A hospital tried to make a dangerous mistake impossible

In 2010, researchers at the University of Pennsylvania tested a change to hospital software. When a doctor tried to order two particular drugs together, a combination that can cause serious bleeding, the system refused. Not a warning. A refusal.

It worked. Doctors stopped ordering the combination at a far higher rate than under the warning it replaced.

The trial was stopped early anyway. Four patients were delayed getting drugs they needed. In those four cases the doctor was right, the rule was wrong, and the software would not let the doctor act on it.

Nothing about the doctor’s position had changed. Same licence, same training, same legal responsibility for the patient. What had changed was that being right was no longer enough.

Companies are building the same basic structure around AI today. The AI does more of the work, while a person remains responsible for reviewing the result, approving it, or stepping in when something goes wrong.

The assumption is that keeping that person at the end of the process preserves control. The history suggests it can fail in several different ways, and the hospital case is only one of them.

Across nine earlier automation systems, from cockpits and courtrooms to tax offices and cars, we looked at what had to remain true for human oversight to mean anything.

Four things have to stay true for oversight to be real

Someone can hold the final decision on paper long after they have lost any realistic ability to use it. The cases point to four things that have to remain true for human oversight to be real rather than nominal.

• Notice

  You realise something needs your attention. If you never look up, the approval button is irrelevant.

• Understand

  You know enough about why the system produced this answer to judge whether it is wrong.

• Act in time

  There is enough time left, and enough of a way to intervene, for your judgement to change the outcome.

• Override

  The system actually lets you. This is the one the hospital software removed.

We use “authority capacity” as shorthand for these four conditions. The label matters less than the fact that they fail separately. A reviewer can be fully authorised and never notice. They can notice and have no time left. They can understand exactly what is wrong and be blocked anyway. Any one of those is enough to make the sign-off meaningless while the process still records that a human approved it.

Two crashes in 2013 show the first condition failing in a setting where every other one was intact. On an August night, a cargo jet came down on approach to Birmingham, Alabama. Investigators found the autopilot had stayed on during the descent and that neither pilot realised the aircraft was too low until the last few seconds. Either pilot could have switched the autopilot off at any moment, instantly, without asking anyone. Nothing stopped them except not knowing they needed to. A month earlier, a flight crashed short of the runway in San Francisco on a clear day, with investigators reporting that the pilots were not fully aware of what mode the autopilot was in.

A federal review that followed found pilots could not meet the required standard flying on basic instruments alone, the situation they would face if the automation quit. Regulators estimate automation handles around 90 percent of a modern flight.

Aviation is worth taking seriously and worth bounding. Pilots are licensed, retrained on a schedule, personally liable, and watched by a regulator. Almost no office job works that way, so this is a clear look at how the failure happens rather than a prediction about how fast it would happen anywhere else.

Attention is often where oversight starts to weaken

Across the nine cases, the weak point was almost always the same one, and it is the one nobody decides on. People do not choose to stop paying attention. It happens to them. Any AI process that puts a reviewer at the end is relying on that reviewer still looking, months into the deployment.

A system can fail by saying too little. That is the cockpit: nothing flagged the problem, so nobody looked.

It can also fail by saying so much that people stop reading. That is hospital alerting. In one study of more than 157,000 alerts across two million prescriptions, doctors dismissed 52.6 percent of them. At a large teaching hospital the figure was 73.3 percent.

The obvious fix is to send fewer alerts, but the number of alerts did not predict how often they were dismissed. What seems to matter is how much attention each alert costs and how often that attention buys something useful. An alert that is usually right earns its interruption. One that is usually irrelevant trains people to click past it, and they will click past the important one too.

In one hospital series, responses to a repeated alert dropped by roughly 2.7 percent a week, from about half of alerts acted on down to about a third over nine months. That decline is not a fact about human nature. It is a response to what the alerts were worth.

That middle number is worth sitting with, because it is the one that catches companies out. In the same hospital, with the same doctors, dismissing an alert about a duplicate prescription was the right call 98 percent of the time. Dismissing an alert about kidney-related dosing was the right call 2.2 percent of the time.

The pattern is not random. Doctors were right when the answer was already in their head, that this order is a duplicate, that this recorded allergy is really a mild side effect. They were wrong when judging the alert would have meant going and looking something up. Kidney function is not something anyone carries around.

So a single company-wide figure for how often people override the system tells you almost nothing. The average here was around 60 percent justified, and the average was hiding everything worth knowing.

A click does not mean someone thought about it

The standard response to all of this is to make people confirm. Add a checkbox. Require a reason. Make them sign off. It is also the most common control placed around AI systems today, and the evidence is unkind to it.

In hospital data, when doctors were under time pressure, 13 percent of alerts were handled wrongly through what the researchers described as automatic actions taken without conscious attention. The click happened. The thinking did not.

A cleaner test comes from tax offices. Several countries now fill in your tax return for you and ask you to check it, change anything wrong, and approve it. Researchers in the Netherlands tested what happened when people were required to confirm each figure was accurate.

Confirmation made people more careful in one situation only: when the pre-filled return said they owed more than they really did. When it said they owed less, the confirmation step changed nothing.

The practical version: if you add a confirmation step to a decision where the person has nothing at stake in catching the mistake, you will get a click. You will also get a record showing a human reviewed it.

The software can shape the decision before you make it

Everything so far is about whether a person can act. There is a second thing going on, and it affects people who can act perfectly well. It matters for AI because most AI in a workflow does not make the final call. It writes the draft, ranks the options, or fills in the fields, and then hands the result to someone for approval.

Go back to those pre-filled tax returns. There is a well-established pattern in tax research: people report their income differently depending on whether they are expecting a refund or expecting a bill. When returns came pre-filled, that difference disappeared.

Nobody had taken anything away from the taxpayer. They could still read every line, change every figure, and refuse to sign. What changed was where they started from. People tend to start from whatever the software hands them, and once the software has written the draft, it has already set the terms of the argument.

The same research turned up a second effect. Not adding income to a blank form is something you did. Not adding it to a form that was already filled in is something you failed to do, and people treat those differently, even when the result is identical. Failing to correct a machine feels smaller than making the same mistake yourself.

The size of this shows up in a Danish study of more than 40,000 randomly audited tax filers. On income the tax office already knew about and had filled in, under-reporting was close to zero. On income people entered themselves, it was substantial. Same person, same form, different behaviour depending on who typed the number.

American courts show a sharper version. When judges began using computer-generated risk scores at sentencing, the judge’s authority never moved an inch. The law said consider the score, and consider was always the word. But in State v. Loomis, the company that built the tool refused to explain how it worked to the court or to the defendant, and the score contributed to a denial of parole. The judge decided. The judge decided using a number nobody in the room could examine.

Two more things were settled before that decision reached the courtroom. Somebody chose which facts about a person the tool would weigh. And somebody chose what score counts as “high risk” — a line that can be drawn to reflect local tolerance, or in some cases how full the jails are. A judge reading “high risk” may be reading a policy decision presented as a measurement.

There is a useful word for what was missing here: contestability. Can the person affected actually challenge how the result was produced? Being told a computer was involved is not the same thing, which is why arguing about the tool’s accuracy never settled the case.

What companies should measure

If the question is whether human oversight is doing anything, these are the things that would show it. Each comes from a specific case rather than from a general principle.

• How long before someone notices

  The gap between something going wrong and a person realising it has. In the cockpit, that gap ran out before anyone looked up.

  Aviation
• How long they need to act, versus how long they have

  Research on automated driving found that a warning several seconds before a hazard changed whether drivers avoided it. Every system has a window below which the ability to intervene stops being real.

  Driving
• Whether overrides were justified, broken down by type

  Never the company-wide average. The hospital figure ranged from 2.2 percent to 98 percent depending on the kind of decision.

  Hospitals
• Whether people check more carefully when the error costs them

  If review only happens when the mistake is expensive for the reviewer, the review step is not doing what you think.

  Tax
• How long people spend on required confirmations

  Look at the spread rather than the average. The very fast responses are where clicking has replaced thinking.

  Hospitals
• Whether attention drops off over time

  In one hospital series, responses to a repeated alert fell by about 2.7 percent every week.

  Hospitals
• Whether people behave differently on machine-written fields

  Compare accuracy on the parts the system filled in against the parts the person filled in themselves.

  Tax
• Whether people can still do the job without the tool

  And how recently they last did. Worth noting that no study found anywhere measured this beyond a few weeks, so long-term skill loss is an open question rather than a finding.

  Aviation · Education

What the history did not support

Several reasonable-sounding assumptions did not survive the cases. Publishing those matters as much as publishing what held.

• That people are more careful when the stakes are high. This failed in both directions. Cases where someone else bore severe, permanent consequences produced completely different outcomes from each other. Meanwhile adaptive cruise control spread across the entire car market while drivers personally bore the risk of injury or death.
• That serious consequences make people pull back. Aviation is the strongest possible test, with catastrophic risk, personal liability and three decades of published warnings. Reliance on automation reached around 90 percent. The pullback that eventually happened was ordered by a regulator, not chosen by pilots.
• That forcing a human to act preserves human control. The one properly randomised test of this idea took control away and hurt people.
• That expertise protects you. Doctors have exactly the expert knowledge a sentencing judge lacks, and it did not stop them clicking through alerts without reading them.
• That having the final say is a yes-or-no matter. The sentencing cases show it held completely on paper and given up in practice at the same moment.