Supervision Can't Be Bolted On
Most people think an autonomous system is safe as long as someone can always intervene. The Cruise robotaxi incident shows why that assumption fails - and what proactive supervision actually looks like when reactive supervision isn't enough.
Most people think an autonomous system is safe as long as someone can always intervene. That sounds reassuring right up until you imagine the system moving through traffic at speed, and it doesn't know it's already made a mistake.
For the past five articles, I've argued that autonomy only becomes useful once it's governed. AI was the natural starting point, because that's where the anxiety lives right now. But if managed autonomy is really an engineering principle and not just an AI design pattern, it should show up wherever machines make decisions without continuous human control. That makes autonomous vehicles the obvious place to start testing it.
Stress test: If managed autonomy is a general engineering framework rather than an AI-specific pattern, it should explain how autonomous vehicles are governed - including where that governance breaks down.
The assumption behind supervision
Most people picture supervision as a loop. A system does something questionable, a human notices, the human intervenes, the problem gets contained. Article 2 described this as a stack of layers: permission, approval, an audit trail, and, if all else fails, a kill switch.
That architecture assumes something important. It assumes the need for intervention can be detected in time, either by the system itself or by something supervising it.
With many software systems, intervention can at least happen before the consequences become physical. A misbehaving process can be paused, access revoked, a deployment rolled back. Autonomous vehicles expose that assumption in a way software rarely does. When a system is moving through the physical world, noticing too late has consequences that can't be rolled back.
What actually happened
On the night of October 2, 2023, a woman crossing the street at Market and 5th in San Francisco was struck by a hit-and-run driver in a separate, human-driven car. The impact threw her into the path of a Cruise robotaxi operating nearby. The Cruise vehicle struck her, braked, and then did what it was designed to do after a collision: it tried to pull over.
Its sensors hadn't registered that she was pinned underneath it. So pulling over dragged her roughly 20 feet.
The California DMV suspended Cruise's driverless permits three weeks later, after learning the company hadn't shown investigators the full video - the part that showed the dragging. More than a year after that, the company admitted to federal prosecutors that it had filed an incomplete report about the incident and agreed to pay a $500,000 fine. Cruise's CEO and co-founder, Kyle Vogt, resigned about four weeks after the DMV's suspension, and other departures and layoffs followed.
Why reactive supervision breaks down
It's tempting to describe this as a story about a car that needed a better kill switch. So what actually failed? It wasn't the brakes. The vehicle stopped. It wasn't hesitation, either. It did exactly what its engineers had defined as the safe response to a collision.
The failure sat one layer earlier than that: the car's model of what was happening under it didn't match reality. It wasn't ignoring an operator, and it wasn't waiting for permission it never got. It was confidently executing the wrong next step, because it didn't know its first step had already gone wrong.
That's a different problem than "can a human intervene in time." It's closer to: what happens when a system doesn't know it has already failed?
Reactive supervision - whether that's a kill switch, a remote operator, or a human reviewing an approval request - depends on someone or something recognizing that intervention is needed. If the system's own picture of events is wrong, reactive supervision never even gets triggered. There's nothing to react to, from the system's point of view. The architecture also needs an independent way to notice when its own picture of events has gone wrong.
Proactive supervision, and where it actually lives
If reactive supervision can't be the whole answer, the rest of the answer has to happen earlier. That's the shift this article is really about: from supervision as something that happens during operation, to supervision as something mostly decided before it.
The clearest example of this is a concept with an unglamorous name: the operational design domain, or ODD.
SAE J3016, the industry standard for describing driving automation, defines an ODD as the specific conditions under which a given automated driving system is designed to function: weather, geography, time of day, road type, traffic conditions, and more. In practice, an ODD is a boundary. Inside it, the vehicle is authorized to make its own driving decisions. Outside it, what happens depends on the vehicle. Cruise's vehicles didn't have anyone to hand control back to - a driverless system operating outside, or losing confidence in, its ODD has to reach a safe stopping condition on its own.
It's tempting to read an ODD as a purely technical spec. I think it's more accurate to read it as a statement of trust. An ODD isn't just "where the car works." It's the set of conditions under which engineers are willing to hand decision-making authority to the system, because that's where they trust its perception, planning, and fallback behavior to be reliable enough. Outside that boundary, the authority hasn't been earned yet.
Supervision before the vehicle ever moves
Most of the actual supervisory work on an autonomous vehicle happens long before it's on a public road. Simulation runs millions of scenario variations that would be far too slow, expensive, or dangerous to test physically. Geofencing restricts where a vehicle is permitted to drive autonomously at all. The intent behind all of it is the same: catching the gap between what a system was trained to expect and what it will actually encounter.
None of this prevents every failure. Cruise had run extensive testing and was operating within a geofenced deployment, and still encountered a failure its post-collision logic didn't handle safely. But it's the difference between hoping a human notices in time and deliberately narrowing the space of situations a system has to handle on its own in the first place.
The shrinking margin for remote operators
Cruise exposed one limit of reactive supervision: a system that misreads its own situation has nothing to react to. Researchers studying remote driving found another kind of limit - one that applies even when a human is genuinely available. A 2024 study led by Newcastle University looked at remote drivers operating Level 4 automated vehicles from teleoperation workstations. When those operators were mentally disengaged, their reaction time when the vehicle needed intervention slowed by an average of 5.3 seconds. Their decision-making time for critical calls slowed by 4.2 seconds.
A 2024 review of vehicle teleoperation research found that delays below roughly 170 milliseconds generally have only a minor effect, while trained operators can often adapt to delays approaching 300 milliseconds. Beyond that, the direction is consistent: network delay eats into an already limited window for intervention. Seconds and milliseconds sound small next to a data breach that takes days to notice, or an AI agent that quietly modifies a spreadsheet. At highway speed, they aren't small. They're most of the available margin.
When prevention isn't enough
Even a well-designed ODD and a well-trained remote operator won't cover everything. That's where graceful degradation comes in - the last architectural layer rather than the first line of defense. A vehicle that encounters conditions outside its ODD, or loses confidence in its own sensor picture, doesn't need to solve the problem it's facing. It needs a predefined fallback appropriate to what's actually happening: reduce speed, increase distance, stop in place if moving would be more dangerous than staying put. Cruise's own pullover is the reminder of why that last option isn't automatically the safe one. The goal isn't cleverness in the moment. It's having already decided, long in advance, what "safe but incomplete" looks like - and having decided it specifically enough that it doesn't default to the wrong maneuver by habit.
That's a genuinely different design philosophy from a kill switch. A kill switch assumes someone catches the problem in time to stop it. Graceful degradation assumes they might not, and builds a soft landing anyway.
Beyond vehicles
Look back at everything that actually did the work in this article. Not the kill switch. The ODD. Geofencing. Simulation. Validation. Graceful degradation. Every one of them is a constraint - decided before the vehicle ever moves - that narrows what the system is allowed to do on its own.
Good supervision, it turns out, is mostly the engineering of constraints, and the feedback that tells you whether you're still inside them. A constraint nobody's watching is just a rule waiting to be broken quietly. If you've spent the last few years thinking about AI agents rather than cars, none of this should feel unfamiliar. Both are systems making decisions faster than a person can continuously watch. Both depend on boundaries decided in advance rather than judgment applied in the moment. Both need a defined, unglamorous fallback for when things go outside those boundaries.
If supervision depends this much on what a system already believes about the world, the next place to look isn't AI, and it isn't vehicles either. It's a system old enough that nobody thinks of it as intelligent at all: the electrical grid, and what happens when feedback breaks down with no AI anywhere in the loop.