SAFETY-CRITICAL ML · PERCEPTION

Ensuring safety in mission-critical AI systems

Dr Akanda Ashraf · 2024 · all articles · back to profile

In mission-critical settings the interesting question is not how good a model is on average, but how it behaves at its worst. Aggregate accuracy hides the cases that matter: the rare object class, the unusual weather, the sensor that degraded overnight. Safety work begins by refusing to accept a single headline metric as evidence of dependability.

That means evaluation designed around failure. Slice the test set by the conditions that will actually occur in deployment and report each slice separately. Build adversarial and long-tail sets deliberately rather than hoping the random split contains them. Track the worst slice over time, because a release that lifts the mean while lowering the floor is a regression in a safety context.

The second pillar is calibrated uncertainty and the right to abstain. A system that knows when it does not know can hand control back — to a fallback algorithm, a redundant sensor, or a human. Calibration and conformal methods give abstention a defensible threshold instead of a hand-tuned constant, and the abstention rate itself becomes a monitored signal.

Third, safety lives in the engineering around the model as much as in the weights. Deterministic preprocessing, versioned data and models, reproducible builds, shadow deployment before live control, and drift monitoring on inputs as well as outputs. Most incidents I have seen trace back to the pipeline or to a silent change in input distribution, not to a novel modelling flaw.

Finally, human oversight has to be designed, not assumed. Operators need interfaces that make the model's confidence and its failure modes legible, and escalation paths that are quick enough to use under pressure. Dependability is a property of the whole system — sensors, software, process and people — and it is only ever demonstrated, never declared.