Anyone who has built simulation or control systems in a regulated context learned early that the model is only half the deliverable. The other half is the validation harness: the explicit checks, bounds, and sanity conditions wrapped around the model that catch outputs the model itself would happily produce.
Machine learning spent a decade largely skipping that half, on the reasonable grounds that statistical performance was the point. One very large industry has now demonstrated what happens next, and the resulting architecture will look extremely familiar to anyone with an engineering background.
The system under examination
American health insurers covering more than thirty million older adults are paid according to the medical conditions documented in each member’s records. Extracting those conditions means parsing clinical notes at scale: unstructured text, inconsistent abbreviations, years of history per patient. Natural language models handle this well, and the industry deployed them enthusiastically.
Then the verification arrived. Roughly two thousand federal certified coders now re-check the industry’s outputs on a rolling quarterly cycle. Reviews published this spring found that at three insurance plans, 81 to 91 percent of certain sampled high-risk diagnosis codes lacked adequate supporting documentation. A major insurer settled federal claims for 117.7 million dollars over technology-assisted review programmes.
The critical detail for engineers: the auditors were not testing aggregate performance. They selected individual outputs and demanded the reasoning behind each one, reconstructed from data retained at the time.
The architecture that survived
What works in that environment is a two-layer design that any control engineer would recognise as plant plus supervisor.
The statistical layer does perception: reading messy natural language, resolving shorthand, locating candidate evidence. This is what neural approaches do better than anything else, and no rule system competes.
The symbolic layer does validation: checking each candidate against explicit, versioned clinical rules. Does this specific documented evidence satisfy the defined criteria for this condition, at this encounter? This is exactly the role of a validation harness, and it is what neural approaches handle unreliably, because a model asked to apply a rule is approximating rule application rather than performing it.
The output carries its derivation. Evaluations of modern ai tools for medicare risk adjustment coding accuracy now test whether each conclusion ships with the source text, the rule satisfied, the confidence value, and the identity of the human who confirmed it. Not logged separately, emitted as part of the inference.
The properties that turn out to matter
Three requirements emerged from the audit era, and each has a direct analogue in traditional engineering practice.
Deterministic reproducibility. A decision made in 2024 must be reproducible under the model version, rule set, and input data that existed then. Teams treating model updates as versioned, reversible configuration changes pass audits. Teams that continuously retrain in place discover that “the model has moved on” satisfies nobody.
Bidirectional error detection. The validation layer must flag outputs that reduce the operator’s revenue with the same diligence as those that increase it. Investigators treated one-directional correction as evidence of design intent, which converts an apparently technical choice into a legal exposure.
Meaningful human authority. Regulators have stated plainly that these systems must remain assistive rather than autonomous, which means the review interface presents evidence rather than conclusions. A supervisor that cannot see why the plant did something cannot supervise it, a principle that predates machine learning by several decades.
The general lesson
There is a tendency in applied machine learning to treat validation architecture as overhead that mature models will eventually make unnecessary. The healthcare case suggests the opposite: as models get more capable and get deployed into higher-consequence settings, the validation layer becomes the part that determines whether deployment is possible at all.
Which is, of course, exactly what regulated engineering disciplines concluded long ago. You do not ship a controller because it performed well in simulation. You ship it with the bounds, the interlocks, the logging, and the ability to explain any single decision after the fact, because somebody will eventually ask.
Healthcare relearned that at a nine-figure cost. The engineering literature had it written down all along.
