Provably Right, Not Probably Right
Provably right means a clinical AI decision can be reconstructed on demand: the exact rule invoked, the exact values checked, the deterministic operation that returned pass or fail, and a named human who attested to it. Probably right means the system was confident. Under inspection, only one of those survives.
This is a category distinction, not a marketing line. It divides clinical AI into two architectures that look identical in a demo and behave completely differently the moment a regulator, an ethics board, or a court asks the only question that matters: show me exactly how this decision was made.
One architecture answers with a probability. The other answers with a proof.
The question no probability can answer
An inspector walks into a clinical site. She picks an enrolled patient at random and says: walk me through how you determined this patient was eligible.
If the decision cannot be reconstructed in front of her, source data, the inclusion and exclusion check, the logic that connected them, that is a protocol compliance finding. Protocol compliance has sat among the most-cited deficiency categories in FDA and EMA GCP inspections for years. It is the oldest, best-understood failure in the field.
A confidence score cannot walk anyone through anything.
Every regulatory framework that governs clinical AI rests on the same requirement: a regulated decision must be reconstructible, from source data through the decision logic to the rule that produced it, at the moment an inspector asks for it. The requirement appears in 21 CFR Part 11, in ALCOA+, in ICH E6(R3), in EU AI Act Articles 11, 12, 13, and 14, in Brazil's CFM Resolution 2.454/2026, and in India's New Drugs and Clinical Trials Rules, 2019. The phrasing varies by jurisdiction. The architectural requirement does not.
A confidence score is, by construction, the output of a probability distribution. It is not the output of a rule applied to source data. You can attach a regulatory label to it after the fact. You cannot derive it from one.
That is the whole distinction, stated at the level of evidence:
🔹 Probably right optimizes for how often the system is correct.
🔹 Provably right optimizes for whether we can prove the system respected every constraint on this specific decision.
Both questions matter. Existing validation paradigms answer the first. The second has been left for regulatory affairs to translate on their own, and it cannot be translated. A confidence score is not a compliance artifact. It is a marketing signal in regulatory clothing.
Simulate versus prove
There is a subtler version of "probably right" that deserves naming, because it is where the most sophisticated systems fail.
Some clinical AI simulates the regulator. It predicts what an inspector, an ethics board, or an agency reviewer would say about a given output. A well-trained model can do this impressively well. But prediction of a regulator's judgment is, by definition, probably right. It is a very good guess about a human verdict that was never actually rendered.
Provably right does not simulate the regulator.
It proves a bounded set of structural properties deterministically, with no probabilistic middle ground, and it routes every judgment outside that boundary to a named human attester. The machine proves what can be proven. The human decides what requires judgment. Nothing is guessed and then dressed as a verdict.
That is the sentence that separates the two categories:
A system that predicts what a regulator would say is probably right.
A system that proves bounded properties and routes the rest to an accountable human is provably right.
What a proof certificate actually is
A proof certificate is not a metaphor and not a dashboard. It is a concrete, structured artifact produced by the system at the moment of decision, containing a defined set of properties. At its core:
🔹 The rule invoked. The specific rule applied, by source, citation, and version. Not "FDA guidance." Protocol Section 4.2.1, Inclusion Criterion 7, verified per ICH E6(R3) source-data requirements.
🔹 The values verified. The exact patient, protocol, site, and operational values checked. Listed, not summarized. Attributable to source.
🔹 The verification operation. The deterministic procedure that returned pass or fail on this specific decision. Inspectable. Reproducible. Re-runnable in front of the inspector.
🔹 The boundary statement. What the operation did not check, and the human-judgment factors deliberately reserved for the responsible human.
The full specification extends this to eight properties, adding the decision's frozen risk class, the identity of the human who attested, the override and escalation record, and an explicit declaration that the operation is evidence presented to a reviewer, not a substitute for the reviewer's judgment. The complete schema is set out in The Regulatory Validation Framework.
No probabilities. No model confidence. No explainability score appended after the fact.
This is what 21 CFR Part 11 has demanded since 1997. It is what ALCOA+ requires every day at every site under inspection. It is what the EU AI Act now demands explicitly. And it is close to what CFM Resolution 2.454/2026 expects of AI in Brazilian medical practice: explicability, decision traceability, and a human-final decision, with the emphasis on traceability of the decision rather than interpretability of the model. The resolution governs medical practice and takes effect in August 2026; the principle it makes explicit, decision-level traceability, is what a proof certificate produces.
The regulatory standard is already here. Much of the vendor stack has not caught up.
The three-second test
There is a test any regulatory affairs professional can run on any clinical AI system today.
Ask: show me the proof artifact for this decision, rule invoked, source values, verification operation, boundary statement, in one document. Then time the answer.
🔹 If the answer navigates to a dashboard, that is a confidence score wrapped in a UI.
🔹 If the answer offers to export an audit log, that is a timestamp database.
🔹 If the system produces, within seconds, a structured artifact carrying those properties, that is a proof certificate.
Most clinical AI deployed today fails this test. Not because the vendors are dishonest, but because the architecture treats compliance as a layer added at the end. By then the regulatory primitive, the verifiable rule-to-decision-to-source-data trace, has already been replaced by a probability. You cannot retrofit a proof certificate onto a confidence score. The system has to be built around the certificate from the first line of code.
Why the evidence has to be uncorrelated
There is one more property that separates proof from performance, and it is the one most often missed.
A confidence score is the model grading its own work. It is generated by the same system whose output it scores, so it inherits that output's blind spots. When it is wrong, it tends to be wrong in exactly the ways the output is wrong. That is correlated evidence wearing the label of a check. Much of what is presented in clinical AI as quality control, one agent checking another agent trained on the same data, has the same defect.
Provably right requires evidence that fails differently.
A deterministic rule check, a formal structural proof, and a human attestation are three different substrates. They are unlikely to share a common cause of failure. A rule check can be wrong in ways a structural proof would catch. A structural proof can pass on a determination a human would reject. A human can catch what neither machine operation was scoped to see. Evidence drawn from substrates that fail differently is defensible in a way a single self-reported number never is.
Correlated evidence is confidence. Uncorrelated evidence is proof.
The category this defines
"Provably right, not probably right" is the design principle underneath a specific category of infrastructure: verification-first AI for regulated clinical decisions. The distinction is architectural before it is anything else.
🔹 A probabilistic model can participate in the decision pipeline, but only because it is gated by a deterministic check it cannot bypass. The model proposes. The proof disposes.
🔹 Anything an inspector, an ethics board, or a court sees is produced by the deterministic verification layer, not by the probabilistic substrate that fed it.
🔹 The verification layer, not the model, produces the regulated artifact.
That is why the category is defined by what it can reconstruct, not by how accurate it claims to be. Accuracy is a property of a model. Admissibility is a property of an architecture.
The regulatory frameworks are converging on this from every direction: continuous, real-time inspection trajectories in the United States, high-risk conformity assessment in the EU, decision-level explicability in Brazil. When inspection becomes continuous, verification has to be deterministic and instantaneous. Procedural verification, documenting conformance after the fact, cannot operate at that cadence. Architectural verification can.
The field is going to have to choose which artifact counts. A probability, or a proof.
Provably right, not probably right. That is the standard. The certificate is the proof.
Frequently asked questions
What does "provably right, not probably right" mean in clinical AI?
It means a clinical AI decision can be deterministically reconstructed and defended, the exact rule, values, verification operation, and human attestation, rather than merely being accompanied by a confidence score indicating how certain the model was. Probability describes the model's internal state. Proof describes what was actually verified.
Why is a confidence score not a compliance artifact?
A confidence score is the output of a probability distribution, not the output of a rule applied to source data. Regulations such as 21 CFR Part 11, ICH E6(R3), and EU AI Act Articles 11, 12, 13, and 14 require that a regulated decision be reconstructible on demand. A confidence score cannot be reconstructed into the rule and values that produced the decision, so it cannot satisfy that requirement.
What is a proof certificate?
A proof certificate is a structured artifact produced at the moment an AI-assisted decision is made. It records the rule invoked, the values verified, the deterministic verification operation, an explicit boundary statement of what was not checked, the frozen risk class, the attesting human's identity, the override and escalation record, and a declaration that the operation is evidence, not a substitute for human judgment.
How is a proof certificate different from an audit log?
An audit log is typically a timestamp database that records that something happened. A proof certificate records exactly what was verified, against which rule and version, with which source values, and what was deliberately left to human judgment, in a single reproducible artifact that can be re-run in front of an inspector.
Does verification-first architecture replace human reviewers?
No. The architecture is explicitly evidence, not substitution. The verification operation is upstream evidence a named human consumes before attesting to the decision. Accountability remains with the human.
NexTrial.ai builds verification-first infrastructure for AI-assisted clinical trial activation. The capabilities described here reflect the framework's design intent. For the full architecture, see The Regulatory Validation Framework.