How Absolute is the “Ground Truth” in Medical AI Annotation?

Companion piece to István Borbíró’s “When AI creates capacity, where does the time go?“

In architecting safe-by-design systems, trustworthy AI must be addressed at every stage of the development lifecycle. Bringing implicit assumptions into the light is foundational. Naming and documenting what has historically been left unsaid is the only way to manage the invisible variables undermining model performance.

One of the most persistent assumptions is the absolute certainty of medical annotation. To engineering teams, what matters is the clean, final annotated dataset: the images neatly bundled with binary labels or precise segmentations. However, the reality in the backstage of clinical data preparation tells a fundamentally different story.

The Reality of the Interpretive Act

In cancer care, a tumor’s discernibility on a 3D CT scan is rarely a black-and-white matter. It is a complex judgment call heavily influenced by systemic and human factors:

• Technical variability: Image quality, acquisition protocols, reconstruction kernels, and hardware manufacturers.
• Clinical complexity: The size and type of the tumor or the co-existence of confounding pathologies.
• Annotation software power: Visualization quality, annotation tool options, and AI assistance
• The human element: The radiologist’s clinical experience and inevitable fatigue.

Sometimes, clinicians simply do not have a definitive answer based on the visual data alone. They may require broader clinical context or an entirely different imaging modality to resolve a doubt.

Based on my own experience working hand in hand with radiologists, I have watched them study a cancer CT scan for a long time, questioning the visual evidence, and ultimately stating that they could not make a final call without a complementary PET scan.

Yet, the pressure of the pipeline often forces this nuanced reality into a binary, high-confidence label.

The Phenomenon of Annotation Drift

When the same image is presented to different experts, opinions frequently diverge. Even more striking is intra-annotator variability: a single radiologist may evaluate the same image differently at the beginning of a dataset versus toward the end, especially when dealing with large datasets.

As annotators progress, they gain specific experience with that particular dataset, building a relative, ensemble view of the cases. This drift is a natural human behavior.

Labeling is not the establishment of an absolute truth. It is an interpretive act that sets a reference framework the model will learn to reproduce.

Managing Ambiguity as a System Pillar

When human expertise is integrated into the AI pipeline, accepting a margin of error and variance is non-negotiable. In trustworthy systems, striving for absolute perfection is unrealistic, and even a liability. Ambiguous labels are tolerated during training, but they surface fast during evaluation where model reliability is determined.

The fix isn’t to pretend ambiguity away. It’s to acknowledge and manage it right from the project’s onset by being as explicit as possible and asking the right questions to the right people. This can be done by taking concrete actions during the data stage in the R&D lifecycle:

1. Map and Isolate Ambiguity: Explicitly group and categorize images with observed ambiguities to define whether the uncertainty stems from the technology (manufacturer, artifacts), imaging representation (unclear borders, mixed pathology), or the clinical presentation (missing context, weak label definition).
2. Establish Protocols for Uncertainty: Design clear data rules ahead of time for when uncertainty arises. Decide structurally whether to exclude the sample, implement confidence grading, or collapse it into a broader label category. For instance, establish clear eligibility rules when cancer co-exists with another pathology.
3. Co-Design with Clinicians: Include the annotating radiologists directly in the decision-making loop to validate these protocols and establish formal arbitration mechanisms for disagreements.

Ambiguity is not what undermines trust in medical AI; unmanaged ambiguity is. When these ambiguities are left unmanaged at the very beginning of the data stage, they do not disappear. They are inherited by the system. They cascade silently through the pipeline and inevitably arise at the end during validation, deployment, and post-deployment monitoring, where the model shows its clear limits.

Technical Development Hand in Hand with Clinical Coordination

A safe-by-design system requires human coordination capacity to absorb and manage this reality from day one to the final clinical impact. This is exactly where the technical layer meets the human coordination layer. Where the time a tool frees actually goes, and who on the team absorbs the output are discussed by István Borbíró of Nexus Recognita in this article.

When uncertainty is acknowledged, structured, and thoroughly documented, it ceases to be a defect to eliminate. It becomes a part of a rigorous, defensible, and safe-by-design trustworthy system.

◆ I work with medical imaging AI teams on questions related to data readiness, evaluation rigor, and technical defensibility across the AI R&D lifecycle.
Feel free to explore my R&D Leadership & Advisory page or to get in touch if these topics resonate with the challenges you are facing.