Can Metadata be trusted as Ground Truth for Medical Imaging AI?

In medical imaging AI, metadata is often used as ground truth without much discussion.

A field exists in a spreadsheet or a DICOM header. It appears to describe the variable of interest. It is therefore used as a label for model development or validation.

Yet the presence of information does not necessarily establish its reliability as ground truth.

I recently worked with GammaMetric [1] on a research project investigating whether CT reconstruction characteristics could be identified directly from image pixels, independently of DICOM metadata availability or quality.

Before developing a reconstruction classifier, however, a more fundamental question needed to be addressed:

Can the available reconstruction metadata actually be relied upon as ground truth?

Before using metadata as a reference label, it is worth establishing what the information represents, where it comes from, and how much evidence supports it. This led to a metadata traceability assessment of the QIBA CT Liver Phantom dataset [2] from The Cancer Imaging Archive (TCIA) [3].

Why the QIBA dataset?

The QIBA CT Liver Phantom dataset contains CT images acquired from anthropomorphic phantoms with replaceable liver inserts. Several acquisition and reconstruction parameters were deliberately varied, including tube current, slice thickness, reconstruction algorithm, convolution kernel and pitch [].

Because these factors were controlled during acquisition, the dataset was well suited to the research question.

Why reconstruction metadata matters for medical imaging AI

CT reconstruction algorithms influence image appearance, noise characteristics, texture and sharpness. These differences matter for AI systems, since a model validated under one set of reconstruction conditions may behave differently when those conditions change.

GammaMetric’s research investigates whether these reconstruction characteristics can be identified from image pixels themselves. Such an approach could support applications such as local acceptance testing and drift monitoring by helping verify whether incoming images remain within validated acquisition conditions.

However, any supervised classification study first needs reliable labels. If reconstruction metadata is used as ground truth, the quality of the experiment depends on the quality of that reference information.

The first step was therefore not model training. It was understanding the evidence behind the labels and answering a more fundamental question: is the ground truth defensible?

The full GammaMetric case study provides a detailed overview of the assessment, findings and recommendations.

A controlled dataset can still require data due diligence

Controlled acquisition does not automatically make ground truth straightforward.

The assessment combined several sources of information:

online dataset documentation
an accompanying metadata spreadsheet
standard DICOM tags
information extracted from the free-text SeriesDescription DICOM field
where available, manufacturer-specific private DICOM tags

The objective was not simply to identify available metadata. It was to determine which reconstruction descriptors were directly observable, which were inferred, whether independent sources agreed, and which assumptions required further validation before model development.

Findings that changed the ground-truth strategy

The assessment identified five findings that affected how reconstruction metadata could be interpreted and used as ground truth for model development.

1. Dataset documentation and available images did not fully align

The first discrepancy appeared at the dataset count level.

The online documentation indicated 642 scans. The accompanying metadata spreadsheet contained 684 unique series. The downloaded data contained 627 unique DICOM series.

For model development, this creates a practical question: which records should be used to generate labels?

The DICOM series represent the image data that can be directly verified. The metadata spreadsheet therefore needs to be cross-referenced against the available image series before labels are generated.

This may appear to be a simple data management issue. In practice, however, it can affect how the experimental population is defined and structured.

If metadata records and available images are not reconciled, the team may generate labels or design dataset splits, including stratified splits, based on records that do not accurately reflect the image series actually available for model development. Assumptions about dataset size, composition and class distributions may therefore need to be verified against the downloaded DICOM data.

The actionable question for an R&D team is not only “How many records do we have?” but “Can every label be traced to the image series that will actually enter the experiment?”

2. Data outliers may provide alternative explanations for unsatisfactory classification results

Three CT series contained only 21 slices, making them clear outliers in the slice-count distribution.

The issue is not that unusual series should automatically be excluded. It is interpretability.

Suppose a reconstruction classification model performs poorly on these images. Without identifying the unusual slice count beforehand, the team might attribute the behavior to reconstruction characteristics when another explanation is present.

Therefore, data assessment before model development helps identify alternative explanations before performance results are interpreted. Without examining potential confounding factors early, the answer may remain unclear.

Distribution of slice counts across CT series. Three series are identified as low-slice-count outliers with 21 slices each. — Figure 1. Distribution of slice counts across CT series. Three series were identified as low-slice-count outliers with only 21 slices.

3. Reconstruction descriptors were not universally available

The distribution of the standard DICOM ConvolutionKernel tag revealed three GE series for which no reconstruction kernel could be identified from the available metadata. The number of affected series was small, but the implication is broader.

For an R&D team, missing labels require an explicit decision. Depending on the research objective and the number of missing labels, the team may exclude the affected series, investigate alternative metadata sources or define another strategy for handling unlabeled data.

What matters is that missing information is identified and handled explicitly before moving forward, rather than being carried into the project as an implicit assumption.

Distribution of values in the ConvolutionKernel DICOM tag across CT series. Most series contain an identifiable reconstruction kernel, while three GE series have missing kernel information. — Figure 2. ConvolutionKernel DICOM tag distribution. Three GE series were identified without an associated reconstruction kernel descriptor.

4. Free-text metadata did not always agree with standard DICOM tags

Additional acquisition parameters had been extracted from the SeriesDescription DICOM field using regular expressions.

These values were compared with the corresponding standard DICOM tags, and discrepancies were found for both slice thickness and tube current.

A value that can be extracted is not necessarily a value that should be treated as reference information. In this case, metadata derived from SeriesDescription should not be considered reliable ground truth without additional validation against standard DICOM metadata.

Comparison between tube current values extracted from the SeriesDescription field and values recorded in the standard XRayTubeCurrent DICOM tag. Several discrepancies are visible, indicating that text-extracted metadata does not always match standard DICOM metadata. — Figure 3. Tube current mismatch between string-extracted metadata and standard DICOM metadata. Several series show inconsistent values across the two sources.

Comparison between slice thickness values extracted from the SeriesDescription field and values recorded in the standard SliceThickness DICOM tag. Several discrepancies are visible, indicating that text-extracted metadata does not always match standard DICOM metadata. — Figure 4. Slice thickness mismatch between string-extracted metadata and standard DICOM metadata. Several series show inconsistent values across the two sources.

5. Metadata reliability can depend on the manufacturer

One of the most consequential findings concerned reconstruction traceability across manufacturers.

For some GE series, reconstruction-related information could be identified through private DICOM tags, although these tags were not consistently available across all series. For the Siemens reconstruction families present in the dataset, no equivalent metadata source was identified.

Therefore, a single reconstruction labeling strategy may not be appropriate for the entire dataset. Different manufacturers may require different rules for generating reference labels. Some subsets may offer directly observable metadata, others may depend on alternative sources, and some information may remain unavailable.

This changes the experimental design. Before applying one labeling rule to heterogeneous imaging data, it is worth asking whether the underlying metadata has the same meaning, availability and traceability across every subset.

What does metadata traceability change for an AI project?

A metadata traceability assessment’s value lies in the decisions it informs. This table presents the implication resulting from every observation.

Observation	Implication
Documentation mismatch in dataset size	Ensure labels are generated only for image series actually available in the dataset.
Images with low slice count	Investigate potential confounding factors before attributing performance differences to reconstruction characteristics.
Incomplete reconstruction descriptors	Expect missing labels and define a strategy for handling unlabeled series. Given that only three affected series were identified, exclusion may be a practical and defensible option.
Discrepancies between string-extracted and standard DICOM tags	Avoid treating text-derived parameters as reference labels without verification.
Manufacturer-dependent reconstruction metadata	Reconstruction labeling strategy needs to differ between manufacturers. Manufacturer-specific label generation rules may be required.

As a result, data due diligence provides evidence for decisions that affect label generation, experimental design, validation strategy, the interpretation of model results, and post-deployment monitoring.

Ground truth is an evidence question

The broader lesson from this work is that ground truth should be treated as an evidence question.

Before using a metadata field as ground truth for AI development, several questions can help clarify whether it is fit for purpose:

What exactly does this field represent?
How was the information generated?
Is it directly observed or inferred?
Do different sources agree?
Is the information consistently available across manufacturers, sites and devices?

The purpose of asking these questions is to reduce the risk of spending months optimizing a model around labels whose meaning, provenance or reliability was never established.

In the GammaMetric project, examining metadata traceability before experimentation clarified which reconstruction descriptors could reasonably support model development, where manufacturer-specific strategies were needed, and which assumptions required further validation.

References

[1] https://gammametric.com/
[2] https://www.cancerimagingarchive.net/collection/qiba-ct-liver-phantom/
[3] https://www.cancerimagingarchive.net/

◆ I work with medical imaging AI teams on questions related to data readiness, evaluation rigor, and technical defensibility across the AI R&D lifecycle.
Feel free to explore my R&D Leadership & Advisory page or to get in touch if these topics resonate with the challenges you are facing.