A medical AI can list more of the diagnoses in a benchmark’s reference answer while also adding more suggestions absent from that answer. Those results can coexist because the scores count different things. Agreement across repeated answers adds another measurement: how much the output changes.
The public MedLDR implementation provides a concrete way to understand these distinctions. Its README identifies it as the official implementation of The accuracy-safety trade-off of large reasoning models in clinical diagnostics, also linked from coauthor Linwei Chen’s website.
This explanation draws on static code inspection on 4 November 2026, pinned to commit a58dbdaf9554f4326567db399052467490174a5c. It covers the inspected 40-code LDR-MIMIC scoring path. We did not run the code, test models or inspect patient records. The full paper and supplement remained inaccessible to this review, so we cannot validate the study’s results or establish which code outputs supplied its published tables.
What the scorer receives before it counts matches
A reference list contains the diagnosis labels the evaluation uses as its answer. A prediction list contains the labels extracted from a model’s response. Matching these labels lets the scorer count coverage and extra entries, but it does not establish whether the reference captures every clinically valid diagnosis.
MedLDR’s extraction code parses structured answers and separates diagnosis codes from their descriptions. It retains repeated code entries and their order. For the list called valid_prediction, it then keeps only codes in a configured vocabulary of 40 codes.
That filtering matters: the measures below that use valid_prediction assess the filtered list. They do not count every suggestion in the original response. An initially nonempty prediction list can become empty after filtering. The separate ranked top10 list remains unfiltered.
How coverage can rise alongside unmatched suggestions
Consider one explicitly invented example. There is no patient or model result behind it. A, B, C and D are distinct labels within the allowed vocabulary, and the reference answer is [A, B].
| Measurement | Short prediction: [A] | Longer prediction: [A, B, C, D] |
|---|---|---|
| Reference labels found, or recall | 1 of 2: 50% | 2 of 2: 100% |
| Predicted labels matching the reference, or precision | 1 of 1: 100% | 2 of 4: 50% |
| Predicted entries absent from the reference | 0 | 2 |
| Share of predictions absent from the reference | 0 of 1: 0% | 2 of 4: 50% |
| Missed reference entries | 1 | 0 |
The longer list finds B, completing coverage of the reference. It also adds C and D, which are absent from that reference. The arithmetic explains how coverage can improve while unmatched suggestions increase; it is no evidence that a particular model behaves this way.
The basic metrics code calculates its main precision and recall values for each case, then averages them with equal weight per merged row. This differs from pooling every label across all cases into one fraction.
It also measures exact equality of the label sets and intersection divided by union, often called Jaccard overlap. Neither invented prediction matches the reference exactly. Both have the same Jaccard overlap, 50%: the short list shares one of two distinct labels, while the longer list shares two of four.
What an unmatched label tells you
The unmatched-label script counts each predicted entry absent from the reference. Its per-case rate divides that count by the prediction-list length. It separately counts missing reference entries and divides those by the reference-list length.
The code calls the extra-entry measure “overdiagnosis”. Here, “unmatched suggestions” describes what was actually counted. Establishing that a suggestion is clinically false, or that it caused patient overdiagnosis or harm, requires evidence beyond absence from a reference list.
List handling also affects the numbers. The basic precision and recall calculations turn labels into presence-or-absence indicators, collapsing duplicates. The unmatched-entry counter uses lists: an unmatched label repeated twice counts twice. Its empty-prediction convention assigns an unmatched rate of zero, even when reference diagnoses were missed. A zero on that measure alone therefore says little about the usefulness of an answer.
Why the primary-label score needs a closer look
In the basic metrics file, top1_accuracy checks whether the reference primary diagnosis appears anywhere in the filtered prediction list. It divides successful matches by the number of merged rows. It does not check only the model’s first prediction.
A separate grouped evaluation script compares the reference primary code with every filtered prediction and takes the best category: exact code, same configured three-digit group, same configured broad group, or the remaining category. These are configured code groupings; this inspection does not establish their clinical equivalence.
An API evaluation script invokes both evaluations. Their different definitions make the exact measurement important when interpreting an “accuracy” label. The inspected files do not establish a connection to particular published table values.
Agreement measures repeated outputs without checking the answer
The consistency code compares labels across repeated outputs. For full filtered prediction lists, pairwise agreement is the number of shared distinct labels divided by the number of distinct labels appearing in either list. Its three-way calculation uses the intersection and union across all three lists.
The code also compares prefixes of ranked lists. Position determines which labels enter a prefix; within that prefix, the overlap calculation ignores order and collapses duplicates. A separate first-label comparison checks whether the first predicted labels agree.
Reference diagnoses never enter these agreement formulas. Repeatedly producing the same incorrect label can yield perfect agreement.
There are input limits, too. Some overlap calculations have no guard against an all-empty union, and the inspected material does not pin the library version needed to establish every empty-input convention in the basic metrics. The worked example above uses unique, nonempty lists to keep its arithmetic unambiguous.
When reading a diagnosis-list benchmark, check which list reaches the scorer, what each rate divides by, and whether an “accuracy” score checks first position or membership anywhere. For consistency results, look for a separate comparison with reference answers before drawing a conclusion about correctness.
