Open benchmark · Healthcare AI

MedCoverBench

Can a language model decide Medicare coverage from the real policy text, cite the rule that decides it, and notice when documentation is missing?

Manoj Dobbali

Project overview · October 2026 · pre-release

The domain

Medicare decides what it pays for through written coverage policy

National Coverage Determinations

CMS publishes one NCD per item or service: who qualifies, under what conditions, with what documentation. They apply nationwide. There are 345 current NCDs.

Long and conditional

A typical NCD runs 2,000 to 15,000 characters of nested criteria: waiting periods, score thresholds, provider volume requirements, exceptions and exclusions.

AI is arriving here

Prior authorization and utilization review are being automated. Errors in either direction hurt patients or providers, and the policy text is the ground truth both sides cite.

The gap

No public benchmark tests models on real CMS coverage policy

What exists

Benchmarks built on synthetic insurance handbooks, and prior-authorization studies on small private document sets that were never released.

What is missing

Real, full-length federal policy, cases with labels traceable to exact policy sentences, and a metric for the failure that matters most: approving what the policy does not cover.

MedCoverBench

Real NCD text from the CMS Coverage API, synthetic patients, verbatim-cited criteria and an open evaluation harness.

The principle

A wrong approval and a wrong denial are different failures. A benchmark should count them separately.

MedCoverBench reports accuracy, but its headline number is the unsafe approval rate: how often a model says “covered” when the policy says otherwise.

The task

One policy, one patient, one structured decision

Full NCD text

Patient case

Model

JSON decision, quotes, missing documentation

covered

Every applicable criterion is met and documented.

not_covered

The facts show at least one applicable criterion is not met.

insufficient_documentation

Nothing is shown to be violated, but documentation the policy requires is missing.

A case from the set · NCD 20.4, implantable defibrillators

64-year-old man with a large anterior MI 21 days ago, now with LVEF 25% and NYHA class II symptoms; no revascularization targets remain. Requesting a primary-prevention ICD. He is clinically stable, with no non-cardiac disease limiting survival. A nurse practitioner held a shared decision-making encounter using an evidence-based ICD decision tool.

"decision": "not_covered",
"violated": ["C6"],
"policy span": "Had an MI within the past 40 days"

Every other criterion is satisfied and stated. The one disqualifier is a date inside a 40-day exclusion window. Qwen 2.5 7B approved it.

Gold labels come from a per-policy checklist. Each criterion carries a span copied verbatim from the NCD, so citations can be scored automatically.

How a policy becomes test cases

Controlled perturbation of a verified checklist

Read the NCD

The same three fields the model sees, from the CMS Coverage API.

Extract criteria

Atomic requirements, each typed and anchored to a verbatim span.

Perturb one thing

All met gives covered. One contradicted gives not covered. One absent gives insufficient documentation.

Validate

A script rejects any spec whose spans or labels don’t check out.

Cases are written to discriminate: exact boundary values, violations buried among satisfied details, exceptions the policy grants, and distractors that sound disqualifying but aren’t.

Guardrails

What every case must satisfy

Verbatim spans

Each criterion’s span must appear in the NCD text, normalized only for quotes, whitespace and hyphens.

Consistent labels

Covered cites nothing. Not covered cites a violated criterion. Insufficient documentation cites only missing items.

NCD text only

No rules from local policies (LCDs) and no clinical knowledge the policy doesn’t state. LCD rules appear only as distractors.

Unambiguous or dropped

If the policy wording is unclear, no case may depend on it. The ambiguity is logged for reviewers.

Synthetic patients

No real people, no PHI. ICD-10-CM and HCPCS Level II only; no CPT codes or descriptors.

Disclosed provenance

Cases were drafted with AI assistance. Every label stays “draft_unreviewed” until a clinician or certified coder signs off.

The dataset today

From a 10-policy pilot to 50 policies

NCDs with cases

513

labeled cases

434

verbatim-anchored criteria

345

NCDs downloaded and indexed

Decision mix

174

249

CoveredNot coveredInsufficient documentation (up from 9 in the pilot)

Coverage across medicine

Policies by clinical area

Cardiovascular procedures

DME and implanted devices

Other procedures and drugs

Screening and prevention

Labs, genetics and imaging

Neurostimulation

Respiratory and sleep

Transplant and cell therapy

Examples: TAVR and mitral TEER, left atrial appendage closure, deep brain stimulation, anti-amyloid antibodies for Alzheimer’s, lung cancer screening, home oxygen, stem cell transplantation, cochlear implants.

Scoring

Six numbers per model

Unsafe approval rate

Predicted covered when the gold answer is not covered or insufficient documentation. The headline safety metric.

Over-denial rate

Gold covered, predicted anything else. The cost to patients who qualify.

Citation recall

Share of gold criteria the model quoted, matched by at least 50% token overlap with the criterion’s span.

Accuracy

Exact match on the three-way decision.

Macro-F1

Averaged over the three classes, so a model can’t score well by never choosing the rare one.

Parse failure rate

Replies with no valid JSON decision. Counted as wrong.

Smoke test · Qwen 2.5 7B Instruct · 506 of 513 cases

The pipeline works end to end, and a small model is easy to fool

62.8%

accuracy

22.0%

unsafe approval rate

27.6%

over-denial rate

70.7%

citation recall

Macro-F1 0.55. Parse failures 0.6% (3 replies). One small open model, run only to prove the harness; the paper’s baselines will span open and frontier models.

What the model gets wrong

The errors are the kind a reviewer would catch

Missing documentation becomes denial

Of 88 missing-documentation cases scored, it called 65 not covered and got only 16 right.

Waiting periods ignored

Approved a defibrillator 21 days after a heart attack, against the policy’s 40-day exclusion.

Exact boundaries denied

Rejected a CHA₂DS₂-VASc score of exactly 3 and an AHI of exactly 15, both of which qualify.

Exceptions missed

Demanded shared decision-making for a cardiac arrest survivor, where the policy waives it.

How it was built

Six independent jobs ran in parallel, each owning its own files

Job	Output	Notes
Cardiovascular	103 cases	Swapped leadless pacemakers, whose only criterion is study enrollment
Neuro, MSK, DME	106 cases	Swapped two policies that are study-only or a blanket non-coverage
Screening, labs, imaging	98 cases	Dropped cases whose labels would have been ambiguous
Transplant and other	105 cases	Logged 9 policy ambiguities for expert review
Harder DBS cases	13 cases	The old set scored 10/10; violations were stated up front
Release packaging	3 files	Hugging Face export, dataset card, licenses

Every job worked from the same written rules and the same validator. Nothing merged unless it passed.

A bug the process caught

Policy text was silently losing every phrase that began with “<”

CMS sends:   Cardiac Index (CI) &amp;lt; 2.2 L/min/m2, while not on inotropes, and also meet one of the following:&lt;ul&gt;
We stored:   Cardiac Index (CI)
Now stored:  Cardiac Index (CI) < 2.2 L/min/m2, while not on inotropes, and also meet one of the following:

Cause

Double-escaped HTML was unescaped twice before tags were stripped, so a literal “<” became a tag opener and everything up to the next “>” was deleted.

Impact

12 of 345 NCDs, 4 of them in the benchmark. All 434 spans still matched the corrected text, and no gold label depended on the lost text.

Found because case authors reported sentences that stopped mid-thought. The fix unescapes once, strips tags, then unescapes the text, and now keeps the original HTML.

Licensing and scope

Only what can be shared freely

In: NCDs

National policy text is a US government work, downloaded from the CMS Coverage API without a license token.

Out: LCDs and billing articles

956 local policies contain AMA CPT content that needs a license and can’t be redistributed.

Release terms

Data under CC BY 4.0, code under Apache 2.0. Each release records its CMS snapshot date, since CMS updates weekly.

Not medical or billing advice. The benchmark measures reasoning over policy text; it doesn’t decide anyone’s coverage.

Roadmap

From draft labels to a public leaderboard

Expert review

A clinician or certified coder reviews every label, starting with the logged ambiguities. This gates everything after it.

Baselines

Open models at several sizes plus frontier APIs, including Claude.

Release

Hugging Face dataset and card, GitHub harness, arXiv paper.

Held-out test set

Private labels scored through a leaderboard, to limit training-data contamination.

Long-context track

The target NCD buried among 10 to 50 other policies, from 32k to 200k tokens.

Domain models

Long-context models adapted to coverage policy, evaluated on the same harness.

Get involved

Coverage decisions are written down. Models should be able to read them.

Reviewers

Clinicians and certified coders who can check labels against NCD text.

Contributors

A research engineering internship is open at manojdobbali.com/careers.

Model builders

The harness runs against any OpenAI-compatible endpoint.

MedCoverBench · open data, open code

← → to move · G for all slides · F for fullscreen