Eval packs

An eval pack is a set of questions with known, verified answers that you give your own agent to measure how often it gets healthcare facts right. There are three packs:

Eval packs are on the Team plan; other plans get 403 plan-upgrade-required (see Plans). Calls to /v1/eval-packs are not metered.

Where the answers come fromCopy Anchor Link

Every answer is a fact read from a published dataset, never written by a model. Questions come from templates ("Is ICD-10-CM E11.9 billable?") filled from the data, with subjects picked in a stable order, so the same data gives the same pack. When a version is built, each answer is read again from the database and from the API response its citation names; a question is kept only when they all agree. Questions never use CPT or CDT descriptions or data a license keeps private.

Each question cites where its answer is:

{
  "id": "icd10cm-billable:E119",
  "template": "icd10cm-billable",
  "question": "Is ICD-10-CM code E11.9 billable (valid on a claim as the code itself)? Answer true or false.",
  "answer_type": "boolean",
  "answer": true,
  "as_of": "2026-10-23",
  "citations": [
    {
      "dataset": "icd10cm",
      "data_version": "fy2027",
      "key": "E119",
      "request": "GET /v1/codes/icd10cm/E11.9",
      "field": "data.billable"
    }
  ],
  "difficulty": "easy",
  "tags": ["icd10cm", "billable"]
}

as_of is the date the answer holds for. A question's id stays the same across versions, so you can compare runs.

VersionsCopy Anchor Link

Pack versions are YYYY.MM.N (e.g. 2026.10.1). A new version is built when the datasets behind a pack publish new releases and the questions or answers change; a version never changes once published. data.datasets lists the dataset versions the answers were verified against, with each dataset's license_terms.

curl https://api.healthcare.dev/v1/eval-packs -H "Authorization: Bearer $HDEV_API_KEY"
curl "https://api.healthcare.dev/v1/eval-packs/coding?version=2026.10.1" \
  -H "Authorization: Bearer $HDEV_API_KEY"

GET /v1/eval-packs lists every pack with its versions, newest first. GET /v1/eval-packs/{pack} returns the newest version, or the one version names, with every question and answer.

Run a pack against your agentCopy Anchor Link

1. Download a version and pin it: keep the version you tested with your results.
2. Give your agent each question as it would get a user's question, with its usual tools.
3. Ask it for the answer in the question's answer_type (the question says how to answer), and collect { id: answer }.
4. Grade the answers (below), and compare scores per tag between runs.

import { HealthcareDev } from "@healthcare.dev/sdk";

const hdev = new HealthcareDev();
const { data: pack } = await hdev.evalPacks.get("coding");
const answers: Record<string, unknown> = {};
for (const q of pack.questions) answers[q.id] = await myAgent.answer(q.question, q.answer_type);
const { data: result } = await hdev.evalPacks.grade("coding", answers, { version: pack.version });
console.log(result.score, result.by_tag);

from healthcare_dev import HealthcareDev

hdev = HealthcareDev()
pack = hdev.eval_packs.get("drugs")["data"]
answers = {q["id"]: my_agent.answer(q["question"], q["answer_type"]) for q in pack["questions"]}
result = hdev.eval_packs.grade("drugs", answers, version=pack["version"])["data"]
print(result["score"], result["by_tag"])

Don't give your agent the answer or citations: the citations name the request that shows the answer.

GradingCopy Anchor Link

POST /v1/eval-packs/{pack}/grade (optionally ?version=) takes { "answers": { id: answer } } and returns the score. Grading is a plain comparison, no model, and nothing is stored. You can also grade locally with the same rules:

{
  "pack": "coding",
  "version": "2026.10.1",
  "total": 212,
  "answered": 210,
  "correct": 198,
  "score": 0.9387,
  "by_tag": { "billable": { "total": 60, "score": 0.98 } },
  "unknown_ids": [],
  "results": [
    {
      "id": "icd10cm-billable:E119",
      "correct": true,
      "score": 1,
      "expected": true,
      "given": "yes",
      "reason": "match"
    }
  ]
}

reason is match, mismatch, missing (no answer sent; it scores 0) or wrong_type (an answer that can't be read as the type, such as "maybe" for a boolean). score is the mean over every question in the version. Ids that aren't in the version are listed in unknown_ids and ignored.

ErrorsCopy Anchor Link

Child keys need the eval-packs scope (or no scopes) to call these routes; see Child keys.