# Eval packs URL: https://docs.healthcare.dev/eval-packs An eval pack is a set of questions with known, verified answers that you give your own agent to measure how often it gets healthcare facts right. There are three packs: | Pack | What it asks | | ----------- | ----------------------------------------------------------------------------------------------------------- | | `coding` | ICD-10-CM and HCPCS Level II: the code for a description, billable or not, a code's parent, valid on a date | | `drugs` | NDC Directory, RxNorm and FDA shortages: labelers, term types, ingredients, current shortage or not | | `providers` | By NPI: Medicare enrollment and its states, Medicare opt-out on a date, OIG exclusions recorded | Eval packs are on the Team plan; other plans get `403` `plan-upgrade-required` (see [Plans](/plans)). Calls to `/v1/eval-packs` are not metered. ## Where the answers come from Every answer is a fact read from a published dataset, never written by a model. Questions come from templates ("Is ICD-10-CM E11.9 billable?") filled from the data, with subjects picked in a stable order, so the same data gives the same pack. When a version is built, each answer is read again from the database and from the API response its citation names; a question is kept only when they all agree. Questions never use CPT or CDT descriptions or data a license keeps private. Each question cites where its answer is: ```json { "id": "icd10cm-billable:E119", "template": "icd10cm-billable", "question": "Is ICD-10-CM code E11.9 billable (valid on a claim as the code itself)? Answer true or false.", "answer_type": "boolean", "answer": true, "as_of": "2026-10-23", "citations": [ { "dataset": "icd10cm", "data_version": "fy2027", "key": "E119", "request": "GET /v1/codes/icd10cm/E11.9", "field": "data.billable" } ], "difficulty": "easy", "tags": ["icd10cm", "billable"] } ``` `as_of` is the date the answer holds for. A question's `id` stays the same across versions, so you can compare runs. ## Versions Pack versions are `YYYY.MM.N` (e.g. `2026.10.1`). A new version is built when the datasets behind a pack publish new releases and the questions or answers change; a version never changes once published. `data.datasets` lists the dataset versions the answers were verified against, with each dataset's `license_terms`. ```sh curl https://api.healthcare.dev/v1/eval-packs -H "Authorization: Bearer $HDEV_API_KEY" curl "https://api.healthcare.dev/v1/eval-packs/coding?version=2026.10.1" \ -H "Authorization: Bearer $HDEV_API_KEY" ``` `GET /v1/eval-packs` lists every pack with its versions, newest first. `GET /v1/eval-packs/{pack}` returns the newest version, or the one `version` names, with every question and answer. ## Run a pack against your agent 1. Download a version and pin it: keep the `version` you tested with your results. 2. Give your agent each `question` as it would get a user's question, with its usual tools. 3. Ask it for the answer in the question's `answer_type` (the question says how to answer), and collect `{ id: answer }`. 4. Grade the answers (below), and compare scores per tag between runs. ```ts import { HealthcareDev } from "@healthcare.dev/sdk"; const hdev = new HealthcareDev(); const { data: pack } = await hdev.evalPacks.get("coding"); const answers: Record = {}; for (const q of pack.questions) answers[q.id] = await myAgent.answer(q.question, q.answer_type); const { data: result } = await hdev.evalPacks.grade("coding", answers, { version: pack.version }); console.log(result.score, result.by_tag); ``` ```python from healthcare_dev import HealthcareDev hdev = HealthcareDev() pack = hdev.eval_packs.get("drugs")["data"] answers = {q["id"]: my_agent.answer(q["question"], q["answer_type"]) for q in pack["questions"]} result = hdev.eval_packs.grade("drugs", answers, version=pack["version"])["data"] print(result["score"], result["by_tag"]) ``` Don't give your agent the `answer` or `citations`: the citations name the request that shows the answer. ## Grading `POST /v1/eval-packs/{pack}/grade` (optionally `?version=`) takes `{ "answers": { id: answer } }` and returns the score. Grading is a plain comparison, no model, and nothing is stored. You can also grade locally with the same rules: | `answer_type` | Correct when | | ------------- | ---------------------------------------------------------------------------- | | `code` | The same code, ignoring case, dots, hyphens and spaces (`e119` is `E11.9`) | | `boolean` | `true`/`false`, or the strings `yes`/`no`/`true`/`false` | | `text` | The same text, ignoring case, spacing and punctuation | | `set` | The same set of codes, in any order; the score is the overlap (shared / all) | | `number` | Within `tolerance` of the answer | ```json { "pack": "coding", "version": "2026.10.1", "total": 212, "answered": 210, "correct": 198, "score": 0.9387, "by_tag": { "billable": { "total": 60, "score": 0.98 } }, "unknown_ids": [], "results": [ { "id": "icd10cm-billable:E119", "correct": true, "score": 1, "expected": true, "given": "yes", "reason": "match" } ] } ``` `reason` is `match`, `mismatch`, `missing` (no answer sent; it scores 0) or `wrong_type` (an answer that can't be read as the type, such as `"maybe"` for a boolean). `score` is the mean over every question in the version. Ids that aren't in the version are listed in `unknown_ids` and ignored. ## Errors | Status | `type` | When | | ------ | ----------------------- | ---------------------------------------------------- | | `400` | `invalid-request` | An unknown pack, a malformed `version` or a bad body | | `403` | `plan-upgrade-required` | The plan isn't Team | | `404` | `not-found` | No such version, or the pack has no version yet | Child keys need the `eval-packs` scope (or no scopes) to call these routes; see [Child keys](/child-keys).