Grade answers to an eval pack
POST /v1/eval-packs/{pack}/grade
POST https://api.healthcare.dev/v1/eval-packs/{pack}/gradeGrades your agent's answers (answers: question id to answer) against one version of a pack (the newest, or version). Deterministic comparison, no model: code ignores case, dots, hyphens and spaces; boolean takes true/false or yes/no; text ignores case, spacing and punctuation; set scores the overlap (Jaccard); number allows tolerance. Questions you leave out score 0 as missing. Returns the overall score, the score per tag and a result per question. Nothing is stored. Team plan (403 plan-upgrade-required on other plans). Not metered.
Requires an API key: Authorization: Bearer hdev_live_.... See Authentication.
Path parameters
| Name | Type | Required | Description | Example |
|---|---|---|---|---|
pack | "coding" | "drugs" | "providers" | yes | coding, drugs or providers | coding |
Query parameters
| Name | Type | Required | Description | Example |
|---|---|---|---|---|
version | string | no | A version from /v1/eval-packs (default: the newest) | 2026.10.1 |
Request body
{
"answers": {}
}Examples
curl
curl -X POST "https://api.healthcare.dev/v1/eval-packs/coding/grade" \
-H "Authorization: Bearer $HDEV_API_KEY"TypeScript
const res = await fetch("https://api.healthcare.dev/v1/eval-packs/coding/grade", {
method: "POST",
headers: { Authorization: `Bearer ${process.env.HDEV_API_KEY}` },
});
const { data, meta } = await res.json();Python
import os
import requests
res = requests.post("https://api.healthcare.dev/v1/eval-packs/coding/grade", headers={"Authorization": f"Bearer {os.environ['HDEV_API_KEY']}"})
body = res.json()Try it
Response
Scores
{
"data": {
"pack": "coding",
"version": "2026.10.1",
"total": 1,
"answered": 1,
"correct": 1,
"score": 0,
"by_tag": {},
"unknown_ids": [
"string"
],
"results": [
{
"id": "string",
"correct": true,
"score": 0,
"expected": null,
"given": null,
"reason": "match"
}
]
},
"meta": {
"source": "cms-icd10",
"data_version": "2026.1",
"retrieved_at": "2026-10-24T12:00:00Z",
"license": "Public domain (CMS). Reference data, not medical advice."
},
"links": {
"self": "/v1/eval-packs/coding/grade",
"next": null
}
}Fields
| Field | Type | Always present | Description |
|---|---|---|---|
pack | EvalPackId | yes | coding (ICD-10-CM and HCPCS Level II), drugs (NDC Directory, RxNorm, FDA shortages) or providers (Medicare enrollment, opt-out and OIG exclusions by NPI) |
version | string | yes | Pack version, YYYY.MM.N (e.g. 2026.10.1) |
total | integer | yes | Questions in the pack version |
answered | integer | yes | |
correct | integer | yes | |
score | number | yes | Mean question score over every question |
by_tag | object | yes | The mean score per tag |
unknown_ids | string[] | yes | Answer ids that aren't in this version (ignored) |
results | EvalQuestionGrade[] | yes |
Errors
Errors are application/problem+json (RFC 9457). Branch on type, not on title. See Errors.
| Status | type | When |
|---|---|---|
| 400 | invalid-request | A parameter or the body failed validation; errors lists each field. |
| 401 | unauthorized | Missing, malformed or revoked API key. |
| 403 | plan-upgrade-required | The plan doesn't include this feature; required_plan names the plan that does. |
| 404 | not-found | No record with that id (retired records are still returned). |
| 429 | rate-limited / quota-exceeded | Too many requests per second (retry after Retry-After), or the monthly quota is used up. |