Release · November 2026

A clinician asks a question.A 4B model writes the SQL.

Arxo Materia EHRSQL 0.1 reaches 67.5% on EHRSQL in MedHELM, and 66.8% with a 0.8B model. The best result on the MedHELM leaderboard for this benchmark is 63.2%. MedHELM’s own evaluator did the scoring.

The task

Answer a clinician’s question about hospital records with one SQL query.

Lab results, prescriptions, diagnoses, costs, lengths of stay.

Materia EHRSQL

Correct

67.5%

1,322 of 1,959 questions.

EHRSQL as MedHELM runs it: validation and test splits. One training run.Open the repository

Two sizes, with and without training

The same 1,959 questions. The same prompt. The same evaluator.

  • Materia EHRSQLThe 4B model writes the query on its own.

    67.5%1,322 / 1,959

  • Materia EHRSQLThe 0.8B model writes the query on its own.

    66.8%1,309 / 1,959

  • Qwen3.5-4B, no trainingThe base model, MedHELM’s prompt.

    16.5%324 / 1,959

  • Qwen3.5-0.8B, no trainingThe base model, MedHELM’s prompt.

    3.3%65 / 1,959

With a fifth of the parameters, the 0.8B model scores 13 questions fewer than the 4B model. Each Materia row is one training run. The untrained rows use the same prompt and the same token limit; they were scored with the evaluator’s comparison outside helm-run, with a limit of 60 seconds per query.

For 302 of the 1,959 questions the reference query returns no rows, and the evaluator credits them only to an empty answer. The Materia models write a query for every question. On the other 1,657 questions the 4B model scores 79.8% and the 0.8B model 79.0%.

Next to the MedHELM leaderboard

The EHRSQL column of the leaderboard, release v6.0.0 of 9 November 2026, on the same 1,959 questions. The circle is the model’s size: its area is the number of parameters.

The Arxo Materia models were trained on the training split of EHRSQL. The leaderboard models are general models that answer from the prompt alone. Sizes are the ones in the model names; a dash means the name gives none.

Other rows of the same column were measured on a different number of questions; they are listed in the repository.

Training and evaluation questions

EHRSQL builds its training, validation and test questions from the same set of question templates. Of the 1,959 evaluation questions, 641 have a training question with the same template and the same values, and 8 are word for word identical to one. These questions were kept in the training data.

  • Same template and values as a training question: 641

    The 4B model answers 448 (69.9%). The 0.8B model answers 434 (67.7%).

  • All other questions: 1,318

    The 4B model answers 874 (66.3%). The 0.8B model answers 875 (66.4%).

What we did not do

  • Train on the test

    No validation or test question was used to train the model. The two splits were opened only to count the overlap and to measure.

  • Tune the prompt

    The prompt and the generation settings are MedHELM’s.

  • Grade ourselves

    Each score of an Arxo Materia model is the execution accuracy computed by MedHELM’s evaluator.

The score comes from helm-run with the unmodified ehr_sql run entry of PacificAI/medhelm at commit 8843816c33. The evaluator ran on SQLite 3.46.1.

How it was taught

Kept.

What is public

The repository holds the models, what they answered and how each answer was scored. Arxo Materia EHRSQL 0.1 is the second Arxo Materia release, after Materia MedCalc.

Arxo-Materia-4B-EHRSQL-0.1
Public.
Arxo-Materia-0.8B-EHRSQL-0.1
Public.
What the models answered on all 1,959 questions
Public.
Per-question scores from MedHELM’s evaluator
Public.
Checksums of every file
Public.

Check it yourself

The models run on their own, and MedHELM’s helm-run measures them. Arxo is in early access: to run it on your own data, or to verify the result independently, write to us.

Open the repositoryhi@arxo.io All notes