Evals

Why we publish evals

A statistic is right or it is not. We publish the cases, the reference answers, the tolerances and every run so anyone can check our numbers, and so a failure sits on the record next to the pass. The same cases can be run against any tool, ours included.

How a case is built

1. Step 1

   Task

   One research question, in the words a study team would use. The catalogue covers scoring, cleaning, description, group comparison, association, repeated measures, data with no true effect, and questions the tool must refuse.

2. Step 2

   Pinned public data

   Source URL, licence and sha256 are pinned; a changed byte refuses the run. Raw rows never enter the repository. A file over 5,000 rows is subsampled with a recorded seed.

3. Step 3

   Reviewed reference answer

   Every expected number carries its source: a published paper, a textbook, a package manual, a manual derivation, or a check computed at build.

4. Step 4

   R twin

   Every printed number is recomputed independently in base R. When the printed value and the recomputation disagree beyond tolerance, the case fails before it runs.

5. Step 5

   Tolerances

   Fixed per kind of number, published with the case, and never widened to make a run pass.

What we score

Correct

Within tolerance on the test that was pinned. A statistic tied to a test is asserted only when that test was actually run; otherwise it is marked not evaluable, never a pass.

Fabricated

A statistic-shaped number that matches nothing: not the data, not the R twin on the rows shown or on the whole file, not the printed source, not a convention.

No statistic

A run that returned no statistic computed from the data at all, however well it explained what it would have done.

Recall

A match on rows the tool never saw. That is the published dataset remembered, not analysed, and it never counts as a pass.

Each case runs 3 times. The result and the report must be byte-identical across runs, and a case passes only when every declared check passes; a missing output fails the row.

Current results

The first 3 of 31 designed cases, run offline through the product's own preparation and analysis path with the plan and cleaning steps pinned, so no model is called. 561 checks pass, 0 fail, 0 not evaluable; routes 6/6 as pinned.

The baseline: one plain model call

One plain model call per run: the question, the first 30 rows of the file, the codebook, and one instruction to analyse and report the statistic, p-value, effect size and interval. No system prompt, no tools, no code execution. Model claude-fable-5-1, 3 runs per case, 9 calls, $1.27 in total.

Read the two tables side by side, not as a head-to-head: they are scored by different runners against different check sets. On the textbook cases the plain call ties. On the 2,800-row file it could not see the data, so it explained, recited or declined. A single runner scoring both is the next step.

What broke in live runs

On 2026-09-23 we ran 6 public datasets with published answers through the product's own screens on our test environment. 3 came back correct, 1 ran a substituted analysis, 2 were never delivered. Delivered studies took 203 to 251 seconds from question to report and cost $0.38 to $0.56; the six together cost $4.63.

• ISwR energy, n = 22

  Correct

  Do lean and obese women differ in 24-hour energy expenditure?

  Welch t = -3.8555, df = 15.9187, p = .0014, difference -2.2316 MJ, 95% CI [-3.4592, -1.0041]. Exact match to the published answer.

  Called a metabolic-ward measurement a survey; per-group medians and the box plot promised in the plan were not delivered.

  3 min 36 s · $0.47

• R PlantGrowth, n = 30

  Correct overall test, second half unanswered

  Do the treatment groups differ in dried plant weight, and which groups differ?

  F(2, 27) = 4.846088, p = .016, eta squared 0.264148. Exact overall test; no pairwise contrasts although asked.

  No group means in the table; no figure; group labels rendered with literal quotes.

  3 min 23 s · $0.38

• medicaldata licorice_gargle, n = 235

  Not delivered

  Does licorice gargle reduce sore throat at 30 minutes after surgery?

  First draft right: difference -0.75, 95% CI [-1.06, -0.44], Welch t(157.30) = -4.80. The checker could not bind the 10 in the outcome's name "(0-10 scale)", two automatic corrections rewrote the answer into stubs, and the study parked with Approve disabled.

  Held 17 minutes, then stopped.

  Not delivered · $1.71

• JSE FEV, n = 654

  Correct

  Do children who smoke have lower lung function after adjusting for age, height and sex?

  Adjusted smoking coefficient -0.0872 L, 95% CI [-0.2036, 0.0291], p = .141, R squared 0.7754. Exact to four decimals; the confounding reversal read correctly.

  Codes 0 and 1 printed instead of labels; duplicate section headings.

  4 min 11 s · $0.56

• medicaldata opt, n = 823

  Not delivered

  Did periodontal treatment reduce preterm birth versus control?

  The approved plan promised a chi-square test with a difference in proportions. The planner then refused the yes/no primary outcome twice, 4 minutes after Begin, and offered only "Send back to preparation".

  Held 15 minutes, then stopped.

  Not delivered · $1.03

• survival lung, n = 228

  Substituted analysis

  Is survival different for men and women?

  A Welch t on observed follow-up days, t(196.51) = -1.98, p = .049, with a footnote that censoring was not modelled. The published answer is a log-rank test, p = 0.001, hazard ratio 0.59.

  Honest limitation, wrong test; the headline still reads as significant.

  3 min 35 s · $0.48

What we found and are fixing

1. 01

   A yes/no primary outcome is refused after the study starts

   The plan a customer approves promises a chi-square test with a difference in proportions. The analysis contract accepts only continuous comparisons as the primary answer, so the study stops 4 minutes in. Any trial with a binary primary endpoint fails at the same seam.

   Fix identified, not yet shipped

2. 02

   Digits inside an outcome's name are checked as numbers, and the correction loop makes the report worse

   The 10 in "(0-10 scale)" could not be bound to a result. Two automatic redrafts deleted the mean difference and interval from the answer, and the owner was left with Approve disabled.

   Fix identified, not yet shipped

3. 03

   Codes reach the report instead of labels

   Arms and sexes print as "0" and "1" although the codebook defines the labels, and each such token lowers the "N of M statements verified" count.

   Fix identified, not yet shipped

4. 04

   Plans promise per-group summaries and box plots the report does not carry

   Every approved plan step named per-group descriptives and box-and-strip plots; no delivered report had them, and the ANOVA table carried no group means.

   Fix identified, not yet shipped

5. 05

   A survival question gets a substituted test, not a refusal

   Survival and censoring are not in the written not-supported list, so the plan ran a two-group test on observed days with a footnote instead of refusing before the study began.

   Fix identified, not yet shipped

6. 06

   The study-type question recommends Survey by default, and the label colours the report

   With no instrument words in the question, the intake recommends Survey; the report then reads "Cross-sectional survey analysed on n = 22 women" for a metabolic-ward measurement.

   Fix identified, not yet shipped

Status as of 2026-09-23. A finding leaves this list when its fix has shipped and the case that caught it passes.

What held

• Every delivered statistic matched R to the printed precision.
• The reports say what they did not do: no pairwise contrasts pre-specified, diagnostics reported not corrected, censoring not modelled.
• 3 to 4 minutes from question to delivered report on the four studies that finished; $0.38 to $0.56 each.

Run it yourself

The cases, the R twins and the runner are under benchmarks/biostat-evals/ in the Sutrix repository, today private and on open pull requests, not yet on the released code (pull requests #1047 and #1048). Opening the cases and the runner is part of the Hugging Face (Hugging Science) work, in progress. Ask us for access in the meantime.

The commands below are the runner's own. They run once those pull requests have merged; until then they show what the run looks like, not something to type today.

1. Pin the public inputs into the local library (once)

   bun benchmarks/biostat-evals/src/fetch.ts

2. Run every case offline, three repeats

   bun benchmarks/biostat-evals/src/run.ts --cases all --mode offline --repeats 3

3. Recompute every countersign with the R twin (needs Rscript)

   bun benchmarks/biostat-evals/src/run.ts --cases all --countersign

4. Run the tests

   cd benchmarks/biostat-evals && bun test

Contribute a case

A case is one folder. It needs a public dataset whose licence allows the use, a question, and a printed answer you can cite. We add the R twin and the tolerances with you, and your name goes on the case.

case.json

the task, the pinned data (URL, licence, sha256), the expected numbers with their source and tolerance, and the R countersign

study-context.json

the question and materials as a study team would enter them

pinned/plan.json and pinned/ops.json

the analysis plan and cleaning steps the offline run replays, so no model is called

expected/clean_dataset.csv

the prepared data the run must reproduce, committed only when the licence allows redistribution

checks/<id>.R

the base-R twin that recomputes every printed number