Methodology

Scoring

Accuracy is the fraction of dataset items whose predicted WordNet sense key matches the gold sense key set.

Confidence intervals are bootstrap intervals over item correctness with a fixed seed, shown as a ± half-width next to accuracy.

Rank ranges list the positions a run could plausibly occupy among the visible rows given overlapping 95% confidence intervals.

The compare view tests paired per-item differences between runs on the same dataset version with McNemar's test, which is far more sensitive than comparing overlapping intervals.

Reference baselines (MFS, BEM, ESCHER, ConSeC, Glite LENS) are scored from per-item system predictions on exactly the same dataset items as the model runs.

Verification

Every public run is reloaded, replayed, and checked before it appears on the site.

Verification checks run metadata, prompt references, dataset hashes, candidate sets, raw output extraction, vote decisions, and correctness.

Ranking

Runs sort by higher accuracy, then lower cost per million items when available, then newer creation time. For self-hosted runs that cost is priced at a fixed reference rate per GPU class rather than at the rate the machine was rented at; see Reference GPU Rates below.

The default leaderboard view lists every verified run; the collapsed view keeps only the best verified run per model and dataset version, across prompts and reasoning efforts.

Self-Hosted Runs

Self-hosted runs record the GPU machine they ran on and a benchmark time that covers only the per-item evaluation loop, excluding model download, weight loading, and inference engine startup.

Machine-hours per 1M items is the benchmark time divided by the item count, scaled to one million items and expressed in machine hours; it is comparable only across runs on the same GPU configuration.

When the machine's hourly rate is known, the run's own recorded cost is machine time multiplied by the rate that machine was actually rented at (cost source machine_time_estimate); otherwise cost is unavailable.

Reference GPU Rates

Rented GPU prices move between rentals: the same H100 class cost between $1.60/h and $2.54/h across the rentals behind this leaderboard. Pricing each model at whatever the spot market charged the hour it happened to run would say more about the timing of the rental than about the model, so actual cost is not comparable across self-hosted runs.

The leaderboard therefore prices every machine-time run at a fixed reference rate for its GPU class, and that is what the cost per million items column, the cost axis of the Pareto charts, and the cost tiebreak in ranking use. As of 2026-07-14 the reference rates are $1.09/h for A100 80GB, $2.26/h for H100 80GB, $3.66/h for H200 141GB, $6.72/h for B300 288GB.

Each reference rate is the mean of what we paid for that GPU class across distinct rentals, so one long-lived instance serving many runs does not pull the rate toward its own price. The rates are frozen constants rather than a rebuild-time average, so a new submission never silently re-prices the runs already on the board. The as-of date above is when they were last recomputed from those rentals; they are not tracked against the live spot market, so they will drift from current prices until they are next revised.

Each run keeps the rate it actually paid and the cost computed from it; run detail pages show both figures.

A machine-time run on a GPU class we have no reference rate for is the one exception: it keeps the cost computed from the rate its machine was rented at, and its cost is marked with an asterisk on the leaderboard. Such a row carries the same spot price distortion described above and is not on the same basis as the reference-priced rows, so treat its cost, and its position on the cost axis of the Pareto charts, as indicative only.

Such a run is still eligible for the Pareto frontier, because excluding it would hide a result rather than qualify it. Its frontier star is marked with an asterisk instead: a cheap rental can put a row on the frontier that a reference-priced run would not reach, so a marked star is a claim about the rental price as much as about the model.

Comparing Across GPUs and Quantization

Self-hosted rows record the quantization used (for example fp8 or bf16) alongside the GPU. The same model may appear at different quantization on different GPUs because each GPU is run at its best practical configuration: native fp8 on H100 and H200, and bf16 on A100, which has no native fp8 hardware. A cross-GPU accuracy difference for one model therefore reflects both the hardware and the quantization, and the two should not be attributed to the GPU alone.

Quantized inference is not bit-identical across GPU architectures, so the same model under greedy decoding can produce slightly different accuracy on different GPUs. Small cross-GPU accuracy differences for an identical model and quantization are expected and are a property of the kernels, not a measurement error.

Throughput is measured at a fixed per-GPU concurrency, reported as machine-hours per 1M items, so figures are comparable at a standard load rather than at each model's individually tuned optimum. Accuracy is computed under greedy decoding (temperature 0) and is deterministic given the weights; reported confidence intervals are fixed-seed bootstrap intervals and pairwise comparisons use McNemar's test.

The per-item generation cap is a runaway guard, not a scoring knob: it is set high enough that compliant models reach their answer and stop at the end-of-sequence token well before the cap. Submissions whose outputs are truncated by the cap on a material fraction of items are rejected by verification.