SenseBench Leaderboard

Method Notes

Public rows are rebuilt from verified run artifacts in the repository. Accuracy is recomputed from predictions, and official runs are checked against the registered dataset hash and prompt ID.

The ± value under each accuracy is half the width of a fixed-seed bootstrap 95% confidence interval. The small range under each rank lists the positions a run could plausibly occupy among the visible rows given overlapping intervals.

The Score against and Sense granularity controls re-score every run and baseline against one of nine scoring schemes: the lexEN v1, Maru 2022 (ALLamended), or original Raganato 2017 gold labels, each at WordNet 3.0 fine-grained sense or at one of two coarse concept levels — Glite or CSI. A prediction is coarse-correct when its WordNet sense maps to the same coarse concept as a gold sense (so coarse accuracy is always at least the fine-grained accuracy). The default and official score is lexEN v1 · WordNet fine-grained. See label schemes and coarsening for what each option means.

The Pareto frontier uses the visible rows and the selected chart metric (cost per million items or machine-hours per 1M items): higher accuracy is better, and a lower metric value is better. Starred rows are on the frontier.

Machine-hours per 1M items is recorded for self-hosted runs only. It measures the per-item evaluation loop on the recorded machine, excluding model and asset loading; cloud API runs are excluded from that chart. Because it measures time rather than money, it is the one hardware metric no pricing assumption touches: multiply it by whatever you would pay per hour to cost a model on your own terms.

Cost per million items prices self-hosted runs at a fixed reference rate for their GPU class, not at the rate each machine happened to be rented at, so a lucky or unlucky spot price neither flatters nor penalizes a model. Each run still records the rate it actually paid, and its run page shows both. As of 2026-07-14 the reference rates are $1.09/h for A100 80GB, $2.26/h for H100 80GB, $3.66/h for H200 141GB, $6.72/h for B300 288GB. A cost marked * is the exception: that run is on a GPU class we have no reference rate for, so it is priced at the rate its machine was rented at and is not on the same basis as the rows around it. Such a run can still reach the Pareto frontier, and its star is marked ★* to show the frontier position rests on that non-comparable cost. See methodology for how the rates are set.

Reference baselines are supervised WSD systems and MFS scored from per-item predictions on the same dataset items; dashed chart lines show them for context.

Comparing selected runs on the same dataset uses McNemar's test on paired per-item correctness, which detects differences that overlapping confidence intervals can miss.