Verified run
gpt-5-mini
gpt-5-mini-medium-reasoning-p002-lexen-v1-20260614
- Rank
- 62
- Accuracy lexEN v1 · WordNet fine
- 90.04%
- Correct
- 4377 / 4861
- Actual run cost
- $1.76
- Cost / M items
- $363
Accuracy across label schemes
The headline accuracy is lexEN v1 · WordNet fine-grained. Each cell re-scores this run's predictions against a different gold label set and sense granularity; coarse counts a prediction correct when its WordNet sense maps to the same Glite or CSI concept as a gold sense. About these label schemes → About coarsening →
Model
- Display
- gpt-5-mini
- Requested
- gpt-5-mini
- Resolved
- gpt-5-mini-2025-08-07
- Vendor
- OpenAI
- API provider
- OpenAI
- Hosting
- cloud_api
- Source
- proprietary
- License
- n/a
- Reasoning
- medium
- Model URL
- n/a
- Endpoint
- n/a
- Resolved counts
-
gpt-5-mini-2025-08-07: 4861
Reproducibility
- Created
- 2026-06-14T03:02:45.517164+00:00
- Run schema
- sensebench-run-v2
- Commit
- 78aebb2a6fd1412a1e8eab85f6209a64f8f441e8
- Dataset ID
- lexen
- Dataset version
- lexen-v1
- Dataset revision
- n/a
- Dataset items
- 4,861
- Dataset hash
- sha256:5fd4382b93f19087a1e31f6dd7d1db17c1eb17ff80fcbe1d3fdd55c0c3ecefe8
- Prompt
- p002
- Prompt version
- 0.1.0
- Repository files
- Browse in GitHub
Runtime
- Calls
- 4861
- Total tokens
- 2,554,535
- Tokens / item
- 525.5
- Total cost
- $1.76
- Cost / M items
- $363
- Cost source
- litellm_estimate
- Elapsed
- 1.22m
- Success items
- 4,861
- Monosemous
- 0
- No candidates
- 0
- No valid vote
- 0
- Invalid votes
- 0
- Transport errors
- 0
Runner
- GitHub
- vassiliphilippov
- Name
- Vassili Philippov
- Contact
- n/a
Sampling
- Temperature
- n/a
- Top P
- n/a
- Max tokens
- 8,192
- Seed
- n/a
- Extra
-
n/a
Policy
- Votes / item
- 1
- Reasks / invalid vote
- 1
- Tie break
- earliest_vote
- Monosemous
- short_circuit
Cost And Token Breakdown
Actual recorded run cost: $1.76. Cost estimate source: litellm_estimate.
Reasoning output tokens are shown separately when the provider reports them; recorded cost is stored in the output cost bucket.
Download Raw Run Files
Run Metadata
run.json
Submitted run.json with model, dataset, prompt, policy, totals, and costs.
2.3 KB
Predictions
predictions.jsonl
One JSON record per benchmark item with candidates, votes, and correctness.
5.9 MB
Raw Calls
calls.jsonl.gz
Gzipped JSONL call log with prompts, raw outputs, token usage, and costs.
1.6 MB
Worst-Bucket Examples
The medicine can only be obtained with a prescription.
Show raw prompt
user
Which numbered sense of "only" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.
Sentence: The medicine can <t>only</t> be obtained with a prescription.
Senses:
1. definition=and nothing more | examples=I was merely asking
2. definition=without any others being included or involved | examples=was entirely to blame
3. definition=with nevertheless the final result | examples=He arrived only to find his wife dead
4. definition=in the final outcome | examples=These news will only make you more upset
5. definition=except that | examples=It was the same story; only this time she came out better
6. definition=never except when | examples=call me only if your cold gets worse
7. definition=as recently as | examples=I spoke to him only an hour ago
Prediction matches gold:
lexEN ✗
Maru 2022 ✓
Raganato ✓
WordNet 3.0 senses
-
1
and nothing more
only%4:02:02:: · merely.r.01
- I was merely asking
- it is simply a matter of time
Synonyms: merely, simply, just, but
-
2
without any others being included or involved
only%4:02:01:: · entirely.r.02
- was entirely to blame
- a school devoted entirely to the needs of problem children
Synonyms: entirely, exclusively, solely, alone
Gold lexEN
-
3
with nevertheless the final result
only%4:02:00:: · only.r.03
- He arrived only to find his wife dead
- We won only to lose again in the next round
-
4
in the final outcome
only%4:02:04:: · only.r.04
- These news will only make you more upset
-
5
except that
only%4:02:06:: · only.r.05
- It was the same story; only this time she came out better
-
6
never except when
only%4:02:03:: · only.r.06
- call me only if your cold gets worse
Synonyms: only if, only when
Gold Maru 2022
Gold Raganato
Selected
-
7
as recently as
only%4:02:05:: · only.r.07
- I spoke to him only an hour ago
However, in both studies, patients whose cancer was not affecting squamous cells had longer survival times if they received Alimta than if they received the comparator.
Show raw prompt
user
Which numbered sense of "however" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.
Sentence: <t>However</t>, in both studies, patients whose cancer was not affecting squamous cells had longer survival times if they received Alimta than if they received the comparator.
Senses:
1. definition=despite anything to the contrary (usually following a concession) | examples=although I'm a little afraid, however I'd like to try it
2. definition=by contrast; on the other hand | examples=the first part was easy; the second, however, took hours
3. definition=to whatever degree or extent | examples=The results, however general, are important
4. definition=in whatever way or manner | examples=Victory, however it was brought about, was sweet
Prediction matches gold:
lexEN ✗
Maru 2022 ✓
Raganato ✗
WordNet 3.0 senses
-
1
despite anything to the contrary (usually following a concession)
however%4:02:00:: · however.r.01
- although I'm a little afraid, however I'd like to try it
- while we disliked each other, nevertheless we agreed
Synonyms: nevertheless, withal, still, yet, all the same, even so, nonetheless, notwithstanding
Gold Maru 2022
Selected
-
2
by contrast; on the other hand
however%4:02:04:: · however.r.02
- the first part was easy; the second, however, took hours
Gold lexEN
Gold Raganato
-
3
to whatever degree or extent
however%4:02:02:: · however.r.03
- The results, however general, are important
- they have begun, however reluctantly, to acknowledge the legitimacy of some of the opposition's concerns
-
4
in whatever way or manner
however%4:02:01:: · however.r.04
- Victory, however it was brought about, was sweet
- however he did it, it was very clever
Here's a screenshot of kalgebra main window
Show raw prompt
user
Which numbered sense of "here" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.
Sentence: <t>Here</t>'s a screenshot of kalgebra main window
Senses:
1. definition=in or at this place; where the speaker or writer is | examples=I work here
2. definition=in this circumstance or respect or on this point or detail | examples=what do we have here?
3. definition=to this place (especially toward the speaker) | examples=come here, please
4. definition=at this time; now | examples=we'll adjourn here for lunch and discuss the remaining issues this afternoon
Prediction matches gold:
lexEN ✗
Maru 2022 ✗
Raganato ✗
WordNet 3.0 senses
-
1
in or at this place; where the speaker or writer is
here%4:02:00:: · here.r.01
Gold lexEN
Gold Maru 2022
Gold Raganato
-
2
in this circumstance or respect or on this point or detail
here%4:02:02:: · here.r.02
- what do we have here?
- here I must disagree
Selected
-
3
to this place (especially toward the speaker)
here%4:02:01:: · here.r.03
Synonyms: hither
-
4
at this time; now
here%4:02:04:: · here.r.04
- we'll adjourn here for lunch and discuss the remaining issues this afternoon
Also, the treatment of vomiting should only be together with other supportive measure or other veterinary therapy while addressing the underlying causes of the vomiting.
Show raw prompt
user
Which numbered sense of "only" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.
Sentence: Also, the treatment of vomiting should <t>only</t> be together with other supportive measure or other veterinary therapy while addressing the underlying causes of the vomiting.
Senses:
1. definition=and nothing more | examples=I was merely asking
2. definition=without any others being included or involved | examples=was entirely to blame
3. definition=with nevertheless the final result | examples=He arrived only to find his wife dead
4. definition=in the final outcome | examples=These news will only make you more upset
5. definition=except that | examples=It was the same story; only this time she came out better
6. definition=never except when | examples=call me only if your cold gets worse
7. definition=as recently as | examples=I spoke to him only an hour ago
Prediction matches gold:
lexEN ✗
Maru 2022 ✗
Raganato ✗
WordNet 3.0 senses
-
1
and nothing more
only%4:02:02:: · merely.r.01
- I was merely asking
- it is simply a matter of time
Synonyms: merely, simply, just, but
-
2
without any others being included or involved
only%4:02:01:: · entirely.r.02
- was entirely to blame
- a school devoted entirely to the needs of problem children
Synonyms: entirely, exclusively, solely, alone
Gold lexEN
Gold Maru 2022
Gold Raganato
-
3
with nevertheless the final result
only%4:02:00:: · only.r.03
- He arrived only to find his wife dead
- We won only to lose again in the next round
-
4
in the final outcome
only%4:02:04:: · only.r.04
- These news will only make you more upset
-
5
except that
only%4:02:06:: · only.r.05
- It was the same story; only this time she came out better
-
6
never except when
only%4:02:03:: · only.r.06
- call me only if your cold gets worse
Synonyms: only if, only when
Selected
-
7
as recently as
only%4:02:05:: · only.r.07
- I spoke to him only an hour ago
Cerenia was generally well tolerated when administered daily at dosages up to 5 times the recommended doses for an extended period of time (up to 3 times the recommended maximum duration of treatment).
Show raw prompt
user
Which numbered sense of "well" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.
Sentence: Cerenia was generally <t>well</t> tolerated when administered daily at dosages up to 5 times the recommended doses for an extended period of time (up to 3 times the recommended maximum duration of treatment).
Senses:
1. definition=(often used as a combining form) in a good or proper or satisfactory manner or to a high standard (`good' is a nonstandard dialectal variant for `well') | examples=the children behaved well
2. definition=thoroughly or completely; fully; often used as a combining form; ; ; ; ; , | examples=The problem is well understood
3. definition=indicating high probability; in all likelihood | examples=I might well do it
4. definition=(used for emphasis or as an intensifier) entirely or fully | examples=a book well worth reading
5. definition=to a suitable or appropriate extent or degree | examples=the project was well underway
6. definition=favorably; with approval | examples=their neighbors spoke well of them
7. definition=to a great extent or degree | examples=I'm afraid the film was well over budget
8. definition=with great or especially intimate knowledge | examples=we knew them well
9. definition=with prudence or propriety | examples=You would do well to say nothing more
10. definition=with skill or in a pleasing manner | examples=she dances well
11. definition=in a manner affording benefit or advantage | examples=she married well
12. definition=in financial comfort | examples=They live well
13. definition=without unusual distress or resentment; with good humor | examples=took the joke well
Prediction matches gold:
lexEN ✗
Maru 2022 ✗
Raganato ✗
WordNet 3.0 senses
-
1
(often used as a combining form) in a good or proper or satisfactory manner or to a high standard (`good' is a nonstandard dialectal variant for `well')
well%4:02:00:: · well.r.01
- the children behaved well
- a task well done
Synonyms: good
Gold lexEN
Gold Maru 2022
Gold Raganato
-
2
thoroughly or completely; fully; often used as a combining form; ; ; ; ; ,
well%4:02:05:: · well.r.02
- The problem is well understood
- she was well informed
-
3
indicating high probability; in all likelihood
well%4:02:03:: · well.r.03
- I might well do it
- a mistake that could easily have ended in disaster
Synonyms: easily
-
4
(used for emphasis or as an intensifier) entirely or fully
well%4:02:15:: · well.r.04
- a book well worth reading
- was well aware of the difficulties ahead
-
5
to a suitable or appropriate extent or degree
well%4:02:08:: · well.r.05
- the project was well underway
- the fetus has well developed organs
-
6
favorably; with approval
well%4:02:07:: · well.r.06
- their neighbors spoke well of them
- he thought well of the book
-
7
to a great extent or degree
well%4:02:11:: · well.r.07
- I'm afraid the film was well over budget
- painting the room white made it seem considerably (or substantially) larger
Synonyms: considerably, substantially
-
8
with great or especially intimate knowledge
well%4:02:14:: · well.r.08
Synonyms: intimately
-
9
with prudence or propriety
well%4:02:13:: · well.r.09
- You would do well to say nothing more
- could not well refuse
-
10
with skill or in a pleasing manner
well%4:02:12:: · well.r.10
- she dances well
- he writes well
-
11
in a manner affording benefit or advantage
well%4:02:10:: · well.r.11
- she married well
- The children were settled advantageously in Seattle
Synonyms: advantageously
-
12
in financial comfort
well%4:02:09:: · well.r.12
- They live well
- she has been able to live comfortably since her husband died
Synonyms: comfortably
-
13
without unusual distress or resentment; with good humor
well%4:02:01:: · well.r.13
- took the joke well
- took the tragic news well
Selected
Now, only one local ringer remains: 64-year-old Derek Hammond.
Show raw prompt
user
Which numbered sense of "only" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.
Sentence: Now, <t>only</t> one local ringer remains: 64-year-old Derek Hammond.
Senses:
1. definition=and nothing more | examples=I was merely asking
2. definition=without any others being included or involved | examples=was entirely to blame
3. definition=with nevertheless the final result | examples=He arrived only to find his wife dead
4. definition=in the final outcome | examples=These news will only make you more upset
5. definition=except that | examples=It was the same story; only this time she came out better
6. definition=never except when | examples=call me only if your cold gets worse
7. definition=as recently as | examples=I spoke to him only an hour ago
Prediction matches gold:
lexEN ✗
Maru 2022 ✗
Raganato ✗
WordNet 3.0 senses
-
1
and nothing more
only%4:02:02:: · merely.r.01
- I was merely asking
- it is simply a matter of time
Synonyms: merely, simply, just, but
Gold lexEN
Gold Maru 2022
Gold Raganato
-
2
without any others being included or involved
only%4:02:01:: · entirely.r.02
- was entirely to blame
- a school devoted entirely to the needs of problem children
Synonyms: entirely, exclusively, solely, alone
Selected
-
3
with nevertheless the final result
only%4:02:00:: · only.r.03
- He arrived only to find his wife dead
- We won only to lose again in the next round
-
4
in the final outcome
only%4:02:04:: · only.r.04
- These news will only make you more upset
-
5
except that
only%4:02:06:: · only.r.05
- It was the same story; only this time she came out better
-
6
never except when
only%4:02:03:: · only.r.06
- call me only if your cold gets worse
Synonyms: only if, only when
-
7
as recently as
only%4:02:05:: · only.r.07
- I spoke to him only an hour ago
We've tried to train the youngsters, but they have their discos and their dances, and they just drift away. "
Show raw prompt
user
Which numbered sense of "just" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.
Sentence: We've tried to train the youngsters, but they have their discos and their dances, and they <t>just</t> drift away. "
Senses:
1. definition=and nothing more | examples=I was merely asking
2. definition=indicating exactness or preciseness | examples=he was doing precisely (or exactly) what she had told him to do
3. definition=only a moment ago | examples=he has just arrived
4. definition=absolutely | examples=I just can't take it anymore
5. definition=only a very short time before; ; ; ; ; - W.B.Yeats | examples=they could barely hear the speaker
6. definition=exactly at this moment or the moment described | examples=we've just finished painting the walls, so don't touch them
Prediction matches gold:
lexEN ✗
Maru 2022 ✗
Raganato ✗
WordNet 3.0 senses
-
1
and nothing more
just%4:02:00:: · merely.r.01
- I was merely asking
- it is simply a matter of time
Synonyms: merely, simply, only, but
Selected
-
2
indicating exactness or preciseness
just%4:02:05:: · precisely.r.01
- he was doing precisely (or exactly) what she had told him to do
- it was just as he said--the jewel was gone
Synonyms: precisely, exactly
-
3
only a moment ago
just%4:02:01:: · just.r.03
- he has just arrived
- the sun just now came out
Synonyms: just now
-
4
absolutely
just%4:02:03:: · just.r.04
- I just can't take it anymore
- he was just grand as Romeo
Synonyms: simply
Gold lexEN
Gold Maru 2022
Gold Raganato
-
5
only a very short time before; ; ; ; ; - W.B.Yeats
just%4:02:04:: · barely.r.01
- they could barely hear the speaker
- we hardly knew them
Synonyms: barely, hardly, scarcely, scarce
-
6
exactly at this moment or the moment described
just%4:02:06:: · just.r.06
- we've just finished painting the walls, so don't touch them
Each variation, or change, can occur only once, the rules state.
Show raw prompt
user
Which numbered sense of "only" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.
Sentence: Each variation, or change, can occur <t>only</t> once, the rules state.
Senses:
1. definition=and nothing more | examples=I was merely asking
2. definition=without any others being included or involved | examples=was entirely to blame
3. definition=with nevertheless the final result | examples=He arrived only to find his wife dead
4. definition=in the final outcome | examples=These news will only make you more upset
5. definition=except that | examples=It was the same story; only this time she came out better
6. definition=never except when | examples=call me only if your cold gets worse
7. definition=as recently as | examples=I spoke to him only an hour ago
Prediction matches gold:
lexEN ✗
Maru 2022 ✗
Raganato ✗
WordNet 3.0 senses
-
1
and nothing more
only%4:02:02:: · merely.r.01
- I was merely asking
- it is simply a matter of time
Synonyms: merely, simply, just, but
Gold lexEN
Gold Maru 2022
Gold Raganato
-
2
without any others being included or involved
only%4:02:01:: · entirely.r.02
- was entirely to blame
- a school devoted entirely to the needs of problem children
Synonyms: entirely, exclusively, solely, alone
Selected
-
3
with nevertheless the final result
only%4:02:00:: · only.r.03
- He arrived only to find his wife dead
- We won only to lose again in the next round
-
4
in the final outcome
only%4:02:04:: · only.r.04
- These news will only make you more upset
-
5
except that
only%4:02:06:: · only.r.05
- It was the same story; only this time she came out better
-
6
never except when
only%4:02:03:: · only.r.06
- call me only if your cold gets worse
Synonyms: only if, only when
-
7
as recently as
only%4:02:05:: · only.r.07
- I spoke to him only an hour ago
Totally absorbed, the ringers stare straight ahead, using peripheral vision (they call it "rope-sight ") to watch the other ropes and thus time their pulls.
Show raw prompt
user
Which numbered sense of "straight" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.
Sentence: Totally absorbed, the ringers stare <t>straight</t> ahead, using peripheral vision (they call it "rope-sight ") to watch the other ropes and thus time their pulls.
Senses:
1. definition=without deviation | examples=the path leads directly to the lake
2. definition=in a forthright manner; candidly or frankly | examples=he didn't answer directly
3. definition=in a straight line; in a direct course | examples=the road runs straight
Prediction matches gold:
lexEN ✗
Maru 2022 ✓
Raganato ✗
WordNet 3.0 senses
-
1
without deviation
straight%4:02:05:: · directly.r.01
- the path leads directly to the lake
- went direct to the office
Synonyms: directly, direct
Gold lexEN
Gold Raganato
-
2
in a forthright manner; candidly or frankly
straight%4:02:04:: · directly.r.04
- he didn't answer directly
- told me straight out
Synonyms: directly, flat
-
3
in a straight line; in a direct course
straight%4:02:00:: · straight.r.03
Gold Maru 2022
Selected
Totally absorbed, the ringers stare straight ahead, using peripheral vision (they call it "rope-sight ") to watch the other ropes and thus time their pulls.
Show raw prompt
user
Which numbered sense of "ahead" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.
Sentence: Totally absorbed, the ringers stare straight <t>ahead</t>, using peripheral vision (they call it "rope-sight ") to watch the other ropes and thus time their pulls.
Senses:
1. definition=at or in the front | examples=I see the lights of a town ahead
2. definition=toward the future; forward in time | examples=I like to look ahead in imagination to what the future may bring
3. definition=in a forward direction | examples=go ahead
4. definition=ahead of time; in anticipation | examples=when you pay ahead (or in advance) you receive a discount
5. definition=to a more advanced or advantageous position | examples=a young man sure to get ahead
6. definition=to a different or a more advanced time (meaning advanced either toward the present or toward the future) | examples=moved the appointment ahead from Tuesday to Monday
7. definition=leading or ahead in a competition | examples=the horse was three lengths ahead going into the home stretch
Prediction matches gold:
lexEN ✗
Maru 2022 ✗
Raganato ✗
WordNet 3.0 senses
-
1
at or in the front
ahead%4:02:00:: · ahead.r.01
- I see the lights of a town ahead
- the road ahead is foggy
Synonyms: in front, before
Gold lexEN
Gold Maru 2022
Gold Raganato
-
2
toward the future; forward in time
ahead%4:02:06:: · ahead.r.02
- I like to look ahead in imagination to what the future may bring
- I look forward to seeing you
Synonyms: forward
-
3
in a forward direction
ahead%4:02:02:: · ahead.r.03
- go ahead
- the train moved ahead slowly
Synonyms: onward, onwards, forward, forwards, forrader
Selected
-
4
ahead of time; in anticipation
ahead%4:02:01:: · ahead.r.04
- when you pay ahead (or in advance) you receive a discount
- We like to plan ahead
Synonyms: in advance, beforehand
-
5
to a more advanced or advantageous position
ahead%4:02:05:: · ahead.r.05
- a young man sure to get ahead
- pushing talented students ahead
-
6
to a different or a more advanced time (meaning advanced either toward the present or toward the future)
ahead%4:02:04:: · ahead.r.06
- moved the appointment ahead from Tuesday to Monday
- pushed the deadline ahead from Tuesday to Wednesday
-
7
leading or ahead in a competition
ahead%4:02:03:: · ahead.r.07
- the horse was three lengths ahead going into the home stretch
- ahead by two pawns
Synonyms: out front, in the lead
Totally absorbed, the ringers stare straight ahead, using peripheral vision (they call it "rope-sight ") to watch the other ropes and thus time their pulls.
Show raw prompt
user
Which numbered sense of "thus" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.
Sentence: Totally absorbed, the ringers stare straight ahead, using peripheral vision (they call it "rope-sight ") to watch the other ropes and <t>thus</t> time their pulls.
Senses:
1. definition=(used to introduce a logical conclusion) from that fact or reason or as a result | examples=therefore X must be true
2. definition=in the way indicated; ; ; (`thusly' is a nonstandard variant) | examples=hold the brush so
Prediction matches gold:
lexEN ✗
Maru 2022 ✗
Raganato ✗
WordNet 3.0 senses
-
1
(used to introduce a logical conclusion) from that fact or reason or as a result
thus%4:02:00:: · therefore.r.01
- therefore X must be true
- the eggs were fresh and hence satisfactory
Synonyms: therefore, hence, thence, so
Gold lexEN
Gold Maru 2022
Gold Raganato
-
2
in the way indicated; ; ; (`thusly' is a nonstandard variant)
thus%4:02:01:: · thus.r.02
- hold the brush so
- set up the pieces thus
Synonyms: thusly, so
Selected
More often than not, ringers think of the church as something stuck on the bottom of the belfry.
Show raw prompt
user
Which numbered sense of "more" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.
Sentence: <t>More</t> often than not, ringers think of the church as something stuck on the bottom of the belfry.
Senses:
1. definition=used to form the comparative of some adjectives and adverbs | examples=more interesting
2. definition=comparative of much; to a greater degree or extent | examples=he works more now
Prediction matches gold:
lexEN ✗
Maru 2022 ✗
Raganato ✗
WordNet 3.0 senses
-
1
used to form the comparative of some adjectives and adverbs
more%4:02:00:: · more.r.01
- more interesting
- more beautiful
Synonyms: to a greater extent
Gold lexEN
Gold Maru 2022
Gold Raganato
-
2
comparative of much; to a greater degree or extent
more%4:02:01:: · more.r.02
- he works more now
- they eat more than they should
Selected