Verified run

Qwen/Qwen3-235B-A22B-Instruct-2507

vllm-qwen3-235b-2507-awq-h200-p001-lexen-v1-20260614

Rank
96
Accuracy lexEN v1 · WordNet fine
86.59%
Correct
4209 / 4861
Actual run cost
$0.448
Cost / M items
$92.19

Accuracy across label schemes

The headline accuracy is lexEN v1 · WordNet fine-grained. Each cell re-scores this run's predictions against a different gold label set and sense granularity; coarse counts a prediction correct when its WordNet sense maps to the same Glite or CSI concept as a gold sense. About these label schemes → About coarsening →

Score against WordNet fine-grainedGlite coarse-grainedCSI coarse-grained (Lacerra 2020)
lexEN v1 86.59% 4209 / 4861 94.65% 4601 / 4861 92.02% 4473 / 4861
Maru 2022 (ALLamended) 84.59% 4112 / 4861 93.99% 4569 / 4861 90.80% 4414 / 4861
Raganato 2017 (original) 79.86% 3882 / 4861 91.67% 4456 / 4861 87.90% 4273 / 4861

Model

Display
Qwen/Qwen3-235B-A22B-Instruct-2507
Requested
hosted_vllm/Qwen/Qwen3-235B-A22B-Instruct-2507
Resolved
Qwen/Qwen3-235B-A22B-Instruct-2507
Vendor
Alibaba
API provider
n/a
Hosting
self_hosted
Source
open_source
License
apache-2.0
Reasoning
n/a
Model URL
https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
Endpoint
http://localhost:8000/v1
Resolved counts
Qwen/Qwen3-235B-A22B-Instruct-2507: 4861
HF revision
96548f657d772187f16e9f04bc50aec78a9f9cdd
Quantization
awq-int4
Inference engine
vllm
Engine version
0.22.1
Container image
vllm/vllm-openai:v0.22.1
Serve command
exec vllm serve QuantTrio/Qwen3-235B-A22B-Instruct-2507-AWQ --port 8000 --served-model-name Qwen/Qwen3-235B-A22B-Instruct-2507 --max-model-len 8192 --gpu-memory-utilization 0.90 --max-num-seqs 64 --default-chat-template-kwargs \{\"enable_thinking\":\ false\} --revision 96548f657d772187f16e9f04bc50aec78a9f9cdd

Reproducibility

Created
2026-06-14T20:01:55.579872+00:00
Run schema
sensebench-run-v2
Commit
n/a
Dataset ID
lexen
Dataset version
lexen-v1
Dataset revision
n/a
Dataset items
4,861
Dataset hash
sha256:5fd4382b93f19087a1e31f6dd7d1db17c1eb17ff80fcbe1d3fdd55c0c3ecefe8
Prompt
p001
Prompt version
0.1.0
Repository files
Browse in GitHub

Runtime

Calls
4861
Total tokens
3,126,228
Tokens / item
643.1
Total cost
$0.448
Cost / M items
$92.19
Cost source
machine_time_estimate
Elapsed
7.03m
Success items
4,861
Monosemous
0
No candidates
0
No valid vote
0
Invalid votes
0
Transport errors
0

Machine & Timing

GPU
NVIDIA H200
VRAM / GPU
143,771 MiB
Driver
595.71.05
CUDA
13.2
CPU
Intel(R) Xeon(R) Platinum 8480+ (224 cores)
RAM
2,015.3 GiB
Platform
Linux-6.8.0-107-generic-x86_64-with-glibc2.35
Provider
vast.ai (40919270)
Hourly rate
$3.82/h
Benchmark time
7.03m
Machine-hours / 1M items
24.1 h/M
Concurrency
64

Benchmark time covers only the per-item evaluation loop on this machine, excluding model and asset loading.

Runner

GitHub
vassiliphilippov
Name
Vassili Philippov
Contact
n/a

Sampling

Temperature
0.0000
Top P
n/a
Max tokens
2,048
Seed
n/a
Extra
n/a

Policy

Votes / item
1
Reasks / invalid vote
1
Tie break
earliest_vote
Monosemous
short_circuit

Cost And Token Breakdown

Actual recorded run cost: $0.448. Cost estimate source: machine_time_estimate. Cost is estimated from machine time multiplied by the recorded hourly rate.

Token Type Tokens Cost Price / 1M tokens
Input uncached 3,087,242 n/a n/a
Input cached n/a n/a n/a
Output 38,986 n/a n/a
Reasoning output n/a included in output cost n/a

Reasoning output tokens are shown separately when the provider reports them; recorded cost is stored in the output cost bucket.

Download Raw Run Files

Slice Breakdown

Group Bucket Accuracy Correct Items
Candidate Count 11+ 78.59% 624 794
Candidate Count 2 93.22% 797 855
Candidate Count 3-5 88.12% 1714 1945
Candidate Count 6-10 84.77% 1074 1267
POS ADJ 89.53% 590 659
POS ADV 77.95% 152 195
POS NOUN 89.13% 2582 2897
POS VERB 79.73% 885 1110
Source Dataset semeval2013 91.67% 1045 1140
Source Dataset semeval2015 89.72% 663 739
Source Dataset senseval2 85.08% 1386 1629
Source Dataset senseval3 82.41% 1115 1353

Worst-Bucket Examples

easily Adv

Instance: semeval2015.d000.s009.t012 Dataset: semeval2015 Bucket: POS / ADV

Incorrect

What is Alimta?

Alimta is a powder that is made up into a solution for infusion (drip into a vein).

It contains the active substance pemetrexed.

What is Alimta used for?

Alimta is used to treat two types of cancer: pleural mesothelioma (cancer of the lining of the lungs that is usually caused by exposure to asbestos).

Alimta is used together with cisplatin (another anticancer medicine) when the cancer is unresectable (cannot be removed by surgery alone) and malignant (has spread, or is likely to spread easily, to other parts of the body), in patients who have not received chemotherapy (medicines for cancer) before advanced or metastatic non-small cell lung cancer that is not affecting the squamous cells.

Alimta is used in combination with cisplatin in patients who have not been treated before, and on its own in patients who have received another type of chemotherapy in the past.

Show raw prompt

system

You are an English linguist performing fine-grained Word Sense Disambiguation. WordNet glosses are schematic. Use the sentence context, the examples, and the synonyms to choose the best indexed sense. Do not hedge. Choose the single best indexed sense.

user

Target lemma: easily
Target surface form: easily
Context:
What is Alimta? Alimta is a powder that is made up into a solution for infusion (drip into a vein). It contains the active substance pemetrexed. What is Alimta used for? Alimta is used to treat two types of cancer: pleural mesothelioma (cancer of the lining of the lungs that is usually caused by exposure to asbestos). Alimta is used together with cisplatin (another anticancer medicine) when the cancer is unresectable (cannot be removed by surgery alone) and malignant (has spread, or is likely to spread <t>easily</t>, to other parts of the body), in patients who have not received chemotherapy (medicines for cancer) before advanced or metastatic non-small cell lung cancer that is not affecting the squamous cells. Alimta is used in combination with cisplatin in patients who have not been treated before, and on its own in patients who have received another type of chemotherapy in the past.

Candidate senses:
1. sense_key=easily%4:02:01:: | definition=with ease (`easy' is sometimes used informally for `easily') | synonyms=easy | examples=she was easily excited; was easily confused
2. sense_key=easily%4:02:06:: | definition=without question | examples=easily the best book she's written
3. sense_key=easily%4:02:05:: | definition=indicating high probability; in all likelihood | synonyms=well | examples=I might well do it; a mistake that could easily have ended in disaster

Instructions:
1. Choose the single best indexed sense for the target word in context.
2. WordNet glosses are schematic; use the context, synonyms, and examples together.
3. Do not answer with a WordNet sense key.
4. Return only a JSON object exactly like {"sense_index": 3}.

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    with ease (`easy' is sometimes used informally for `easily')

    easily%4:02:01:: · easily.r.01

    • she was easily excited
    • was easily confused

    Synonyms: easy

    Gold lexEN Gold Maru 2022 Gold Raganato
  2. 2

    without question

    easily%4:02:06:: · easily.r.02

    • easily the best book she's written
  3. 3

    indicating high probability; in all likelihood

    easily%4:02:05:: · well.r.03

    • I might well do it
    • a mistake that could easily have ended in disaster

    Synonyms: well

    Selected

only Adv

Instance: semeval2015.d000.s011.t001 Dataset: semeval2015 Bucket: POS / ADV

Incorrect

It contains the active substance pemetrexed.

What is Alimta used for?

Alimta is used to treat two types of cancer: pleural mesothelioma (cancer of the lining of the lungs that is usually caused by exposure to asbestos).

Alimta is used together with cisplatin (another anticancer medicine) when the cancer is unresectable (cannot be removed by surgery alone) and malignant (has spread, or is likely to spread easily, to other parts of the body), in patients who have not received chemotherapy (medicines for cancer) before advanced or metastatic non-small cell lung cancer that is not affecting the squamous cells.

Alimta is used in combination with cisplatin in patients who have not been treated before, and on its own in patients who have received another type of chemotherapy in the past.

The medicine can only be obtained with a prescription.

How is Alimta used?

Show raw prompt

system

You are an English linguist performing fine-grained Word Sense Disambiguation. WordNet glosses are schematic. Use the sentence context, the examples, and the synonyms to choose the best indexed sense. Do not hedge. Choose the single best indexed sense.

user

Target lemma: only
Target surface form: only
Context:
It contains the active substance pemetrexed. What is Alimta used for? Alimta is used to treat two types of cancer: pleural mesothelioma (cancer of the lining of the lungs that is usually caused by exposure to asbestos). Alimta is used together with cisplatin (another anticancer medicine) when the cancer is unresectable (cannot be removed by surgery alone) and malignant (has spread, or is likely to spread easily, to other parts of the body), in patients who have not received chemotherapy (medicines for cancer) before advanced or metastatic non-small cell lung cancer that is not affecting the squamous cells. Alimta is used in combination with cisplatin in patients who have not been treated before, and on its own in patients who have received another type of chemotherapy in the past. The medicine can <t>only</t> be obtained with a prescription. How is Alimta used?

Candidate senses:
1. sense_key=only%4:02:02:: | definition=and nothing more | synonyms=merely, simply, just, but | examples=I was merely asking; it is simply a matter of time
2. sense_key=only%4:02:01:: | definition=without any others being included or involved | synonyms=entirely, exclusively, solely, alone | examples=was entirely to blame; a school devoted entirely to the needs of problem children
3. sense_key=only%4:02:00:: | definition=with nevertheless the final result | examples=He arrived only to find his wife dead; We won only to lose again in the next round
4. sense_key=only%4:02:04:: | definition=in the final outcome | examples=These news will only make you more upset
5. sense_key=only%4:02:06:: | definition=except that | examples=It was the same story; only this time she came out better
6. sense_key=only%4:02:03:: | definition=never except when | synonyms=only if, only when | examples=call me only if your cold gets worse
7. sense_key=only%4:02:05:: | definition=as recently as | examples=I spoke to him only an hour ago

Instructions:
1. Choose the single best indexed sense for the target word in context.
2. WordNet glosses are schematic; use the context, synonyms, and examples together.
3. Do not answer with a WordNet sense key.
4. Return only a JSON object exactly like {"sense_index": 3}.

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    and nothing more

    only%4:02:02:: · merely.r.01

    • I was merely asking
    • it is simply a matter of time

    Synonyms: merely, simply, just, but

    Selected
  2. 2

    without any others being included or involved

    only%4:02:01:: · entirely.r.02

    • was entirely to blame
    • a school devoted entirely to the needs of problem children

    Synonyms: entirely, exclusively, solely, alone

    Gold lexEN
  3. 3

    with nevertheless the final result

    only%4:02:00:: · only.r.03

    • He arrived only to find his wife dead
    • We won only to lose again in the next round
  4. 4

    in the final outcome

    only%4:02:04:: · only.r.04

    • These news will only make you more upset
  5. 5

    except that

    only%4:02:06:: · only.r.05

    • It was the same story; only this time she came out better
  6. 6

    never except when

    only%4:02:03:: · only.r.06

    • call me only if your cold gets worse

    Synonyms: only if, only when

    Gold Maru 2022 Gold Raganato
  7. 7

    as recently as

    only%4:02:05:: · only.r.07

    • I spoke to him only an hour ago

only Adv

Instance: semeval2015.d000.s013.t000 Dataset: semeval2015 Bucket: POS / ADV

Incorrect

Alimta is used to treat two types of cancer: pleural mesothelioma (cancer of the lining of the lungs that is usually caused by exposure to asbestos).

Alimta is used together with cisplatin (another anticancer medicine) when the cancer is unresectable (cannot be removed by surgery alone) and malignant (has spread, or is likely to spread easily, to other parts of the body), in patients who have not received chemotherapy (medicines for cancer) before advanced or metastatic non-small cell lung cancer that is not affecting the squamous cells.

Alimta is used in combination with cisplatin in patients who have not been treated before, and on its own in patients who have received another type of chemotherapy in the past.

The medicine can only be obtained with a prescription.

How is Alimta used?

Alimta should only be given under the supervision of a doctor who is qualified in the use of anticancer chemotherapy.

The recommended dose of Alimta is 500 mg per square metre of body surface area (calculated using the patient s height and weight).

Show raw prompt

system

You are an English linguist performing fine-grained Word Sense Disambiguation. WordNet glosses are schematic. Use the sentence context, the examples, and the synonyms to choose the best indexed sense. Do not hedge. Choose the single best indexed sense.

user

Target lemma: only
Target surface form: only
Context:
Alimta is used to treat two types of cancer: pleural mesothelioma (cancer of the lining of the lungs that is usually caused by exposure to asbestos). Alimta is used together with cisplatin (another anticancer medicine) when the cancer is unresectable (cannot be removed by surgery alone) and malignant (has spread, or is likely to spread easily, to other parts of the body), in patients who have not received chemotherapy (medicines for cancer) before advanced or metastatic non-small cell lung cancer that is not affecting the squamous cells. Alimta is used in combination with cisplatin in patients who have not been treated before, and on its own in patients who have received another type of chemotherapy in the past. The medicine can only be obtained with a prescription. How is Alimta used? Alimta should <t>only</t> be given under the supervision of a doctor who is qualified in the use of anticancer chemotherapy. The recommended dose of Alimta is 500 mg per square metre of body surface area (calculated using the patient s height and weight).

Candidate senses:
1. sense_key=only%4:02:02:: | definition=and nothing more | synonyms=merely, simply, just, but | examples=I was merely asking; it is simply a matter of time
2. sense_key=only%4:02:01:: | definition=without any others being included or involved | synonyms=entirely, exclusively, solely, alone | examples=was entirely to blame; a school devoted entirely to the needs of problem children
3. sense_key=only%4:02:00:: | definition=with nevertheless the final result | examples=He arrived only to find his wife dead; We won only to lose again in the next round
4. sense_key=only%4:02:04:: | definition=in the final outcome | examples=These news will only make you more upset
5. sense_key=only%4:02:06:: | definition=except that | examples=It was the same story; only this time she came out better
6. sense_key=only%4:02:03:: | definition=never except when | synonyms=only if, only when | examples=call me only if your cold gets worse
7. sense_key=only%4:02:05:: | definition=as recently as | examples=I spoke to him only an hour ago

Instructions:
1. Choose the single best indexed sense for the target word in context.
2. WordNet glosses are schematic; use the context, synonyms, and examples together.
3. Do not answer with a WordNet sense key.
4. Return only a JSON object exactly like {"sense_index": 3}.

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    and nothing more

    only%4:02:02:: · merely.r.01

    • I was merely asking
    • it is simply a matter of time

    Synonyms: merely, simply, just, but

  2. 2

    without any others being included or involved

    only%4:02:01:: · entirely.r.02

    • was entirely to blame
    • a school devoted entirely to the needs of problem children

    Synonyms: entirely, exclusively, solely, alone

    Selected
  3. 3

    with nevertheless the final result

    only%4:02:00:: · only.r.03

    • He arrived only to find his wife dead
    • We won only to lose again in the next round
  4. 4

    in the final outcome

    only%4:02:04:: · only.r.04

    • These news will only make you more upset
  5. 5

    except that

    only%4:02:06:: · only.r.05

    • It was the same story; only this time she came out better
  6. 6

    never except when

    only%4:02:03:: · only.r.06

    • call me only if your cold gets worse

    Synonyms: only if, only when

    Gold lexEN Gold Maru 2022 Gold Raganato
  7. 7

    as recently as

    only%4:02:05:: · only.r.07

    • I spoke to him only an hour ago

however Adv

Instance: semeval2015.d000.s036.t000 Dataset: semeval2015 Bucket: POS / ADV

Incorrect

Alimta increased the survival time of patients with malignant pleural mesothelioma.

Patients receiving Alimta and cisplatin survived for an average of 12.1 months, compared with 9.3 months in those receiving cisplatin alone.

In the treatment of patients with non-small cell lung cancer, Alimta was as effective as the comparators.

In patients who had received chemotherapy in the past, the average survival time was 8.3 months with Alimta, compared with 7.9 months with docetaxel.

In patients who had not received chemotherapy in the past, the average survival time was 10.3 months in both groups.

However, in both studies, patients whose cancer was not affecting squamous cells had longer survival times if they received Alimta than if they received the comparator.

In contrast, patients whose cancer was affecting squamous cells had shorter survival times if they received Alimta.

Show raw prompt

system

You are an English linguist performing fine-grained Word Sense Disambiguation. WordNet glosses are schematic. Use the sentence context, the examples, and the synonyms to choose the best indexed sense. Do not hedge. Choose the single best indexed sense.

user

Target lemma: however
Target surface form: However
Context:
Alimta increased the survival time of patients with malignant pleural mesothelioma. Patients receiving Alimta and cisplatin survived for an average of 12.1 months, compared with 9.3 months in those receiving cisplatin alone. In the treatment of patients with non-small cell lung cancer, Alimta was as effective as the comparators. In patients who had received chemotherapy in the past, the average survival time was 8.3 months with Alimta, compared with 7.9 months with docetaxel. In patients who had not received chemotherapy in the past, the average survival time was 10.3 months in both groups. <t>However</t>, in both studies, patients whose cancer was not affecting squamous cells had longer survival times if they received Alimta than if they received the comparator. In contrast, patients whose cancer was affecting squamous cells had shorter survival times if they received Alimta.

Candidate senses:
1. sense_key=however%4:02:00:: | definition=despite anything to the contrary (usually following a concession) | synonyms=nevertheless, withal, still, yet, all the same, even so | examples=although I'm a little afraid, however I'd like to try it; while we disliked each other, nevertheless we agreed
2. sense_key=however%4:02:04:: | definition=by contrast; on the other hand | examples=the first part was easy; the second, however, took hours
3. sense_key=however%4:02:02:: | definition=to whatever degree or extent | examples=The results, however general, are important; they have begun, however reluctantly, to acknowledge the legitimacy of some of the opposition's concerns
4. sense_key=however%4:02:01:: | definition=in whatever way or manner | examples=Victory, however it was brought about, was sweet; however he did it, it was very clever

Instructions:
1. Choose the single best indexed sense for the target word in context.
2. WordNet glosses are schematic; use the context, synonyms, and examples together.
3. Do not answer with a WordNet sense key.
4. Return only a JSON object exactly like {"sense_index": 3}.

Prediction matches gold: lexEN ✗ Maru 2022 ✓ Raganato ✗

WordNet 3.0 senses

  1. 1

    despite anything to the contrary (usually following a concession)

    however%4:02:00:: · however.r.01

    • although I'm a little afraid, however I'd like to try it
    • while we disliked each other, nevertheless we agreed

    Synonyms: nevertheless, withal, still, yet, all the same, even so, nonetheless, notwithstanding

    Gold Maru 2022 Selected
  2. 2

    by contrast; on the other hand

    however%4:02:04:: · however.r.02

    • the first part was easy; the second, however, took hours
    Gold lexEN Gold Raganato
  3. 3

    to whatever degree or extent

    however%4:02:02:: · however.r.03

    • The results, however general, are important
    • they have begun, however reluctantly, to acknowledge the legitimacy of some of the opposition's concerns
  4. 4

    in whatever way or manner

    however%4:02:01:: · however.r.04

    • Victory, however it was brought about, was sweet
    • however he did it, it was very clever

actually Adv

Instance: semeval2015.d001.s002.t000 Dataset: semeval2015 Bucket: POS / ADV

Incorrect

kalgebra is a mathematical calculator based on content markup MathML language.

Nowadays it is capable to make simple MathML operations (arithmetic and logical) and to representate 2D and 3D graphs.

It is actually not necessary to know MathML to use kalgebra.

kalgebra is a MathML based graph calculator.

Show raw prompt

system

You are an English linguist performing fine-grained Word Sense Disambiguation. WordNet glosses are schematic. Use the sentence context, the examples, and the synonyms to choose the best indexed sense. Do not hedge. Choose the single best indexed sense.

user

Target lemma: actually
Target surface form: actually
Context:
kalgebra is a mathematical calculator based on content markup MathML language. Nowadays it is capable to make simple MathML operations (arithmetic and logical) and to representate 2D and 3D graphs. It is <t>actually</t> not necessary to know MathML to use kalgebra. kalgebra is a MathML based graph calculator.

Candidate senses:
1. sense_key=actually%4:02:01:: | definition=in actual fact | synonyms=really | examples=to be nominally but not actually independent; no one actually saw the shark
2. sense_key=actually%4:02:00:: | definition=used to imply that one would expect the fact to be the opposite of that stated; surprisingly | synonyms=in reality | examples=you may actually be doing the right thing by walking out; she actually spoke Latin
3. sense_key=actually%4:02:03:: | definition=at the present moment | examples=the transmission screen shows the picture that is actually on the air
4. sense_key=actually%4:02:02:: | definition=as a sentence modifier to add slight emphasis | examples=actually, we all help clear up after a meal; actually, I haven't seen the film

Instructions:
1. Choose the single best indexed sense for the target word in context.
2. WordNet glosses are schematic; use the context, synonyms, and examples together.
3. Do not answer with a WordNet sense key.
4. Return only a JSON object exactly like {"sense_index": 3}.

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    in actual fact

    actually%4:02:01:: · actually.r.01

    • to be nominally but not actually independent
    • no one actually saw the shark

    Synonyms: really

    Selected
  2. 2

    used to imply that one would expect the fact to be the opposite of that stated; surprisingly

    actually%4:02:00:: · actually.r.02

    • you may actually be doing the right thing by walking out
    • she actually spoke Latin

    Synonyms: in reality

    Gold lexEN Gold Maru 2022 Gold Raganato
  3. 3

    at the present moment

    actually%4:02:03:: · actually.r.03

    • the transmission screen shows the picture that is actually on the air
  4. 4

    as a sentence modifier to add slight emphasis

    actually%4:02:02:: · actually.r.04

    • actually, we all help clear up after a meal
    • actually, I haven't seen the film
    Gold lexEN Gold Maru 2022 Gold Raganato

too Adv

Instance: semeval2015.d001.s015.t004 Dataset: semeval2015 Bucket: POS / ADV

Incorrect

I have based the syntax basically on common sense, it should not be hard for anyone.

I ve looked closely to maxima and maple s syntax so they look very similar.

+ - * /: Addition, subtraction, multiplication and division.

^, **: Power, you can use them both.

Also it is possible to use the unicode ² characters.

Powers are one way to make roots too.

-gt: It s the way to specify the bounded variable in a function.

Show raw prompt

system

You are an English linguist performing fine-grained Word Sense Disambiguation. WordNet glosses are schematic. Use the sentence context, the examples, and the synonyms to choose the best indexed sense. Do not hedge. Choose the single best indexed sense.

user

Target lemma: too
Target surface form: too
Context:
I have based the syntax basically on common sense, it should not be hard for anyone. I ve looked closely to maxima and maple s syntax so they look very similar. + - * /: Addition, subtraction, multiplication and division. ^, **: Power, you can use them both. Also it is possible to use the unicode ² characters. Powers are one way to make roots <t>too</t>. -gt: It s the way to specify the bounded variable in a function.

Candidate senses:
1. sense_key=too%4:02:00:: | definition=to a degree exceeding normal or proper limits | synonyms=excessively, overly, to a fault | examples=too big
2. sense_key=too%4:02:01:: | definition=in addition | synonyms=besides, also, likewise, as well | examples=he has a Mercedes, too

Instructions:
1. Choose the single best indexed sense for the target word in context.
2. WordNet glosses are schematic; use the context, synonyms, and examples together.
3. Do not answer with a WordNet sense key.
4. Return only a JSON object exactly like {"sense_index": 3}.

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    to a degree exceeding normal or proper limits

    too%4:02:00:: · excessively.r.01

    • too big

    Synonyms: excessively, overly, to a fault

    Selected
  2. 2

    in addition

    too%4:02:01:: · besides.r.02

    • he has a Mercedes, too

    Synonyms: besides, also, likewise, as well

    Gold lexEN Gold Maru 2022 Gold Raganato

more Adv

Instance: semeval2015.d002.s002.t013 Dataset: semeval2015 Bucket: POS / ADV

Incorrect

The Foundation has recently been involved in organising a conference which dealt with this critical challenge.

Over 100 participants from the main interest groups attended and debated issues such as the role of key actors including public authorities, social partners, voluntary and community organisations as well as service users.

There were relatively few cases reported of attempts to involve users in service planning but their involvement in service provision was found to be more common.

Note was also taken of the variety of different roles played by public welfare

Show raw prompt

system

You are an English linguist performing fine-grained Word Sense Disambiguation. WordNet glosses are schematic. Use the sentence context, the examples, and the synonyms to choose the best indexed sense. Do not hedge. Choose the single best indexed sense.

user

Target lemma: more
Target surface form: more
Context:
The Foundation has recently been involved in organising a conference which dealt with this critical challenge. Over 100 participants from the main interest groups attended and debated issues such as the role of key actors including public authorities, social partners, voluntary and community organisations as well as service users. There were relatively few cases reported of attempts to involve users in service planning but their involvement in service provision was found to be <t>more</t> common. Note was also taken of the variety of different roles played by public welfare

Candidate senses:
1. sense_key=more%4:02:00:: | definition=used to form the comparative of some adjectives and adverbs | synonyms=to a greater extent | examples=more interesting; more beautiful
2. sense_key=more%4:02:01:: | definition=comparative of much; to a greater degree or extent | examples=he works more now; they eat more than they should

Instructions:
1. Choose the single best indexed sense for the target word in context.
2. WordNet glosses are schematic; use the context, synonyms, and examples together.
3. Do not answer with a WordNet sense key.
4. Return only a JSON object exactly like {"sense_index": 3}.

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    used to form the comparative of some adjectives and adverbs

    more%4:02:00:: · more.r.01

    • more interesting
    • more beautiful

    Synonyms: to a greater extent

    Gold lexEN Gold Maru 2022 Gold Raganato
  2. 2

    comparative of much; to a greater degree or extent

    more%4:02:01:: · more.r.02

    • he works more now
    • they eat more than they should
    Selected

more Adv

Instance: semeval2015.d003.s016.t003 Dataset: semeval2015 Bucket: POS / ADV

Incorrect

How does Cerenia work?

Cerenia blocks a neurokinin 1 (NK1) receptor, which acts in the central nervous system.

How has Cerenia been studied?

A large number of studies with Cerenia were carried out either in laboratory dogs or in animal patients at veterinary practises in several European countries and in the USA.

What benefit has Cerenia shown during the studies?

The results of the studies showed that Cerenia was more effective than the placebo: less vomiting was seen in dogs that received the medicine than in dogs that received a placebo, both in the treatment or in the prevention of vomiting.

The treatment of a vomiting dog should be started with the injection, since vomiting animals might not be able to keep a tablet down (as they might throw the intact tablet up again).

Show raw prompt

system

You are an English linguist performing fine-grained Word Sense Disambiguation. WordNet glosses are schematic. Use the sentence context, the examples, and the synonyms to choose the best indexed sense. Do not hedge. Choose the single best indexed sense.

user

Target lemma: more
Target surface form: more
Context:
How does Cerenia work? Cerenia blocks a neurokinin 1 (NK1) receptor, which acts in the central nervous system. How has Cerenia been studied? A large number of studies with Cerenia were carried out either in laboratory dogs or in animal patients at veterinary practises in several European countries and in the USA. What benefit has Cerenia shown during the studies? The results of the studies showed that Cerenia was <t>more</t> effective than the placebo: less vomiting was seen in dogs that received the medicine than in dogs that received a placebo, both in the treatment or in the prevention of vomiting. The treatment of a vomiting dog should be started with the injection, since vomiting animals might not be able to keep a tablet down (as they might throw the intact tablet up again).

Candidate senses:
1. sense_key=more%4:02:00:: | definition=used to form the comparative of some adjectives and adverbs | synonyms=to a greater extent | examples=more interesting; more beautiful
2. sense_key=more%4:02:01:: | definition=comparative of much; to a greater degree or extent | examples=he works more now; they eat more than they should

Instructions:
1. Choose the single best indexed sense for the target word in context.
2. WordNet glosses are schematic; use the context, synonyms, and examples together.
3. Do not answer with a WordNet sense key.
4. Return only a JSON object exactly like {"sense_index": 3}.

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    used to form the comparative of some adjectives and adverbs

    more%4:02:00:: · more.r.01

    • more interesting
    • more beautiful

    Synonyms: to a greater extent

    Gold lexEN Gold Maru 2022 Gold Raganato
  2. 2

    comparative of much; to a greater degree or extent

    more%4:02:01:: · more.r.02

    • he works more now
    • they eat more than they should
    Selected

always Adv

Instance: senseval2.d000.s004.t007 Dataset: senseval2 Bucket: POS / ADV

Incorrect

The art of change-ringing is peculiar to the English, and, like most English peculiarities, unintelligible to the rest of the world.

Dorothy L. Sayers, "The Nine Tailors "

ASLACTON, England --

Of all scenes that evoke rural England, this is one of the loveliest: An ancient stone church stands amid the fields, the sound of bells cascading from its tower, calling the faithful to evensong.

The parishioners of St. Michael and All Angels stop to chat at the church door, as members here always have.

In the tower, five men and women pull rhythmically on ropes attached to the same five bells that first sounded here in 1614.

Show raw prompt

system

You are an English linguist performing fine-grained Word Sense Disambiguation. WordNet glosses are schematic. Use the sentence context, the examples, and the synonyms to choose the best indexed sense. Do not hedge. Choose the single best indexed sense.

user

Target lemma: always
Target surface form: always
Context:
The art of change-ringing is peculiar to the English, and, like most English peculiarities, unintelligible to the rest of the world. Dorothy L. Sayers, "The Nine Tailors "ASLACTON, England -- Of all scenes that evoke rural England, this is one of the loveliest: An ancient stone church stands amid the fields, the sound of bells cascading from its tower, calling the faithful to evensong. The parishioners of St. Michael and All Angels stop to chat at the church door, as members here <t>always</t> have. In the tower, five men and women pull rhythmically on ropes attached to the same five bells that first sounded here in 1614.

Candidate senses:
1. sense_key=always%4:02:00:: | definition=at all times; all the time and on every occasion | synonyms=ever, e'er | examples=I will always be there to help you; always arrives on time
2. sense_key=always%4:02:04:: | definition=without variation or change, in every case | synonyms=constantly, invariably | examples=constantly kind and gracious; he always arrives on time
3. sense_key=always%4:02:03:: | definition=without interruption | synonyms=constantly, forever, perpetually, incessantly | examples=the world is constantly changing
4. sense_key=always%4:02:02:: | definition=at any time or in any event | examples=you can always resign if you don't like it; you could always take a day off
5. sense_key=always%4:02:01:: | definition=forever; throughout all time | examples=we will always be friends; I shall treasure it always

Instructions:
1. Choose the single best indexed sense for the target word in context.
2. WordNet glosses are schematic; use the context, synonyms, and examples together.
3. Do not answer with a WordNet sense key.
4. Return only a JSON object exactly like {"sense_index": 3}.

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    at all times; all the time and on every occasion

    always%4:02:00:: · always.r.01

    • I will always be there to help you
    • always arrives on time

    Synonyms: ever, e'er

    Gold lexEN Gold Maru 2022 Gold Raganato
  2. 2

    without variation or change, in every case

    always%4:02:04:: · constantly.r.01

    • constantly kind and gracious
    • he always arrives on time

    Synonyms: constantly, invariably

    Selected
  3. 3

    without interruption

    always%4:02:03:: · constantly.r.02

    • the world is constantly changing

    Synonyms: constantly, forever, perpetually, incessantly

  4. 4

    at any time or in any event

    always%4:02:02:: · always.r.04

    • you can always resign if you don't like it
    • you could always take a day off
  5. 5

    forever; throughout all time

    always%4:02:01:: · always.r.05

    • we will always be friends
    • I shall treasure it always

now Adv

Instance: senseval2.d000.s008.t000 Dataset: senseval2 Bucket: POS / ADV

Incorrect

Of all scenes that evoke rural England, this is one of the loveliest: An ancient stone church stands amid the fields, the sound of bells cascading from its tower, calling the faithful to evensong.

The parishioners of St. Michael and All Angels stop to chat at the church door, as members here always have.

In the tower, five men and women pull rhythmically on ropes attached to the same five bells that first sounded here in 1614.

But there is also a discordant, modern note in Aslacton, though it can not be heard by the church-goers enjoying the peal of bells this cool autumn evening.

Like most of the other 6,000 churches in Britain with sets of bells, St. Michael once had its own "band "of ringers, who would herald every Sunday morning and evening service.

Now, only one local ringer remains: 64-year-old Derek Hammond.

The others here today live elsewhere.

Show raw prompt

system

You are an English linguist performing fine-grained Word Sense Disambiguation. WordNet glosses are schematic. Use the sentence context, the examples, and the synonyms to choose the best indexed sense. Do not hedge. Choose the single best indexed sense.

user

Target lemma: now
Target surface form: Now
Context:
Of all scenes that evoke rural England, this is one of the loveliest: An ancient stone church stands amid the fields, the sound of bells cascading from its tower, calling the faithful to evensong. The parishioners of St. Michael and All Angels stop to chat at the church door, as members here always have. In the tower, five men and women pull rhythmically on ropes attached to the same five bells that first sounded here in 1614. But there is also a discordant, modern note in Aslacton, though it can not be heard by the church-goers enjoying the peal of bells this cool autumn evening. Like most of the other 6,000 churches in Britain with sets of bells, St. Michael once had its own "band "of ringers, who would herald every Sunday morning and evening service. <t>Now</t>, only one local ringer remains: 64-year-old Derek Hammond. The others here today live elsewhere.

Candidate senses:
1. sense_key=now%4:02:05:: | definition=in the historical present; at this point in the narration of a series of past events | examples=President Kennedy now calls in the National Guard; Washington now decides to cross the Delaware
2. sense_key=now%4:02:00:: | definition=in these times; - Nancy Mitford | synonyms=nowadays, today | examples=it is solely by their language that the upper classes nowadays are distinguished; we now rarely see horse-drawn vehicles on city streets
3. sense_key=now%4:02:02:: | definition=used to preface a command or reproof or request | examples=now hear this!; now pay attention
4. sense_key=now%4:02:04:: | definition=at the present moment | synonyms=at present | examples=goods now on sale; the now-aging dictator
5. sense_key=now%4:02:01:: | definition=without delay or hesitation; with no time intervening | synonyms=immediately, instantly, straightaway, straight off, directly, right away | examples=he answered immediately; found an answer straightaway
6. sense_key=now%4:02:07:: | definition=(prefatory or transitional) indicates a change of subject or activity | examples=Now the next problem is...
7. sense_key=now%4:02:06:: | definition=in the immediate past | examples=told me just now

Instructions:
1. Choose the single best indexed sense for the target word in context.
2. WordNet glosses are schematic; use the context, synonyms, and examples together.
3. Do not answer with a WordNet sense key.
4. Return only a JSON object exactly like {"sense_index": 3}.

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✓

WordNet 3.0 senses

  1. 1

    in the historical present; at this point in the narration of a series of past events

    now%4:02:05:: · now.r.01

    • President Kennedy now calls in the National Guard
    • Washington now decides to cross the Delaware
  2. 2

    in these times; - Nancy Mitford

    now%4:02:00:: · nowadays.r.01

    • it is solely by their language that the upper classes nowadays are distinguished
    • we now rarely see horse-drawn vehicles on city streets

    Synonyms: nowadays, today

    Gold Raganato Selected
  3. 3

    used to preface a command or reproof or request

    now%4:02:02:: · now.r.03

    • now hear this!
    • now pay attention
  4. 4

    at the present moment

    now%4:02:04:: · now.r.04

    • goods now on sale
    • the now-aging dictator

    Synonyms: at present

    Gold lexEN Gold Maru 2022
  5. 5

    without delay or hesitation; with no time intervening

    now%4:02:01:: · immediately.r.01

    • he answered immediately
    • found an answer straightaway

    Synonyms: immediately, instantly, straightaway, straight off, directly, right away, at once, forthwith, like a shot

  6. 6

    (prefatory or transitional) indicates a change of subject or activity

    now%4:02:07:: · now.r.06

    • Now the next problem is...
  7. 7

    in the immediate past

    now%4:02:06:: · now.r.07

    • told me just now

just Adv

Instance: senseval2.d000.s012.t005 Dataset: senseval2 Bucket: POS / ADV

Incorrect

Like most of the other 6,000 churches in Britain with sets of bells, St. Michael once had its own "band "of ringers, who would herald every Sunday morning and evening service.

Now, only one local ringer remains: 64-year-old Derek Hammond.

The others here today live elsewhere.

They belong to a group of 15 ringers -- including two octogenarians and four youngsters in training -- who drive every Sunday from church to church in a sometimes-exhausting effort to keep the bells sounding in the many belfries of East Anglia.

"To ring for even one service at this tower, we have to scrape, "says Mr. Hammond, a retired water-authority worker. "

We've tried to train the youngsters, but they have their discos and their dances, and they just drift away. "

Mr. Hammond worries that old age and the flightiness of youth will diminish the ranks of the East Anglian group that keeps the Aslacton bells pealing.

Show raw prompt

system

You are an English linguist performing fine-grained Word Sense Disambiguation. WordNet glosses are schematic. Use the sentence context, the examples, and the synonyms to choose the best indexed sense. Do not hedge. Choose the single best indexed sense.

user

Target lemma: just
Target surface form: just
Context:
Like most of the other 6,000 churches in Britain with sets of bells, St. Michael once had its own "band "of ringers, who would herald every Sunday morning and evening service. Now, only one local ringer remains: 64-year-old Derek Hammond. The others here today live elsewhere. They belong to a group of 15 ringers -- including two octogenarians and four youngsters in training -- who drive every Sunday from church to church in a sometimes-exhausting effort to keep the bells sounding in the many belfries of East Anglia. "To ring for even one service at this tower, we have to scrape, "says Mr. Hammond, a retired water-authority worker. "We've tried to train the youngsters, but they have their discos and their dances, and they <t>just</t> drift away. "Mr. Hammond worries that old age and the flightiness of youth will diminish the ranks of the East Anglian group that keeps the Aslacton bells pealing.

Candidate senses:
1. sense_key=just%4:02:00:: | definition=and nothing more | synonyms=merely, simply, only, but | examples=I was merely asking; it is simply a matter of time
2. sense_key=just%4:02:05:: | definition=indicating exactness or preciseness | synonyms=precisely, exactly | examples=he was doing precisely (or exactly) what she had told him to do; it was just as he said--the jewel was gone
3. sense_key=just%4:02:01:: | definition=only a moment ago | synonyms=just now | examples=he has just arrived; the sun just now came out
4. sense_key=just%4:02:03:: | definition=absolutely | synonyms=simply | examples=I just can't take it anymore; he was just grand as Romeo
5. sense_key=just%4:02:04:: | definition=only a very short time before; ; ; ; ; - W.B.Yeats | synonyms=barely, hardly, scarcely, scarce | examples=they could barely hear the speaker; we hardly knew them
6. sense_key=just%4:02:06:: | definition=exactly at this moment or the moment described | examples=we've just finished painting the walls, so don't touch them

Instructions:
1. Choose the single best indexed sense for the target word in context.
2. WordNet glosses are schematic; use the context, synonyms, and examples together.
3. Do not answer with a WordNet sense key.
4. Return only a JSON object exactly like {"sense_index": 3}.

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    and nothing more

    just%4:02:00:: · merely.r.01

    • I was merely asking
    • it is simply a matter of time

    Synonyms: merely, simply, only, but

    Selected
  2. 2

    indicating exactness or preciseness

    just%4:02:05:: · precisely.r.01

    • he was doing precisely (or exactly) what she had told him to do
    • it was just as he said--the jewel was gone

    Synonyms: precisely, exactly

  3. 3

    only a moment ago

    just%4:02:01:: · just.r.03

    • he has just arrived
    • the sun just now came out

    Synonyms: just now

  4. 4

    absolutely

    just%4:02:03:: · just.r.04

    • I just can't take it anymore
    • he was just grand as Romeo

    Synonyms: simply

    Gold lexEN Gold Maru 2022 Gold Raganato
  5. 5

    only a very short time before; ; ; ; ; - W.B.Yeats

    just%4:02:04:: · barely.r.01

    • they could barely hear the speaker
    • we hardly knew them

    Synonyms: barely, hardly, scarcely, scarce

  6. 6

    exactly at this moment or the moment described

    just%4:02:06:: · just.r.06

    • we've just finished painting the walls, so don't touch them

only Adv

Instance: senseval2.d000.s017.t013 Dataset: senseval2 Bucket: POS / ADV

Incorrect

We've tried to train the youngsters, but they have their discos and their dances, and they just drift away. "

Mr. Hammond worries that old age and the flightiness of youth will diminish the ranks of the East Anglian group that keeps the Aslacton bells pealing.

History, after all, is not on his side.

According to a nationwide survey taken a year ago, nearly a third of England's church bells are no longer rung on Sundays because there is no one to ring them.

It is easy to see why the ancient art is on the ropes.

The less complicated version of playing tunes on bells, as do the carillons of continental Europe, is considered by the English to be childish, fit only for foreigners.

Change-ringing, a mind-boggling exercise the English invented 380 years ago, requires physical dexterity -- some bells weigh more than a ton -- combined with intense mental concentration.

Show raw prompt

system

You are an English linguist performing fine-grained Word Sense Disambiguation. WordNet glosses are schematic. Use the sentence context, the examples, and the synonyms to choose the best indexed sense. Do not hedge. Choose the single best indexed sense.

user

Target lemma: only
Target surface form: only
Context:
We've tried to train the youngsters, but they have their discos and their dances, and they just drift away. "Mr. Hammond worries that old age and the flightiness of youth will diminish the ranks of the East Anglian group that keeps the Aslacton bells pealing. History, after all, is not on his side. According to a nationwide survey taken a year ago, nearly a third of England's church bells are no longer rung on Sundays because there is no one to ring them. It is easy to see why the ancient art is on the ropes. The less complicated version of playing tunes on bells, as do the carillons of continental Europe, is considered by the English to be childish, fit <t>only</t> for foreigners. Change-ringing, a mind-boggling exercise the English invented 380 years ago, requires physical dexterity -- some bells weigh more than a ton -- combined with intense mental concentration.

Candidate senses:
1. sense_key=only%4:02:02:: | definition=and nothing more | synonyms=merely, simply, just, but | examples=I was merely asking; it is simply a matter of time
2. sense_key=only%4:02:01:: | definition=without any others being included or involved | synonyms=entirely, exclusively, solely, alone | examples=was entirely to blame; a school devoted entirely to the needs of problem children
3. sense_key=only%4:02:00:: | definition=with nevertheless the final result | examples=He arrived only to find his wife dead; We won only to lose again in the next round
4. sense_key=only%4:02:04:: | definition=in the final outcome | examples=These news will only make you more upset
5. sense_key=only%4:02:06:: | definition=except that | examples=It was the same story; only this time she came out better
6. sense_key=only%4:02:03:: | definition=never except when | synonyms=only if, only when | examples=call me only if your cold gets worse
7. sense_key=only%4:02:05:: | definition=as recently as | examples=I spoke to him only an hour ago

Instructions:
1. Choose the single best indexed sense for the target word in context.
2. WordNet glosses are schematic; use the context, synonyms, and examples together.
3. Do not answer with a WordNet sense key.
4. Return only a JSON object exactly like {"sense_index": 3}.

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    and nothing more

    only%4:02:02:: · merely.r.01

    • I was merely asking
    • it is simply a matter of time

    Synonyms: merely, simply, just, but

    Selected
  2. 2

    without any others being included or involved

    only%4:02:01:: · entirely.r.02

    • was entirely to blame
    • a school devoted entirely to the needs of problem children

    Synonyms: entirely, exclusively, solely, alone

    Gold lexEN Gold Maru 2022 Gold Raganato
  3. 3

    with nevertheless the final result

    only%4:02:00:: · only.r.03

    • He arrived only to find his wife dead
    • We won only to lose again in the next round
  4. 4

    in the final outcome

    only%4:02:04:: · only.r.04

    • These news will only make you more upset
  5. 5

    except that

    only%4:02:06:: · only.r.05

    • It was the same story; only this time she came out better
  6. 6

    never except when

    only%4:02:03:: · only.r.06

    • call me only if your cold gets worse

    Synonyms: only if, only when

  7. 7

    as recently as

    only%4:02:05:: · only.r.07

    • I spoke to him only an hour ago