Verified run

claude-haiku-5-5

claude-haiku-5.5-xhigh-reasoning-p002-lexen-v1-20261009

Rank
112
Accuracy lexEN v1 · WordNet fine
91.13%
Correct
4430 / 4861
Actual run cost
$0.741
Cost / M items
$152

Accuracy across label schemes

The headline accuracy is lexEN v1 · WordNet fine-grained. Each cell re-scores this run's predictions against a different gold label set and sense granularity; coarse counts a prediction correct when its WordNet sense maps to the same Glite or CSI concept as a gold sense. About these label schemes → About coarsening →

Score against WordNet fine-grainedGlite coarse-grainedCSI coarse-grained (Lacerra 2020)
lexEN v1 91.13% 4430 / 4861 96.79% 4705 / 4861 94.16% 4577 / 4861
Maru 2022 (ALLamended) 88.44% 4299 / 4861 95.74% 4654 / 4861 92.59% 4501 / 4861
Raganato 2017 (original) 82.06% 3989 / 4861 93.15% 4528 / 4861 88.56% 4305 / 4861

Model

Display
claude-haiku-5-5
Requested
claude-haiku-5-5
Resolved
claude-haiku-5-5
Vendor
Anthropic
API provider
Anthropic
Hosting
cloud_api
Source
proprietary
License
n/a
Reasoning
xhigh
Model URL
n/a
Endpoint
n/a
Resolved counts
claude-haiku-5-5: 4861

Reproducibility

Created
2026-10-09T12:51:26.658583+00:00
Run schema
sensebench-run-v2
Commit
9040664a78ce4afacbe5400eaf6538ee664f3dea
Dataset ID
lexen
Dataset version
lexen-v1
Dataset revision
n/a
Dataset items
4,861
Dataset hash
sha256:5fd4382b93f19087a1e31f6dd7d1db17c1eb17ff80fcbe1d3fdd55c0c3ecefe8
Prompt
p002
Prompt version
0.1.0
Repository files
Browse in GitHub

Runtime

Calls
4861
Total tokens
2,773,328
Tokens / item
570.5
Total cost
$0.741
Cost / M items
$152
Cost source
litellm_estimate
Elapsed
2.44m
Success items
4,861
Monosemous
0
No candidates
0
No valid vote
0
Invalid votes
0
Transport errors
0

Runner

GitHub
vassiliphilippov
Name
Vassili Philippov
Contact
n/a

Sampling

Temperature
n/a
Top P
n/a
Max tokens
32,768
Seed
n/a
Extra
n/a

Policy

Votes / item
1
Reasks / invalid vote
1
Tie break
earliest_vote
Monosemous
short_circuit

Cost And Token Breakdown

Actual recorded run cost: $0.741. Cost estimate source: litellm_estimate.

Token Type Tokens Cost Price / 1M tokens
Input uncached 1,614,402 $0.161 $0.100
Input cached 0 $0.00 $0.0100
Output 1,158,926 $0.579 $0.500
Reasoning output 0 included in output cost n/a

Reasoning output tokens are shown separately when the provider reports them; recorded cost is stored in the output cost bucket.

Download Raw Run Files

Slice Breakdown

Group Bucket Accuracy Correct Items
Candidate Count 11+ 88.04% 699 794
Candidate Count 2 95.79% 819 855
Candidate Count 3-5 91.57% 1781 1945
Candidate Count 6-10 89.27% 1131 1267
POS ADJ 92.26% 608 659
POS ADV 90.26% 176 195
POS NOUN 91.85% 2661 2897
POS VERB 88.74% 985 1110
Source Dataset semeval2013 93.07% 1061 1140
Source Dataset semeval2015 90.39% 668 739
Source Dataset senseval2 91.10% 1484 1629
Source Dataset senseval3 89.95% 1217 1353

Worst-Bucket Examples

figure Noun

Instance: semeval2013.d002.s002.t001 Dataset: semeval2013 Bucket: Candidate Count / 11+

Incorrect

According to final closing figures, the Dow Jones Industrial Average has risen by 65.67 points to 10 471.50 points, whereas the NASDAQ, with a technology predominance, fell by 0.55 point to 2 190.31 points.

Show raw prompt

user

Which numbered sense of "figure" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.

Sentence: According to final closing <t>figures</t>, the Dow Jones Industrial Average has risen by 65.67 points to 10 471.50 points, whereas the NASDAQ, with a technology predominance, fell by 0.55 point to 2 190.31 points.

Senses:
1. definition=a diagram or picture illustrating textual material | examples=the area covered can be seen from Figure 2
2. definition=alternative names for the body of a human being | examples=Leonardo studied the human body
3. definition=one of the elements that collectively form a system of numeration | examples=0 and 1 are digits
4. definition=a model of a bodily form (especially of a person) | examples=he made a figure of Santa Claus
5. definition=a well-known or notable person | examples=they studied all the great names in the history of France
6. definition=a combination of points and lines and planes that form a visible palpable shape
7. definition=an amount of money expressed numerically | examples=a figure of $17 was suggested
8. definition=the impression produced by a person | examples=he cut a fine figure
9. definition=the property possessed by a sum or total or indefinite quantity of units or individuals | examples=he had a number of chores to do
10. definition=language used in a figurative or nonliteral sense
11. definition=a unitary percept having structure and coherence that is the object of attention and that stands out against a ground
12. definition=a decorative or artistic work | examples=the coach had a design on the doors
13. definition=a predetermined set of movements in dancing or skating | examples=she made the best score on compulsory figures

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    a diagram or picture illustrating textual material

    figure%1:10:00:: · figure.n.01

    • the area covered can be seen from Figure 2

    Synonyms: fig

  2. 2

    alternative names for the body of a human being

    figure%1:08:00:: · human_body.n.01

    • Leonardo studied the human body
    • he has a strong physique

    Synonyms: human body, physical body, material body, soma, build, physique, anatomy, shape, bod, chassis, frame, form, flesh

  3. 3

    one of the elements that collectively form a system of numeration

    figure%1:23:00:: · digit.n.01

    • 0 and 1 are digits

    Synonyms: digit

  4. 4

    a model of a bodily form (especially of a person)

    figure%1:06:00:: · figure.n.04

    • he made a figure of Santa Claus
  5. 5

    a well-known or notable person

    figure%1:18:00:: · name.n.04

    • they studied all the great names in the history of France
    • she is an important figure in modern music

    Synonyms: name, public figure

  6. 6

    a combination of points and lines and planes that form a visible palpable shape

    figure%1:25:00:: · figure.n.06

  7. 7

    an amount of money expressed numerically

    figure%1:21:00:: · figure.n.07

    • a figure of $17 was suggested
    Selected
  8. 8

    the impression produced by a person

    figure%1:07:00:: · figure.n.08

    • he cut a fine figure
    • a heroic figure
  9. 9

    the property possessed by a sum or total or indefinite quantity of units or individuals

    figure%1:07:01:: · number.n.01

    • he had a number of chores to do
    • the number of parameters is small

    Synonyms: number

    Gold lexEN Gold Maru 2022 Gold Raganato
  10. 10

    language used in a figurative or nonliteral sense

    figure%1:10:01:: · trope.n.01

    Synonyms: trope, figure of speech, image

  11. 11

    a unitary percept having structure and coherence that is the object of attention and that stands out against a ground

    figure%1:09:00:: · figure.n.11

  12. 12

    a decorative or artistic work

    figure%1:06:01:: · design.n.04

    • the coach had a design on the doors

    Synonyms: design, pattern

  13. 13

    a predetermined set of movements in dancing or skating

    figure%1:04:00:: · figure.n.13

    • she made the best score on compulsory figures

court Noun

Instance: semeval2013.d005.s005.t005 Dataset: semeval2013 Bucket: Candidate Count / 11+

Incorrect

But lawyers for Mr. Hayes have already made court filings that sketch out appeals arguments that are likely to occupy the courts for years.

Show raw prompt

user

Which numbered sense of "court" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.

Sentence: But lawyers for Mr. Hayes have already made court filings that sketch out appeals arguments that are likely to occupy the <t>courts</t> for years.

Senses:
1. definition=an assembly (including one or more judges) to conduct judicial business
2. definition=a room in which a lawcourt sits | examples=television cameras were admitted in the courtroom
3. definition=the sovereign and his advisers who are the governing power of a state
4. definition=a specially marked horizontal area within which a game is played | examples=players had to reserve a court in advance
5. definition=Australian woman tennis player who won many major championships (born in 1947)
6. definition=the family and retinue of a sovereign or prince
7. definition=a hotel for motorists; provides direct access from rooms to parking area
8. definition=a tribunal that is presided over by a magistrate or by one or more judges who administer justice according to the laws
9. definition=the residence of a sovereign or nobleman | examples=the king will visit the duke's court
10. definition=an area wholly or partly surrounded by walls or buildings | examples=the house was built around an inner court
11. definition=respectful deference | examples=pay court to the emperor

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    an assembly (including one or more judges) to conduct judicial business

    court%1:14:00:: · court.n.01

    Synonyms: tribunal, judicature

    Gold lexEN Gold Maru 2022 Gold Raganato
  2. 2

    a room in which a lawcourt sits

    court%1:06:02:: · court.n.02

    • television cameras were admitted in the courtroom

    Synonyms: courtroom

  3. 3

    the sovereign and his advisers who are the governing power of a state

    court%1:14:01:: · court.n.03

    Synonyms: royal court

  4. 4

    a specially marked horizontal area within which a game is played

    court%1:06:01:: · court.n.04

    • players had to reserve a court in advance
  5. 5

    Australian woman tennis player who won many major championships (born in 1947)

    court%1:18:00:: · court.n.05

    Synonyms: Margaret Court

  6. 6

    the family and retinue of a sovereign or prince

    court%1:14:02:: · court.n.06

    Synonyms: royal court

  7. 7

    a hotel for motorists; provides direct access from rooms to parking area

    court%1:06:03:: · motor_hotel.n.01

    Synonyms: motor hotel, motor inn, motor lodge, tourist court

  8. 8

    a tribunal that is presided over by a magistrate or by one or more judges who administer justice according to the laws

    court%1:06:05:: · court.n.08

    Synonyms: lawcourt, court of law, court of justice

    Selected
  9. 9

    the residence of a sovereign or nobleman

    court%1:06:04:: · court.n.09

    • the king will visit the duke's court
  10. 10

    an area wholly or partly surrounded by walls or buildings

    court%1:06:00:: · court.n.10

    • the house was built around an inner court

    Synonyms: courtyard

  11. 11

    respectful deference

    court%1:04:00:: · court.n.11

    • pay court to the emperor

    Synonyms: homage

court Noun

Instance: semeval2013.d005.s027.t006 Dataset: semeval2013 Bucket: Candidate Count / 11+

Incorrect

In interviews, some journalists said the claim that instant news was more incendiary than reports delivered more slowly was a Luddite view that would make poor court policy governing coverage of future trials.

Show raw prompt

user

Which numbered sense of "court" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.

Sentence: In interviews, some journalists said the claim that instant news was more incendiary than reports delivered more slowly was a Luddite view that would make poor <t>court</t> policy governing coverage of future trials.

Senses:
1. definition=an assembly (including one or more judges) to conduct judicial business
2. definition=a room in which a lawcourt sits | examples=television cameras were admitted in the courtroom
3. definition=the sovereign and his advisers who are the governing power of a state
4. definition=a specially marked horizontal area within which a game is played | examples=players had to reserve a court in advance
5. definition=Australian woman tennis player who won many major championships (born in 1947)
6. definition=the family and retinue of a sovereign or prince
7. definition=a hotel for motorists; provides direct access from rooms to parking area
8. definition=a tribunal that is presided over by a magistrate or by one or more judges who administer justice according to the laws
9. definition=the residence of a sovereign or nobleman | examples=the king will visit the duke's court
10. definition=an area wholly or partly surrounded by walls or buildings | examples=the house was built around an inner court
11. definition=respectful deference | examples=pay court to the emperor

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    an assembly (including one or more judges) to conduct judicial business

    court%1:14:00:: · court.n.01

    Synonyms: tribunal, judicature

    Gold lexEN Gold Maru 2022 Gold Raganato
  2. 2

    a room in which a lawcourt sits

    court%1:06:02:: · court.n.02

    • television cameras were admitted in the courtroom

    Synonyms: courtroom

  3. 3

    the sovereign and his advisers who are the governing power of a state

    court%1:14:01:: · court.n.03

    Synonyms: royal court

  4. 4

    a specially marked horizontal area within which a game is played

    court%1:06:01:: · court.n.04

    • players had to reserve a court in advance
  5. 5

    Australian woman tennis player who won many major championships (born in 1947)

    court%1:18:00:: · court.n.05

    Synonyms: Margaret Court

  6. 6

    the family and retinue of a sovereign or prince

    court%1:14:02:: · court.n.06

    Synonyms: royal court

  7. 7

    a hotel for motorists; provides direct access from rooms to parking area

    court%1:06:03:: · motor_hotel.n.01

    Synonyms: motor hotel, motor inn, motor lodge, tourist court

  8. 8

    a tribunal that is presided over by a magistrate or by one or more judges who administer justice according to the laws

    court%1:06:05:: · court.n.08

    Synonyms: lawcourt, court of law, court of justice

    Selected
  9. 9

    the residence of a sovereign or nobleman

    court%1:06:04:: · court.n.09

    • the king will visit the duke's court
  10. 10

    an area wholly or partly surrounded by walls or buildings

    court%1:06:00:: · court.n.10

    • the house was built around an inner court

    Synonyms: courtyard

  11. 11

    respectful deference

    court%1:04:00:: · court.n.11

    • pay court to the emperor

    Synonyms: homage

court Noun

Instance: semeval2013.d005.s028.t005 Dataset: semeval2013 Bucket: Candidate Count / 11+

Incorrect

But some lawyers said appeals courts were often attracted to arguments that social and technological changes required adaptation by the courts.

Show raw prompt

user

Which numbered sense of "court" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.

Sentence: But some lawyers said appeals courts were often attracted to arguments that social and technological changes required adaptation by the <t>courts</t>.

Senses:
1. definition=an assembly (including one or more judges) to conduct judicial business
2. definition=a room in which a lawcourt sits | examples=television cameras were admitted in the courtroom
3. definition=the sovereign and his advisers who are the governing power of a state
4. definition=a specially marked horizontal area within which a game is played | examples=players had to reserve a court in advance
5. definition=Australian woman tennis player who won many major championships (born in 1947)
6. definition=the family and retinue of a sovereign or prince
7. definition=a hotel for motorists; provides direct access from rooms to parking area
8. definition=a tribunal that is presided over by a magistrate or by one or more judges who administer justice according to the laws
9. definition=the residence of a sovereign or nobleman | examples=the king will visit the duke's court
10. definition=an area wholly or partly surrounded by walls or buildings | examples=the house was built around an inner court
11. definition=respectful deference | examples=pay court to the emperor

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    an assembly (including one or more judges) to conduct judicial business

    court%1:14:00:: · court.n.01

    Synonyms: tribunal, judicature

    Gold lexEN Gold Maru 2022 Gold Raganato
  2. 2

    a room in which a lawcourt sits

    court%1:06:02:: · court.n.02

    • television cameras were admitted in the courtroom

    Synonyms: courtroom

  3. 3

    the sovereign and his advisers who are the governing power of a state

    court%1:14:01:: · court.n.03

    Synonyms: royal court

  4. 4

    a specially marked horizontal area within which a game is played

    court%1:06:01:: · court.n.04

    • players had to reserve a court in advance
  5. 5

    Australian woman tennis player who won many major championships (born in 1947)

    court%1:18:00:: · court.n.05

    Synonyms: Margaret Court

  6. 6

    the family and retinue of a sovereign or prince

    court%1:14:02:: · court.n.06

    Synonyms: royal court

  7. 7

    a hotel for motorists; provides direct access from rooms to parking area

    court%1:06:03:: · motor_hotel.n.01

    Synonyms: motor hotel, motor inn, motor lodge, tourist court

  8. 8

    a tribunal that is presided over by a magistrate or by one or more judges who administer justice according to the laws

    court%1:06:05:: · court.n.08

    Synonyms: lawcourt, court of law, court of justice

    Selected
  9. 9

    the residence of a sovereign or nobleman

    court%1:06:04:: · court.n.09

    • the king will visit the duke's court
  10. 10

    an area wholly or partly surrounded by walls or buildings

    court%1:06:00:: · court.n.10

    • the house was built around an inner court

    Synonyms: courtyard

  11. 11

    respectful deference

    court%1:04:00:: · court.n.11

    • pay court to the emperor

    Synonyms: homage

rule Noun

Instance: semeval2013.d006.s007.t001 Dataset: semeval2013 Bucket: Candidate Count / 11+

Incorrect

It is so extended that it part of the ethos, of the unwritten rules, but followed nonetheless by this sport.

Show raw prompt

user

Which numbered sense of "rule" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.

Sentence: It is so extended that it part of the ethos, of the unwritten <t>rules</t>, but followed nonetheless by this sport.

Senses:
1. definition=a principle or condition that customarily governs behavior | examples=it was his rule to take a walk before breakfast
2. definition=something regarded as a normative example | examples=the convention of not naming the main character
3. definition=prescribed guide for conduct or action
4. definition=(linguistics) a rule describing (or prescribing) a linguistic practice
5. definition=a basic generalization that is accepted as true and that can be used as a basis for reasoning or conduct | examples=their principles of composition characterized all their works
6. definition=the duration of a monarch's or government's power | examples=during the rule of Elizabeth
7. definition=dominance or power through legal authority | examples=France held undisputed dominion over vast areas of Africa
8. definition=directions that define the way a game or sport is to be conducted | examples=he knew the rules of chess
9. definition=any one of a systematic body of regulations defining the way of life of members of a religious order | examples=the rule of St. Dominic
10. definition=a rule or law concerning a natural phenomenon or the function of a complex system | examples=the principle of the conservation of mass
11. definition=(mathematics) a standard procedure for solving a class of mathematical problems | examples=he determined the upper bound with Descartes' rule of signs
12. definition=measuring stick consisting of a strip of wood or metal or plastic with a straight edge that is used for drawing straight lines and measuring lengths

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✓

WordNet 3.0 senses

  1. 1

    a principle or condition that customarily governs behavior

    rule%1:09:00:: · rule.n.01

    • it was his rule to take a walk before breakfast
    • short haircuts were the regulation

    Synonyms: regulation

    Gold Raganato Selected
  2. 2

    something regarded as a normative example

    rule%1:09:01:: · convention.n.02

    • the convention of not naming the main character
    • violence is the rule not the exception

    Synonyms: convention, normal, pattern, formula

  3. 3

    prescribed guide for conduct or action

    rule%1:10:00:: · rule.n.03

    Synonyms: prescript

  4. 4

    (linguistics) a rule describing (or prescribing) a linguistic practice

    rule%1:10:01:: · rule.n.04

    Synonyms: linguistic rule

  5. 5

    a basic generalization that is accepted as true and that can be used as a basis for reasoning or conduct

    rule%1:09:03:: · principle.n.01

    • their principles of composition characterized all their works

    Synonyms: principle

  6. 6

    the duration of a monarch's or government's power

    rule%1:28:00:: · rule.n.06

    • during the rule of Elizabeth
  7. 7

    dominance or power through legal authority

    rule%1:26:00:: · dominion.n.01

    • France held undisputed dominion over vast areas of Africa
    • the rule of Caesar

    Synonyms: dominion

  8. 8

    directions that define the way a game or sport is to be conducted

    rule%1:10:02:: · rule.n.08

    • he knew the rules of chess
    Gold lexEN Gold Maru 2022
  9. 9

    any one of a systematic body of regulations defining the way of life of members of a religious order

    rule%1:10:03:: · rule.n.09

    • the rule of St. Dominic
  10. 10

    a rule or law concerning a natural phenomenon or the function of a complex system

    rule%1:09:02:: · principle.n.04

    • the principle of the conservation of mass
    • the principle of jet propulsion

    Synonyms: principle

  11. 11

    (mathematics) a standard procedure for solving a class of mathematical problems

    rule%1:09:04:: · rule.n.11

    • he determined the upper bound with Descartes' rule of signs
    • he gave us a general formula for attacking polynomials

    Synonyms: formula

  12. 12

    measuring stick consisting of a strip of wood or metal or plastic with a straight edge that is used for drawing straight lines and measuring lengths

    rule%1:06:00:: · rule.n.12

    Synonyms: ruler

part Noun

Instance: semeval2013.d006.s008.t000 Dataset: semeval2013 Bucket: Candidate Count / 11+

Incorrect

Part of the debate arisen in the related case is caused by the fact that has been carried out by team of the Real Madrid organization (that in its hymn presumes of nobility and honor, and that has fined its players for analogous acts) and by the dramatization orchestrated, apparently, by Mourinho, who adds one more to his expanded history of histrionic and provoking technician.

Show raw prompt

user

Which numbered sense of "part" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.

Sentence: <t>Part</t> of the debate arisen in the related case is caused by the fact that has been carried out by team of the Real Madrid organization (that in its hymn presumes of nobility and honor, and that has fined its players for analogous acts) and by the dramatization orchestrated, apparently, by Mourinho, who adds one more to his expanded history of histrionic and provoking technician.

Senses:
1. definition=something determined in relation to something that includes it | examples=he wanted to feel a part of something bigger than himself
2. definition=something less than the whole of a human artifact | examples=the rear part of the house
3. definition=a portion of a natural object | examples=they analyzed the river into three parts
4. definition=that which concerns a person with regard to a particular role or situation | examples=it requires vigilance on our part
5. definition=the extended spatial location of something | examples=the farming regions of France
6. definition=the actions and activities assigned to or required or expected of a person or group | examples=the function of a teacher
7. definition=an actor's portrayal of someone in a play | examples=she played the part of Desdemona
8. definition=assets belonging to or due to or contributed by an individual person or group | examples=he wanted his share in cash
9. definition=one of the portions into which something is regarded as divided and which together constitute a whole | examples=the written part of the exam
10. definition=a line of scalp that can be seen when sections of hair are combed in opposite directions | examples=his part was right in the middle
11. definition=the melody carried by a particular voice or instrument in polyphonic music | examples=he tried to sing the tenor part
12. definition=the part played by a person in bringing about a result | examples=I am proud of my contribution in advancing the project

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    something determined in relation to something that includes it

    part%1:24:00:: · part.n.01

    • he wanted to feel a part of something bigger than himself
    • I read a portion of the manuscript

    Synonyms: portion, component part, component, constituent

    Gold lexEN Gold Maru 2022 Gold Raganato
  2. 2

    something less than the whole of a human artifact

    part%1:06:00:: · part.n.02

    • the rear part of the house
    • glue the two parts together

    Synonyms: portion

  3. 3

    a portion of a natural object

    part%1:17:00:: · part.n.03

    • they analyzed the river into three parts
    • he needed a piece of granite

    Synonyms: piece

  4. 4

    that which concerns a person with regard to a particular role or situation

    part%1:09:02:: · part.n.04

    • it requires vigilance on our part
    • they resisted every effort on his part
  5. 5

    the extended spatial location of something

    part%1:15:00:: · region.n.01

    • the farming regions of France
    • religions in all parts of the world

    Synonyms: region

  6. 6

    the actions and activities assigned to or required or expected of a person or group

    part%1:04:01:: · function.n.03

    • the function of a teacher
    • the government must do its part

    Synonyms: function, office, role

  7. 7

    an actor's portrayal of someone in a play

    part%1:09:01:: · character.n.04

    • she played the part of Desdemona

    Synonyms: character, role, theatrical role, persona

  8. 8

    assets belonging to or due to or contributed by an individual person or group

    part%1:21:00:: · share.n.01

    • he wanted his share in cash

    Synonyms: share, portion, percentage

  9. 9

    one of the portions into which something is regarded as divided and which together constitute a whole

    part%1:09:00:: · part.n.09

    • the written part of the exam
    • the finance section of the company

    Synonyms: section, division

    Selected
  10. 10

    a line of scalp that can be seen when sections of hair are combed in opposite directions

    part%1:08:01:: · part.n.10

    • his part was right in the middle

    Synonyms: parting

  11. 11

    the melody carried by a particular voice or instrument in polyphonic music

    part%1:10:00:: · part.n.11

    • he tried to sing the tenor part

    Synonyms: voice

  12. 12

    the part played by a person in bringing about a result

    part%1:04:03:: · contribution.n.01

    • I am proud of my contribution in advancing the project
    • they all did their share of the work

    Synonyms: contribution, share

thing Noun

Instance: semeval2013.d008.s006.t000 Dataset: semeval2013 Bucket: Candidate Count / 11+

Incorrect

Two related things stand out in the results of this year's poll, taken in September and early October.

Show raw prompt

user

Which numbered sense of "thing" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.

Sentence: Two related <t>things</t> stand out in the results of this year's poll, taken in September and early October.

Senses:
1. definition=a special situation | examples=this thing has got to end
2. definition=an action | examples=how could you do such a thing?
3. definition=a special abstraction | examples=a thing of the spirit
4. definition=an artifact | examples=how does this thing work?
5. definition=an event | examples=a funny thing happened on the way to the...
6. definition=a vaguely specified concern | examples=several matters to attend to
7. definition=a statement regarded as an object | examples=to say the same thing in other terms
8. definition=an entity that is not named specifically | examples=I couldn't tell what the thing was
9. definition=any attribute or quality considered as having its own existence | examples=the thing I like about her is ...
10. definition=a special objective | examples=the thing is to stay in bounds
11. definition=a persistent illogical feeling of desire or aversion | examples=he has a thing about seafood
12. definition=a separate and self-contained entity

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    a special situation

    thing%1:26:00:: · thing.n.01

    • this thing has got to end
    • it is a remarkable thing
    Gold Raganato
  2. 2

    an action

    thing%1:04:00:: · thing.n.02

    • how could you do such a thing?
  3. 3

    a special abstraction

    thing%1:09:00:: · thing.n.03

    • a thing of the spirit
    • things of the heart
  4. 4

    an artifact

    thing%1:06:00:: · thing.n.04

    • how does this thing work?
  5. 5

    an event

    thing%1:11:00:: · thing.n.05

    • a funny thing happened on the way to the...
  6. 6

    a vaguely specified concern

    thing%1:09:02:: · matter.n.01

    • several matters to attend to
    • it is none of your affair

    Synonyms: matter, affair

    Gold lexEN
  7. 7

    a statement regarded as an object

    thing%1:10:00:: · thing.n.07

    • to say the same thing in other terms
    • how can you say such a thing?
  8. 8

    an entity that is not named specifically

    thing%1:06:01:: · thing.n.08

    • I couldn't tell what the thing was
  9. 9

    any attribute or quality considered as having its own existence

    thing%1:07:00:: · thing.n.09

    • the thing I like about her is ...
    Selected
  10. 10

    a special objective

    thing%1:09:01:: · thing.n.10

    • the thing is to stay in bounds
  11. 11

    a persistent illogical feeling of desire or aversion

    thing%1:12:00:: · thing.n.11

    • he has a thing about seafood
    • she has a thing about him
  12. 12

    a separate and self-contained entity

    thing%1:03:00:: · thing.n.12

    Gold Maru 2022

figure Noun

Instance: semeval2013.d008.s013.t005 Dataset: semeval2013 Bucket: Candidate Count / 11+

Incorrect

Some 44% of respondents across the region pronounced themselves satisfied with how their country's democracy works in practice, the same figure as last year but a big increase on the 25% in 2001 (chart 3).

Show raw prompt

user

Which numbered sense of "figure" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.

Sentence: Some 44% of respondents across the region pronounced themselves satisfied with how their country's democracy works in practice, the same <t>figure</t> as last year but a big increase on the 25% in 2001 (chart 3).

Senses:
1. definition=a diagram or picture illustrating textual material | examples=the area covered can be seen from Figure 2
2. definition=alternative names for the body of a human being | examples=Leonardo studied the human body
3. definition=one of the elements that collectively form a system of numeration | examples=0 and 1 are digits
4. definition=a model of a bodily form (especially of a person) | examples=he made a figure of Santa Claus
5. definition=a well-known or notable person | examples=they studied all the great names in the history of France
6. definition=a combination of points and lines and planes that form a visible palpable shape
7. definition=an amount of money expressed numerically | examples=a figure of $17 was suggested
8. definition=the impression produced by a person | examples=he cut a fine figure
9. definition=the property possessed by a sum or total or indefinite quantity of units or individuals | examples=he had a number of chores to do
10. definition=language used in a figurative or nonliteral sense
11. definition=a unitary percept having structure and coherence that is the object of attention and that stands out against a ground
12. definition=a decorative or artistic work | examples=the coach had a design on the doors
13. definition=a predetermined set of movements in dancing or skating | examples=she made the best score on compulsory figures

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    a diagram or picture illustrating textual material

    figure%1:10:00:: · figure.n.01

    • the area covered can be seen from Figure 2

    Synonyms: fig

  2. 2

    alternative names for the body of a human being

    figure%1:08:00:: · human_body.n.01

    • Leonardo studied the human body
    • he has a strong physique

    Synonyms: human body, physical body, material body, soma, build, physique, anatomy, shape, bod, chassis, frame, form, flesh

  3. 3

    one of the elements that collectively form a system of numeration

    figure%1:23:00:: · digit.n.01

    • 0 and 1 are digits

    Synonyms: digit

  4. 4

    a model of a bodily form (especially of a person)

    figure%1:06:00:: · figure.n.04

    • he made a figure of Santa Claus
  5. 5

    a well-known or notable person

    figure%1:18:00:: · name.n.04

    • they studied all the great names in the history of France
    • she is an important figure in modern music

    Synonyms: name, public figure

  6. 6

    a combination of points and lines and planes that form a visible palpable shape

    figure%1:25:00:: · figure.n.06

  7. 7

    an amount of money expressed numerically

    figure%1:21:00:: · figure.n.07

    • a figure of $17 was suggested
    Selected
  8. 8

    the impression produced by a person

    figure%1:07:00:: · figure.n.08

    • he cut a fine figure
    • a heroic figure
  9. 9

    the property possessed by a sum or total or indefinite quantity of units or individuals

    figure%1:07:01:: · number.n.01

    • he had a number of chores to do
    • the number of parameters is small

    Synonyms: number

    Gold lexEN Gold Maru 2022 Gold Raganato
  10. 10

    language used in a figurative or nonliteral sense

    figure%1:10:01:: · trope.n.01

    Synonyms: trope, figure of speech, image

  11. 11

    a unitary percept having structure and coherence that is the object of attention and that stands out against a ground

    figure%1:09:00:: · figure.n.11

  12. 12

    a decorative or artistic work

    figure%1:06:01:: · design.n.04

    • the coach had a design on the doors

    Synonyms: design, pattern

  13. 13

    a predetermined set of movements in dancing or skating

    figure%1:04:00:: · figure.n.13

    • she made the best score on compulsory figures

court Noun

Instance: semeval2013.d009.s005.t001 Dataset: semeval2013 Bucket: Candidate Count / 11+

Incorrect

The lawyer who will stand before the court and argue that the law should be thrown out is likely to be Paul Clement, who served as U.S. solicitor general during the George W. Bush administration.

Show raw prompt

user

Which numbered sense of "court" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.

Sentence: The lawyer who will stand before the <t>court</t> and argue that the law should be thrown out is likely to be Paul Clement, who served as U.S. solicitor general during the George W. Bush administration.

Senses:
1. definition=an assembly (including one or more judges) to conduct judicial business
2. definition=a room in which a lawcourt sits | examples=television cameras were admitted in the courtroom
3. definition=the sovereign and his advisers who are the governing power of a state
4. definition=a specially marked horizontal area within which a game is played | examples=players had to reserve a court in advance
5. definition=Australian woman tennis player who won many major championships (born in 1947)
6. definition=the family and retinue of a sovereign or prince
7. definition=a hotel for motorists; provides direct access from rooms to parking area
8. definition=a tribunal that is presided over by a magistrate or by one or more judges who administer justice according to the laws
9. definition=the residence of a sovereign or nobleman | examples=the king will visit the duke's court
10. definition=an area wholly or partly surrounded by walls or buildings | examples=the house was built around an inner court
11. definition=respectful deference | examples=pay court to the emperor

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    an assembly (including one or more judges) to conduct judicial business

    court%1:14:00:: · court.n.01

    Synonyms: tribunal, judicature

    Gold lexEN Gold Maru 2022 Gold Raganato
  2. 2

    a room in which a lawcourt sits

    court%1:06:02:: · court.n.02

    • television cameras were admitted in the courtroom

    Synonyms: courtroom

  3. 3

    the sovereign and his advisers who are the governing power of a state

    court%1:14:01:: · court.n.03

    Synonyms: royal court

  4. 4

    a specially marked horizontal area within which a game is played

    court%1:06:01:: · court.n.04

    • players had to reserve a court in advance
  5. 5

    Australian woman tennis player who won many major championships (born in 1947)

    court%1:18:00:: · court.n.05

    Synonyms: Margaret Court

  6. 6

    the family and retinue of a sovereign or prince

    court%1:14:02:: · court.n.06

    Synonyms: royal court

  7. 7

    a hotel for motorists; provides direct access from rooms to parking area

    court%1:06:03:: · motor_hotel.n.01

    Synonyms: motor hotel, motor inn, motor lodge, tourist court

  8. 8

    a tribunal that is presided over by a magistrate or by one or more judges who administer justice according to the laws

    court%1:06:05:: · court.n.08

    Synonyms: lawcourt, court of law, court of justice

    Selected
  9. 9

    the residence of a sovereign or nobleman

    court%1:06:04:: · court.n.09

    • the king will visit the duke's court
  10. 10

    an area wholly or partly surrounded by walls or buildings

    court%1:06:00:: · court.n.10

    • the house was built around an inner court

    Synonyms: courtyard

  11. 11

    respectful deference

    court%1:04:00:: · court.n.11

    • pay court to the emperor

    Synonyms: homage

number Noun

Instance: semeval2013.d010.s013.t000 Dataset: semeval2013 Bucket: Candidate Count / 11+

Incorrect

Almost near the centennial number is Carles Puyol, with 97 games.

Show raw prompt

user

Which numbered sense of "number" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.

Sentence: Almost near the centennial <t>number</t> is Carles Puyol, with 97 games.

Senses:
1. definition=the property possessed by a sum or total or indefinite quantity of units or individuals | examples=he had a number of chores to do
2. definition=a concept of quantity involving zero and units | examples=every number has a unique position in the sequence
3. definition=a short theatrical performance that is part of a longer program | examples=he did his act three times every evening
4. definition=the number is used in calling a particular telephone | examples=he has an unlisted number
5. definition=a symbol used to represent a number | examples=he learned to write the numerals before he went to school
6. definition=one of a series published periodically | examples=she found an old issue of the magazine in her dentist's waiting room
7. definition=a select company of people | examples=I hope to become one of their number before I die
8. definition=a numeral or string of numerals that is used for identification | examples=she refused to give them her Social Security number
9. definition=a clothing measurement | examples=a number 13 shoe
10. definition=the grammatical category for the forms of nouns and pronouns and verbs that are used depending on the number of entities involved (singular or dual or plural) | examples=in English the subject and the verb must agree in number
11. definition=an item of merchandise offered for sale | examples=she preferred the black nylon number

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✗

WordNet 3.0 senses

  1. 1

    the property possessed by a sum or total or indefinite quantity of units or individuals

    number%1:07:00:: · number.n.01

    • he had a number of chores to do
    • the number of parameters is small

    Synonyms: figure

    Gold lexEN Gold Maru 2022 Gold Raganato
  2. 2

    a concept of quantity involving zero and units

    number%1:23:00:: · number.n.02

    • every number has a unique position in the sequence
    Selected
  3. 3

    a short theatrical performance that is part of a longer program

    number%1:10:04:: · act.n.04

    • he did his act three times every evening
    • she had a catchy little routine

    Synonyms: act, routine, turn, bit

  4. 4

    the number is used in calling a particular telephone

    number%1:10:03:: · phone_number.n.01

    • he has an unlisted number

    Synonyms: phone number, telephone number

  5. 5

    a symbol used to represent a number

    number%1:10:01:: · numeral.n.01

    • he learned to write the numerals before he went to school

    Synonyms: numeral

  6. 6

    one of a series published periodically

    number%1:10:00:: · issue.n.02

    • she found an old issue of the magazine in her dentist's waiting room

    Synonyms: issue

  7. 7

    a select company of people

    number%1:14:00:: · number.n.07

    • I hope to become one of their number before I die
  8. 8

    a numeral or string of numerals that is used for identification

    number%1:10:02:: · number.n.08

    • she refused to give them her Social Security number

    Synonyms: identification number

  9. 9

    a clothing measurement

    number%1:07:01:: · number.n.09

    • a number 13 shoe
  10. 10

    the grammatical category for the forms of nouns and pronouns and verbs that are used depending on the number of entities involved (singular or dual or plural)

    number%1:10:05:: · number.n.10

    • in English the subject and the verb must agree in number
  11. 11

    an item of merchandise offered for sale

    number%1:06:00:: · number.n.11

    • she preferred the black nylon number
    • this sweater is an all-wool number

part Noun

Instance: semeval2015.d000.s009.t014 Dataset: semeval2015 Bucket: Candidate Count / 11+

Incorrect

Alimta is used together with cisplatin (another anticancer medicine) when the cancer is unresectable (cannot be removed by surgery alone) and malignant (has spread, or is likely to spread easily, to other parts of the body), in patients who have not received chemotherapy (medicines for cancer) before advanced or metastatic non-small cell lung cancer that is not affecting the squamous cells.

Show raw prompt

user

Which numbered sense of "part" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.

Sentence: Alimta is used together with cisplatin (another anticancer medicine) when the cancer is unresectable (cannot be removed by surgery alone) and malignant (has spread, or is likely to spread easily, to other <t>parts</t> of the body), in patients who have not received chemotherapy (medicines for cancer) before advanced or metastatic non-small cell lung cancer that is not affecting the squamous cells.

Senses:
1. definition=something determined in relation to something that includes it | examples=he wanted to feel a part of something bigger than himself
2. definition=something less than the whole of a human artifact | examples=the rear part of the house
3. definition=a portion of a natural object | examples=they analyzed the river into three parts
4. definition=that which concerns a person with regard to a particular role or situation | examples=it requires vigilance on our part
5. definition=the extended spatial location of something | examples=the farming regions of France
6. definition=the actions and activities assigned to or required or expected of a person or group | examples=the function of a teacher
7. definition=an actor's portrayal of someone in a play | examples=she played the part of Desdemona
8. definition=assets belonging to or due to or contributed by an individual person or group | examples=he wanted his share in cash
9. definition=one of the portions into which something is regarded as divided and which together constitute a whole | examples=the written part of the exam
10. definition=a line of scalp that can be seen when sections of hair are combed in opposite directions | examples=his part was right in the middle
11. definition=the melody carried by a particular voice or instrument in polyphonic music | examples=he tried to sing the tenor part
12. definition=the part played by a person in bringing about a result | examples=I am proud of my contribution in advancing the project

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✓

WordNet 3.0 senses

  1. 1

    something determined in relation to something that includes it

    part%1:24:00:: · part.n.01

    • he wanted to feel a part of something bigger than himself
    • I read a portion of the manuscript

    Synonyms: portion, component part, component, constituent

    Gold lexEN Gold Maru 2022 Gold Raganato
  2. 2

    something less than the whole of a human artifact

    part%1:06:00:: · part.n.02

    • the rear part of the house
    • glue the two parts together

    Synonyms: portion

  3. 3

    a portion of a natural object

    part%1:17:00:: · part.n.03

    • they analyzed the river into three parts
    • he needed a piece of granite

    Synonyms: piece

    Gold lexEN Gold Maru 2022 Gold Raganato
  4. 4

    that which concerns a person with regard to a particular role or situation

    part%1:09:02:: · part.n.04

    • it requires vigilance on our part
    • they resisted every effort on his part
  5. 5

    the extended spatial location of something

    part%1:15:00:: · region.n.01

    • the farming regions of France
    • religions in all parts of the world

    Synonyms: region

    Gold lexEN Gold Maru 2022
  6. 6

    the actions and activities assigned to or required or expected of a person or group

    part%1:04:01:: · function.n.03

    • the function of a teacher
    • the government must do its part

    Synonyms: function, office, role

  7. 7

    an actor's portrayal of someone in a play

    part%1:09:01:: · character.n.04

    • she played the part of Desdemona

    Synonyms: character, role, theatrical role, persona

  8. 8

    assets belonging to or due to or contributed by an individual person or group

    part%1:21:00:: · share.n.01

    • he wanted his share in cash

    Synonyms: share, portion, percentage

  9. 9

    one of the portions into which something is regarded as divided and which together constitute a whole

    part%1:09:00:: · part.n.09

    • the written part of the exam
    • the finance section of the company

    Synonyms: section, division

    Gold Raganato Selected
  10. 10

    a line of scalp that can be seen when sections of hair are combed in opposite directions

    part%1:08:01:: · part.n.10

    • his part was right in the middle

    Synonyms: parting

  11. 11

    the melody carried by a particular voice or instrument in polyphonic music

    part%1:10:00:: · part.n.11

    • he tried to sing the tenor part

    Synonyms: voice

  12. 12

    the part played by a person in bringing about a result

    part%1:04:03:: · contribution.n.01

    • I am proud of my contribution in advancing the project
    • they all did their share of the work

    Synonyms: contribution, share

have Verb

Instance: semeval2015.d000.s018.t008 Dataset: semeval2015 Bucket: Candidate Count / 11+

Incorrect

Treatment should be delayed or discontinued, or the dose reduced, in patients whose blood counts are abnormal or who have certain other side effects.

Show raw prompt

user

Which numbered sense of "have" (marked with <t></t>) is used in the sentence below? Return only the number of the correct sense.

Sentence: Treatment should be delayed or discontinued, or the dose reduced, in patients whose blood counts are abnormal or who <t>have</t> certain other side effects.

Senses:
1. definition=have or possess, either in a concrete or an abstract sense | examples=She has $1,000 in the bank
2. definition=have as a feature | examples=This restaurant features the most famous chefs in France
3. definition=go through (mental or physical states or experiences) | examples=get an idea
4. definition=have ownership or possession of | examples=He owns three houses in Florida
5. definition=cause to move; cause to be in a certain position or condition | examples=He got his squad on the ball
6. definition=serve oneself to, or consume regularly | examples=Have another bowl of chicken soup!
7. definition=have a personal or business relationship with someone | examples=have a postdoc
8. definition=organize or be responsible for | examples=hold a reception
9. definition=have left | examples=I have two years left
10. definition=be confronted with | examples=What do we have here?
11. definition=undergo | examples=The stocks had a fast run-up
12. definition=suffer from; be ill with | examples=She has arthritis
13. definition=cause to do; cause to act in a specified manner | examples=The ads induced me to buy a VCR
14. definition=receive willingly something given or offered | examples=The only girl who would have him was the miller's daughter
15. definition=get something; come into possession of | examples=receive payment
16. definition=undergo (as of injuries and illnesses) | examples=She suffered a fracture in the accident
17. definition=achieve a point or goal | examples=Nicklaus had a 70
18. definition=cause to be born | examples=My wife had twins yesterday!
19. definition=have sex with; archaic use | examples=He had taken this woman when she was most vulnerable

Prediction matches gold: lexEN ✗ Maru 2022 ✗ Raganato ✓

WordNet 3.0 senses

  1. 1

    have or possess, either in a concrete or an abstract sense

    have%2:40:00:: · have.v.01

    • She has $1,000 in the bank
    • He has got two beautiful daughters

    Synonyms: have got, hold

  2. 2

    have as a feature

    have%2:42:00:: · have.v.02

    • This restaurant features the most famous chefs in France

    Synonyms: feature

  3. 3

    go through (mental or physical states or experiences)

    have%2:39:06:: · experience.v.03

    • get an idea
    • experience vertigo

    Synonyms: experience, receive, get

    Gold lexEN Gold Maru 2022
  4. 4

    have ownership or possession of

    have%2:40:04:: · own.v.01

    • He owns three houses in Florida
    • How many cars does she have?

    Synonyms: own, possess

  5. 5

    cause to move; cause to be in a certain position or condition

    have%2:30:00:: · get.v.03

    • He got his squad on the ball
    • This let me in for a big surprise

    Synonyms: get, let

  6. 6

    serve oneself to, or consume regularly

    have%2:34:00:: · consume.v.02

    • Have another bowl of chicken soup!
    • I don't take sugar in my coffee

    Synonyms: consume, ingest, take in, take

  7. 7

    have a personal or business relationship with someone

    have%2:41:00:: · have.v.07

    • have a postdoc
    • have an assistant
  8. 8

    organize or be responsible for

    have%2:36:00:: · hold.v.03

    • hold a reception
    • have, throw, or make a party

    Synonyms: hold, throw, make, give

  9. 9

    have left

    have%2:40:01:: · have.v.09

    • I have two years left
    • I don't have any money left
  10. 10

    be confronted with

    have%2:42:12:: · have.v.10

    • What do we have here?
    • Now we have a fine mess
  11. 11

    undergo

    have%2:30:01:: · have.v.11

    • The stocks had a fast run-up

    Synonyms: experience

    Gold lexEN Gold Maru 2022
  12. 12

    suffer from; be ill with

    have%2:29:02:: · have.v.12

    • She has arthritis
    Gold Raganato
  13. 13

    cause to do; cause to act in a specified manner

    have%2:32:00:: · induce.v.02

    • The ads induced me to buy a VCR
    • My children finally got me to buy a computer

    Synonyms: induce, stimulate, cause, get, make

  14. 14

    receive willingly something given or offered

    have%2:40:05:: · accept.v.02

    • The only girl who would have him was the miller's daughter
    • I won't have this dog in my house!

    Synonyms: accept, take

  15. 15

    get something; come into possession of

    have%2:40:03:: · receive.v.01

    • receive payment
    • receive a gift

    Synonyms: receive

  16. 16

    undergo (as of injuries and illnesses)

    have%2:29:05:: · suffer.v.02

    • She suffered a fracture in the accident
    • He had an insulin shock after eating three candy bars

    Synonyms: suffer, sustain, get

    Gold Raganato Selected
  17. 17

    achieve a point or goal

    have%2:40:02:: · have.v.17

    • Nicklaus had a 70
    • The Brazilian team got 4 goals

    Synonyms: get, make

  18. 18

    cause to be born

    have%2:29:00:: · give_birth.v.01

    • My wife had twins yesterday!

    Synonyms: give birth, deliver, bear, birth

  19. 19

    have sex with; archaic use

    have%2:35:00:: · take.v.35

    • He had taken this woman when she was most vulnerable

    Synonyms: take