Different methods of using the judgements of natural language - - PowerPoint PPT Presentation

Motivation/ Background The semantic similarity task Hypothesis/ Contribution

Different methods of using the judgements of natural language speakers on a semantic similarity task.

Irma Cornelisse

Institute for Logic, Language and Computation

December 13, 2010

1 / 12

Motivation/ Background The semantic similarity task Hypothesis/ Contribution

Outline

Motivation/ Background The semantic similarity task Hypothesis/ Contribution

2 / 12

Motivation/ Background The semantic similarity task Hypothesis/ Contribution

Problem description

The problem that will be adressed in the paper is the following: How to evaluate the quality of semantic similarity judgments?

• I will discuss 2 methodologies:
• Gold-standards
• Judgements of natural language users

3 / 12

Motivation/ Background The semantic similarity task Hypothesis/ Contribution

Gold-standards

• Most of the time not complete.
• Often don’t give information on how similar a term is to the target

term.

• Don’t reflect that syonymy is a matter of degree.
• Don’t take into account that judgements of synonymy are not strict,

there are borderline cases.

• Not necessarily reflect the judgement of natural language users.

4 / 12

Motivation/ Background The semantic similarity task Hypothesis/ Contribution

Judgements of natural language users

• Getting them is time consuming.
• Different ways of getting judgements by natural language users,

which lead to different results

• Spontaneously produce
• We know humans face problems spontaneously producing, they don’t

have acces to all their knowledge.

• Judge given terms: where do we get these terms from?
• From your model (evaluate only precision, not recall)
• From a gold-standard
• From an earlier spontaneously producing task by natural language

users.

Judgements of natural language users is used a lot, but there is no insight in how the results of these different methods relate to each other, i.e. what are the consequences of the different design choices?

5 / 12

Motivation/ Background The semantic similarity task Hypothesis/ Contribution

Research question

The main goal of this research is: To obtain insight in how the information obtained by the different methods relate to each other

• Where and how does it differ?
• Where and how does it coincides?

There is probably not one right way, but we can, by characterizing the different methods, argue which methods fits which purpose best.

6 / 12

Motivation/ Background The semantic similarity task Hypothesis/ Contribution

Method

Subjects

• Aproximately 60 1st year students Beta Gamma.

Distributional model

• Cornetto Dutch Set Demo

(http://www.let.rug.nl/erikt/bin/setdemo.cgi) Gold Standard

• Van Dalen ‘Synoniemenwoordenboek’ (thesaurus)

7 / 12

Motivation/ Background The semantic similarity task Hypothesis/ Contribution

The task

10 Dutch terms:

• 5 nouns
• 5 adverbs

Randomly chosen, satisfying the following criteria:

• No polysemy (according to the Van Dale dictionary)
• Only one POS tag possible (according to the Van Dale dictionary)
• 3 or more synonyms according to the Van Dale thesaurus
• 2 or more possible synoyms given by CDSD
• No 2 terms are synonyms according to the Van Dale thesaurus
• No 2 terms are connected to each other by CDSD

8 / 12

Motivation/ Background The semantic similarity task Hypothesis/ Contribution

Subjects: 4 conditions

The spontaneously producing task.

• Come up with a word that is as similar as possible to the target term.

The judging subjects task.

• Classify the terms produced by the previous task into:
• 1. the meaning is the same as the meaning of the target term
• 2. the meaning is very similar to the meaning of the target term
• 3. the meaning is reasonably similar to the meaning of the target term
• 4. the meaning is a bit similar to the meaning of the target term
• 5. the meaning is not similar to the meaning of the target term

The judging gold-standard task.

• Classify the synonyms of the target terms given by Van Dale

thesaurus as above. The judging model task.

• Classify the nearest neighbours of the target terms given by CDSD

as above.

9 / 12

Motivation/ Background The semantic similarity task Hypothesis/ Contribution

Hypothesis

• 1. The spontaneously producing task gives a similar output to the

judging subjects task, when the group of subjects is big enough.

• 2. Only considering the best synonyms from the judging subjects task

will give not enough graduation results.

• 3. The judging gold-standard task will come up with terms not

produced in the spontaneously producing task and vice versa.

• 4. Not all terms produced by subjects, the thesaurus and the computer

model will be judged as similar to the target term by the subjects.

• 5. There are terms produced by the computer model which are not in

the thesaurus (and vice versa).

10 / 12

Motivation/ Background The semantic similarity task Hypothesis/ Contribution

The interesting results

The interesting contributions of the results of this exeperiment are:

• They give insight in the precision and recall of the different

methods, i.e.

• Which method returns terms that are judged by non similar by most

subjects?

• Which method doesn’t return terms that are judged as similar by

most subjects?

• They give insight in the way different methods return a similarity

rate (graduation) of the similar terms, i.e.

• How similar is the term to the target term?

.

11 / 12

Motivation/ Background The semantic similarity task Hypothesis/ Contribution

Discussion

The main goal of this research is: To obtain insight in how the information obtained by the different methods relate to each other

• Where and how does it differ?
• Where and how does it coincides?

There is probably not one right way, but we can, by characterizing the different methods, argue which methods fits which purpose best.

12 / 12