SLIDE 1
Entity Type Modeling for Multi-Document Summarization: Generating - - PowerPoint PPT Presentation
Entity Type Modeling for Multi-Document Summarization: Generating - - PowerPoint PPT Presentation
Entity Type Modeling for Multi-Document Summarization: Generating Descriptive Summaries of Geo-Located Entities Ahmet Aker Natural Language Processing Group Department of Computer Science Research question Multi-document summarization
SLIDE 2
SLIDE 3
Sets of patterns that capture the ways an entity is described in natural language
What are entity type models?
Church: When built Location Visiting Preacher Events History … Volcano: When last erupted Visiting Location Surroundings Height Status … Tower: Visiting Location When built Design Purpose Height …
SLIDE 4
Do entity type models exist?
How can we derive entity type models?
Do entity type models help to select relevant sentences from the documents?
Do entity type models help to reduce redundancy and lead to more coherent summaries?
─
Manual approach
─
Automatic approach
Further research questions
SLIDE 5
Do entity type models exist?
How can we derive entity type models?
Do entity type models help to select relevant sentences from the documents?
Do entity type models help to reduce redundancy and lead to more coherent summaries?
─
Manual approach
─
Automatic approach
Further research questions
SLIDE 6
Do humans associate sets of attributes with entity types?
Height, Year, Location, Designer, Entrance Fee
SLIDE 7
Investigation:
Where is it located? When is it constructed? How tall is it? Location Year Height
SLIDE 8
Investigation shows
Humans have an “entity type model” of what is salient regarding a certain entity type and this model informs their choice of what attributes to seek when seeing an instance of this type (Aker & Gaizauskas 2011, Aker et al. 2013)
Church: When built Location Visiting Preacher Events History … Volcano: When last erupted Visiting Location Surroundings Height Status … Tower: Visiting Location When built Design Purpose Height …
SLIDE 9
Do entity type models exist?
How can we derive entity type models?
Do entity type models help to select relevant sentences from the documents?
Do entity type models help to reduce redundancy and lead to more coherent summaries?
─
Manual approach
─
Automatic approach
Further research questions
SLIDE 10
Can we derive entity type models from existing text resources such as Wikipedia articles?
Height, Year, Location, Designer, Entrance Fee
Wikipedia articles
SLIDE 11
Investigation Results show:
Height, Year, Location, Designer, Entrance Fee Height, Year, Location, Designer, Entrance Fee
Attributes humans associate with entity types are also found in Wikipedia articles Entity type models can be derived from existing text resources
?
Wikipedia articles
SLIDE 12
For each Wikipedia article Extract entity type Wikipedia Add article to corpus Entity Type Corpus Collection
Entity Type 1 Entity Type 2 Entity Type 3
- 107 different entity types are
extracted: village (40k), school (15k), mountain (5k), church (3k), lake (3k), etc.
- Accuracy: 90%
(Aker & Gaizauskas 2009)
SLIDE 13
Signature words
N-gram language models
Dependency patterns
How can we represent the entity type models
SLIDE 14
Signature words
N-gram language models
Dependency patterns
is located 200 is constructed 150 is located 200=0.1 is constructed 150=0.08 [Entity] is [entityType] 400 was built [date] 300 [Entity] has design 200
Church corpus How can we represent the entity type models
SLIDE 15
Do entity type models exist?
How can we derive entity type models?
Do entity type models help to select relevant sentences from the documents?
Do entity type models help to reduce redundancy and lead to more coherent summaries?
─
Manual approach
─
Automatic approach
Further research questions
SLIDE 16
The-MDS:
Sentence splitting Tokenizing POS tagging Lemmatizing NE tagging
WEB SEARCH Eiffel Tower, Paris, France
The Eiffel Tower (French: Tour Eiffel, [tuʁ ɛfɛl], nickname La dame de fer, the iron woman) is an 1889 iron lattice tower located on the Champ de Mars in Paris that…
Sentence position Centroid similarity Query similarity Starter similarity Entity Type Model
Web Documents
Scoring sentences Sorting sentences
The-MDS:
Selecting sentences From the sorted list Redundancy reduction
Preprocessing Feature Extraction Sentence Scoring Sentence Selection
Summary generation process
SLIDE 17
Entity type model feature
Signature words model feature N-gram language model feature Dependency pattern model feature
SLIDE 18
Experiments
SLIDE 19
Evaluation settings – image set
Image collection contains 310 images from sites worldwide (Aker & Gaizauskas 2010a)
Training 205 Testing 105
Eiffel Tower, Paris, France
10 web-documents
The Eiffel Tower (French: Tour Eiffel, [tuʁ ɛfɛl], nickname La dame de fer, the iron woman) is an 1889 iron lattice tower located on the Champ de Mars in Paris that… The Eiffel Tower (French: Tour Eiffel, [tuʁ ɛfɛl], nickname La dame de fer, the iron woman) is an 1889 iron lattice tower located on the Champ de Mars in Paris that… The Eiffel Tower (French: Tour Eiffel, [tuʁ ɛfɛl], nickname La dame de fer, the iron woman) is an 1889 iron lattice tower located on the Champ de Mars in Paris that… The Eiffel Tower (French: Tour Eiffel, [tuʁ ɛfɛl], nickname La dame de fer, the iron woman) is an 1889 iron lattice tower located on the Champ de Mars in Paris that…
SLIDE 20
We use ROUGE (Lin, 2004) to evaluate our image captions
automatically
─ Need model captions
We use model captions described in Aker & Gaizauskas (2010a) For comparison two baselines are generated:
─ From the top retrieved web-document (FirstDoc) ─ From the Wikipedia article (Wiki) – upper bound
Evaluation settings – ROUGE evaluation Training 205 Testing 105 model summaries
SLIDE 21
We also evaluated our summaries using a readability assessment
as in DUC
Five criteria approach: grammaticality, redundancy, clarity, focus
and coherence
Each criterion is scored on a five point scale with high scores
indicating a better result
We asked four humans to perform this task
Evaluation settings – Manual evaluation
SLIDE 22
Experimental results – ROUGE evaluation
Entity type models help to achieve better results
─
However, this is not the case for signature words model The representation method is also relevant
DpMSim captions are significantly better than all other automated captions (except LMSim captions)
Only moderate improvement between DpMSim and LMSim. Same with Wiki baseline captions (in RSU4)
FirstDoc Wiki centroidSim sentencePos querySim starterSim SigSim LMSim DpMSim
R2 .042 .097 .0734 .066 .0774 .0869 .079 .0895 .093 RSU4 .079 .14 .12 .11 .12 .137 .133 .142 .145
SLIDE 23
Experimental results – ROUGE evaluation
We performed different feature combinations
Best performing feature combination is starterSim + LMSim (=> DpMSim)
starterSim + LMSim DpMSim Wiki
R2 .095 .093 .097 RSU4 .145 .145 .14
SLIDE 24
Experimental results – Manual evaluation
Table shows scores for level 5 and 4
Each score has to be read as “X% of the summaries were judged with at least 4 for the criterion Y”
There is a lot of room to improve
starterSim + LMSim Wiki
Clarity 80% 94.3% Focus 75% 92.6% coherence 70% 90.7% redundancy 60% 91.5% grammar 84% 81.6%
SLIDE 25
Do entity type models exist?
How can we derive entity type models?
Do entity type models help to select relevant sentences from the documents?
Do entity type models help to reduce redundancy and lead to more coherent summaries?
─
Manual approach
─
Automatic approach
Further research questions
SLIDE 26
We also manually categorized the dependency patterns and use
them for redundancy reduction and sentence ordering (DepCat feature)
type year location background surrounding visiting
SLIDE 27
Experiments
SLIDE 28
Experimental results – ROUGE evaluation
We performed different feature combinations
Best performing feature combination is starterSim + LMSim (> DpMSim)
To the best performing feature combination we added the DepCat feature Both R2 and RSU4 results are significantly better than Wikipedia baseline captions
starterSim + LMSim starterSim + LMSim + DepCat Wiki
R2 .095 .102 .097 RSU4 .145 .155 .14
SLIDE 29
Experimental results – Manual evaluation
Table shows scores for level 5 and 4
Each score has to be read as “X% of the summaries were judged with at least 4 for the criterion Y”
Adding DepCat (for sentence ordering) helps to improve readability
In all criteria starterSim + LMSim + DepCat summaries obtain better results than starterSim + LMSim
starterSim + LMSim starterSim + LMSim + DepCat Wiki
Clarity 80% 85% 94.3% Focus 75% 76.4% 92.6% coherence 70% 74% 90.7% redundancy 60% 83% 91.5% grammar 84% 92% 81.6%
SLIDE 30
Replacing DepCat with AutoDepCat
Do entity type models help to reduce redundancy and lead to more coherent summaries?
─
Manual approach
─
Automatic approach
SLIDE 31
No question to which “bin” or “category” a sentence belongs to and
where in the summary the sentence to include
Inclusion of the sentence is automatically derived based on the final
sentence added to the summary
AutoDepCat
SLIDE 32
Summarization as search
SLIDE 33
1 5 2 3 4 2,3 2,7 2,4 2,5 2,6
…
State 1 L = l (S1) S = s (S1) H = s (S1) + s (S2), …, + s (Sn)
Summary
1,2 1,6 1,7 1,3 1,4 1,5 1,8
1,2,3 1,2,4 1,2,5 Start State: Summary Length L = 0 Summary Score S = 0 Heuristic Score H = 0 Goal State
AutoDepCat is applied
Summarization as search – AutoDepCat is applied
SLIDE 34
A flow model generated using the dependency patterns are used to
compute the flow between sentences
From a corpus every two adjacent sentences are processed For each sentence dependency patterns are generated Each pattern from the first sentence is paired with every pattern in
the other sentence
Frequency counts of the pairs are recorded over the entire corpus
Dependency pattern flow model
@@ entity is entityType 400=0.5 entity is entityType @@ was built date 300=0.3 was built date @@ is feet long 200=0.1 is feet long @@ entity has design 100=0.07
…
SLIDE 35
Flow probability between the candidate sentence and the last
sentence in the summary is computed
Also the flow probability between the next sentence (sentence after
the candidate sentence) and the last sentence in the summary is computed
If the flow probability of the candidate sentence is smaller than the
next sentence, then the candidate sentence is jumped over
Summarization as search – AutoDepCat is applied
SLIDE 36
Experiments
SLIDE 37
Experimental results – ROUGE evaluation
Best ROUGE scores are obtained with AutoDepCat AutoDepCat avoids DepCat but also significantly improves summary quality measured by ROUGE
starterSim + LMSim + DepCat A* search without AutoDepCat A* search with AutoDepCat
R2 .102 .094 .111 RSU4 .155 .146 .167
SLIDE 38
Experimental results – Manual evaluation
Best results are achieved with AutoDepCat summaries
Results are better than the manual approach DepCat
Inclusion of AutoDepCat improves summary quality measured by human evaluation
Idea of dependency entity type modelling can be ported to any other domain where entities can be grouped into entity types
starterSim + LMSim + DepCat Wiki A* search without AutoDepCat A* search with AutoDepCat
Clarity 85% 94.3% 47.9 94 Focus 76.4% 92.6% 39.5 88 coherence 74% 90.7% 25 82 redundancy 83% 91.5% 12.5 92 grammar 92% 81.6% 30.2 80
SLIDE 39
Do entity type models exist?
- Yes. We showed that humans associate sets of attributes with
entity types.
How can we derive entity type models?
Showed that entity types can be derived from Wikipedia articles.
Do entity type models help to select relevant sentences from the documents?
- Yes. However, also the representation is relevant.
Do entity type models help to reduce redundancy and lead to more coherent summaries?
- Yes. We showed a manual and an automated approach.
Conclusion
SLIDE 40
Papers
Aker, A. Plaza, L., Lloret, E., (2013): Do humans have conceptual models about Geographic Objects? A user study. Journal of the American Society for Information Science and Technology, 64 (4), pp. 689–700. Aker, A., Cohn, T., Gaizauskas, R. (2012), Redundancy reduction for multi-document summaries using A* search and discriminative training, in ‘Proceedings of the 2nd International Workshop on Exploiting Large Knowledge Repositories, in conjunction with the 1st International Workshop on Automatic Text Summarization for the Future (ATSF-2012)’, Universitat Jaume I, Spain, pp. 58–68. Aker, A. Plaza, L., Lloret, E., Gaizauskas, R. (2012). Multi-document Summarization Techniques for Generating Image Descriptions: A Comparative Analysis. In Poibeau, T., Saggion, H., Piskorski, J., Yangarber, R. (Eds), Multi-source, Multilingual Information Extraction and
- Summarization. pp. 299-320. Springer.
Aker, A., Gaizauskas, R. (2011), Understanding the types of information humans associate with geographic objects, in Proceedings of the 20th ACM International Conference on Information and Knowledge Management (CIKM), pp. 1929-1932. Aker, A., Gaizauskas, R. (2010), Generating image descriptions using dependency relational patterns, in Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1250-1258. Aker, A., Cohn, T., Gaizauskas, R. (2010), Multi-document summarization using A* search and discriminative training, in Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, pp. 482- 491 Aker, A., Gaizauskas, R. (2010a), Model summaries for location-related images, in ‘Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC’10)’, European Language Resources Association (ELRA), Valletta, Malta, pp. 3119–3124. Aker, A., Gaizauskas, R. (2009), Summary generation for toponym-referenced images using object type language models, Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP) pp. 6-11 Aker, A., Gaizauskas, R. (2008), Evaluating automatically generated user-focused multi-document summaries for geo-referenced images, in Proceedings of the Workshop on Multi-source Multilingual Information Extraction and Summarization (MMIES), pp. 41-48
SLIDE 41
Questions? Thank you for your attention! Questions
SLIDE 42
To derive a dependency relational pattern model for a specific entity
type we use all its Wikipedia articles
For each article we split the text in sentences using OpenNLP
(sentence boundaries are identified)
For each sentence (using OpenNLP) we perform some pre-
processing steps:
─ we identify named entities (NEs) ─ we replace the term denoting the entity type with the string “entityType” ─ we identify the verbs (based on POS tags)
We apply the Stanford Parser to obtain dependency parse from the
pre-processed sentence
Finally from the dependency parse we derive our relational patterns
(inspired by Sudo et al. (2001) )
Entity type modelling – dependency patterns
SLIDE 43