From unsupervised induction of linguistic structures from text - PowerPoint PPT Presentation

Inducing word sense representations May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 16/109

Inducing word sense representations Word vs sense embeddings May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 17/109

Inducing word sense representations Word vs sense embeddings May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 18/109

Inducing word sense representations Related work May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 19/109

Inducing word sense representations Related work: knowledge-based AutoExtend [Rothe & Schütze, 2015] * image is reproduced from the original paper May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 20/109

-- a hidden variable: a sense index of word in context ; -- a meta-parameter controlling number of senses. See also : [Neelakantan et al., 2014] and [Li and Jurafsky, 2015] Inducing word sense representations Related work: knowledge-free Adagram [Bartunov et al., 2016] Multiple vector representations θ for each word: May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 21/109

See also : [Neelakantan et al., 2014] and [Li and Jurafsky, 2015] Inducing word sense representations Related work: knowledge-free Adagram [Bartunov et al., 2016] Multiple vector representations θ for each word: V N C ∞ ∏ ∏ ∏ ∏ p ( Y, Z, β | X, α, θ ) = p ( β wk | α ) [ p ( z i | x i , β ) p ( y ij | z i , x i , θ w =1 k =1 i =1 j =1 z i -- a hidden variable: a sense index of word x i in context C ; α -- a meta-parameter controlling number of senses. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 21/109

Inducing word sense representations Related work: knowledge-free Adagram [Bartunov et al., 2016] Multiple vector representations θ for each word: V N C ∞ ∏ ∏ ∏ ∏ p ( Y, Z, β | X, α, θ ) = p ( β wk | α ) [ p ( z i | x i , β ) p ( y ij | z i , x i , θ w =1 k =1 i =1 j =1 z i -- a hidden variable: a sense index of word x i in context C ; α -- a meta-parameter controlling number of senses. See also : [Neelakantan et al., 2014] and [Li and Jurafsky, 2015] May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 21/109

Inducing word sense representations Related work: word sense induction Word sense induction (WSI) based on graph clustering : [Lin, 1998] [Pantel and Lin, 2002] [Widdows and Dorow, 2002] Chinese Whispers [Biemann, 2006] [Hope and Keller, 2013] May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 22/109

Inducing word sense representations Related work: Chinese Whispers#1 * source of the image: http://ic.pics.livejournal.com/blagin_anton/33716210/2701748/2701748_800.jpg May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 23/109

Vector formulation [Biemann, 2006] Inducing word sense representations Related work: Chinese Whispers#2 Iterative formulation [Biemann, 2006] May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 24/109

Inducing word sense representations Related work: Chinese Whispers#2 Iterative formulation [Biemann, 2006] Vector formulation [Biemann, 2006] May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 24/109

Inducing word sense representations Related work: Chinese Whispers#2 May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 25/109

Inducing word sense representations Related work: Chinese Whispers#2 May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 26/109

Inducing word sense representations Related work: Chinese Whispers#2 May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 27/109

Inducing word sense representations Sense embeddings using retrofjtting RepL4NLP@ACL'16 [Pelevina et al., 2016], LREC'18 [Remus & Biemann, 2018] Prior methods: Induce inventory by clustering of word instances Use existing sense inventories Our method: Input: word embeddings Output: word sense embeddings Word sense induction by clustering of word ego-networks May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 28/109

Inducing word sense representations Sense embeddings using retrofjtting From word embeddings to sense embeddings 1 2 Calculate Word Learning Word Vectors Similarity Graph Word Vectors Text Corpus Word Similarity Graph 3 4 Pooling of Word Vectors Word Sense Induction Sense Inventory Sense Vectors May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 29/109

Inducing word sense representations Sense embeddings using retrofjtting Word sense induction using ego-network clustering May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 30/109

table#0 leftmost#0, column#1, tableau#1, indent#1, bracket#3, pointer#0, footer#1, cursor#1, diagram#0, grid#0 table#1 pile#1, stool#1, tray#0, basket#0, bowl#1, bucket#0, box#0, cage#0, saucer#3, mirror#1, pan#1, lid#0 Inducing word sense representations Sense embeddings using retrofjtting Neighbours of Word and Sense Vectors Vector Nearest Neighbors table tray, bottom, diagram, bucket, brackets, stack, basket, list, parenthesis, cup, saucer, pile, playfjeld, bracket, pot, drop-down, cue, plate May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 31/109

Inducing word sense representations Sense embeddings using retrofjtting Neighbours of Word and Sense Vectors Vector Nearest Neighbors table tray, bottom, diagram, bucket, brackets, stack, basket, list, parenthesis, cup, saucer, pile, playfjeld, bracket, pot, drop-down, cue, plate table#0 leftmost#0, column#1, tableau#1, indent#1, bracket#3, pointer#0, footer#1, cursor#1, diagram#0, grid#0 table#1 pile#1, stool#1, tray#0, basket#0, bowl#1, bucket#0, box#0, cage#0, saucer#3, mirror#1, pan#1, lid#0 May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 31/109

Inducing word sense representations Sense embeddings using retrofjtting Word and sense embeddings of words iron and vitamin . LREC'18 [Remus & Biemann, 2018] May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 32/109

Inducing word sense representations Sense embeddings using retrofjtting Word Sense Disambiguation 1 Context extraction : use context words around the target word 2 Context fjltering : based on context word's relevance for disambiguation 3 Sense choice in context : maximise similarity between a context vector and a sense vector May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 33/109

Inducing word sense representations Sense embeddings using retrofjtting May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 34/109

Inducing word sense representations Sense embeddings using retrofjtting May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 35/109

Inducing word sense representations Sense embeddings using retrofjtting May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 36/109

Inducing word sense representations Sense embeddings using retrofjtting May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 37/109

Inducing word sense representations Sense embeddings using retrofjtting Unsupervised WSD SemEval'13, ReprL4NLP [Pelevina et al., 2016]: Model Jacc. Tau WNDCG F.NMI F.B-Cubed AI-KU (add1000) 0.176 0.609 0.205 0.033 0.317 AI-KU 0.176 0.619 0.393 0.066 0.382 AI-KU (remove5-add1000) 0.228 0.654 0.330 0.040 0.463 Unimelb (5p) 0.198 0.623 0.374 0.056 0.475 Unimelb (50k) 0.198 0.633 0.384 0.060 0.494 UoS (#WN senses) 0.171 0.600 0.298 0.046 0.186 UoS (top-3) 0.220 0.637 0.370 0.044 0.451 La Sapienza (1) 0.131 0.544 0.332 -- -- La Sapienza (2) 0.131 0.535 0.394 -- -- AdaGram, α = 0.05, 100 dim 0.274 0.644 0.318 0.058 0.470 w2v 0.197 0.615 0.291 0.011 0.615 w2v (nouns) 0.179 0.626 0.304 0.011 0.623 JBT 0.205 0.624 0.291 0.017 0.598 JBT (nouns) 0.198 0.643 0.310 0.031 0.595 TWSI (nouns) 0.215 0.651 0.318 0.030 0.573 May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 38/109

Inducing word sense representations Sense embeddings using retrofjtting Semantic relatedness , LREC'2018 [Remus & Biemann, 2018]: d L S n m e m t a a w x t r l r a g e e o a o g p S v h a b a m t N o r A A u d G a l y S S a a g S s L L p . . . . . . SimLex999 0 . 45 0 . 29 0 . 44 0 . 37 0 . 54 0 . 30 0 . 27 0 . 68 MEN 0 . 72 0 . 67 0 . 77 0 . 73 0 . 53 0 . 67 0 . 71 0 . 77 SimVerb 0 . 43 0 . 27 0 . 36 0 . 23 0 . 37 0 . 15 0 . 19 0 . 53 WordSim353 0 . 58 0 . 61 0 . 70 0 . 61 0 . 47 0 . 67 0 . 59 0 . 72 SimLex999-N 0 . 44 0 . 33 0 . 45 0 . 39 0 . 48 0 . 32 0 . 34 0 . 68 MEN-N ____ ____ ____ ____ ____ ____ 0 . 72 0 . 68 0 . 77 0 . 76 0 . 57 0 . 71 0 . 73 0 . 78 May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 39/109

Inducing word sense representations Sense embeddings using retrofjtting Unsupervised WSD SemEval'13, ReprL4NLP [Pelevina et al., 2016]: comparable to SOTA, incl. sense embeddings. Semantic relatedness , LREC'2018 [Remus & Biemann, 2018]: s e s n s e s e s s e s + s e s s e n s d n L L e s n S S e n n e s m m s e e m n s e + + s t a a e s + a w w x t t r r s + l l a a e r + g g e e o o a a o g p p a a S S v v h h b b a N N m m t o o r r A A A A u d G G a a l l y y S S S S a a g g p p S S s s L L L L SimLex999 0 . 45 0 . 29 0 . 44 0 . 46 0 . 37 0 . 41 0 . 54 0 . 55 0 . 30 0 . 39 0 . 27 0 . 38 0 . 68 0 . 64 MEN 0 . 72 0 . 67 0 . 77 0 . 78 0 . 73 0 . 77 0 . 53 0 . 68 0 . 67 0 . 70 0 . 71 0 . 74 0 . 77 0 . 80 SimVerb 0 . 43 0 . 27 0 . 36 0 . 39 0 . 23 0 . 30 0 . 37 0 . 45 0 . 15 0 . 22 0 . 19 0 . 28 0 . 53 0 . 53 WordSim353 0 . 58 0 . 61 0 . 70 0 . 69 0 . 61 0 . 65 0 . 47 0 . 62 0 . 67 0 . 66 0 . 59 0 . 63 0 . 72 0 . 73 SimLex999-N 0 . 44 0 . 33 0 . 45 0 . 50 0 . 39 0 . 47 0 . 48 0 . 55 0 . 32 0 . 46 0 . 34 0 . 44 0 . 68 0 . 66 MEN-N 0 . 72 0 . 68 0 . 77 0 . 79 0 . 76 0 . 80 0 . 57 0 . 74 0 . 71 0 . 73 0 . 73 0 . 76 0 . 78 0 . 81 May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 40/109

Inducing word sense representations Synset induction ACL'17 [Ustalov et al., 2017b] Examples of extracted synsets: Size Synset 2 { decimal point , dot } 3 { gullet , throat , food pipe } 4 { microwave meal , ready meal , TV dinner , frozen dinner } 5 { objective case , accusative case , oblique case , object case , accusative } 6 { radiotheater , dramatized audiobook , audio theater , ra- dio play , radio drama , audio play } May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 41/109

Inducing word sense representations Synset induction Outline of the 'Watset' method: Learning Local-Global Fuzzy Graph Clustering Word Embeddings Word Ambiguous Disambiguated Background Corpus Sense Inventory Similarities Weighted Graph Weighted Graph Local Clustering: Disambiguation of Global Clustering: Graph Construction Word Sense Induction Neighbors Synset Induction Synonymy Dictionary Synsets May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 42/109

Inducing word sense representations Synset induction May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 43/109

Inducing word sense representations Synset induction May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 44/109

Inducing word sense representations Synset induction May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 45/109

Inducing word sense representations Synset induction 0.3 0.3 F−score F−score 0.2 0.2 0.1 0.1 0.0 0.0 CW MCL MaxMax ECO CPM Watset CW MCL MaxMax ECO CPM Watset WordNet (English) BabelNet (English) May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 46/109

Inducing word sense representations Synset induction 0.3 0.3 F−score F−score 0.2 0.2 0.1 0.1 0.0 0.0 CW MCL MaxMax ECO CPM Watset CW MCL MaxMax ECO CPM Watset WordNet (English) BabelNet (English) 0.20 0.4 0.15 0.3 F−score F−score 0.10 0.2 0.05 0.1 0.00 0.0 CW MCL MaxMax ECO CPM Watset CW MCL MaxMax ECO CPM Watset RuWordNet (Russian) YARN (Russian) May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 46/109

Inducing word sense representations Sample of induced sense inventory Word Sense Local Sense Cluster: Related Senses Hypernyms mango#0 peach#1, grape#0, plum#0, apple#0, apricot#0, fruit#0, food#0, … watermelon#1, banana#1, coconut#0, pear#0, fjg#0, melon#0, mangosteen#0 , … apple#0 mango#0, pineapple#0, banana#1, melon#0, fruit#0, crop#0, … grape#0, peach#1, watermelon#1, apricot#0, cranberry#0, pumpkin#0, mangosteen#0 , … Java#1 C#4, Python#3, Apache#3, Ruby#6, Flash#1, programming C++#0, SQL#0, ASP#2, Visual Basic#1, CSS#0, language#3, lan- Delphi#2, MySQL#0, Excel#0, Pascal#0, … guage#0, … Python#3 PHP#0, Pascal#0, Java#1, SQL#0, Visual Ba- language#0, tech- sic#1, C++#0, JavaScript#0, Apache#3, Haskell#5, nology#0, … .NET#1, C#4, SQL Server#0, … May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 47/109

Inducing word sense representations Sample of induced semantic classes ID Global Sense Cluster: Semantic Class Hypernyms 1 peach#1, banana#1, pineapple#0, berry#0, black- vegetable#0, fruit#0, berry#0, grapefruit#0, strawberry#0, blueberry#0, crop#0, ingredi- mango#0, grape#0, melon#0, orange#0, pear#0, ent#0, food#0, · plum#0, raspberry#0, watermelon#0, apple#0, apri- cot#0, watermelon#0, pumpkin#0, berry#0, man- gosteen#0 , … 2 C#4, Basic#2, Haskell#5, Flash#1, Java#1, Pas- programming lan- cal#0, Ruby#6, PHP#0, Ada#1, Oracle#3, Python#3, guage#3, technol- Apache#3, Visual Basic#1, ASP#2, Delphi#2, SQL ogy#0, language#0, Server#0, CSS#0, AJAX#0, JavaScript#0, SQL format#2, app#0 Server#0, Apache#3, Delphi#2, Haskell#5, .NET#1, CSS#0, … May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 48/109

Inducing word sense representations Induction of semantic classes Induction of Semantic Classes Induced Word Senses Sense Ego-Networks Global Sense Graph Word Sense Induction Representing Senses Sense Graph Clustering of with Ego Networks from Text Corpus Construction Word Senes Global Sense Clusters s Noisy Hypernyms Labeling Sense Clusters with Hypernyms Semantic Classes Text Corpus Cleansed Hypernyms May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 49/109

Inducing word sense representations Induction of sense semantic classes Filtering noisy hypernyms with semantic classes LREC'18 [Panchenko et al., 2018b]: Hypernyms, city#2 fruit#1 food#0 Added Removed Missing Wrong pear#0 mangosteen#0 apple#2 mango#0 Sense Cluster, May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 50/109

http://panchenko.me/data/joint/nodes20000-layers7 Inducing word sense representations Global sense clustering May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 51/109

Inducing word sense representations Global sense clustering May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 52/109

Inducing word sense representations Induction of sense semantic classes Filtering of a noisy hypernymy database with semantic classes. LREC'18 [Panchenko et al., 2018b] Precision Recall F-score Original Hypernyms (Seitner et al., 2016) 0 . 475 0 . 546 0 . 508 Semantic Classes (coarse-grained) 0 . 541 0 . 679 0 . 602 May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 53/109

Making induced senses interpretable May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 54/109

Making induced senses interpretable Making induced senses interpretable Knowledge-based sense representations are interpretable May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 55/109

Making induced senses interpretable Making induced senses interpretable Most knowledge-free sense representations are uninterpretable May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 56/109

Making induced senses interpretable Making induced senses interpretable May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 57/109

Making induced senses interpretable Making induced senses interpretable Hypernymy prediction in context. EMNLP'17 [Panchenko et al., 2017b] May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 58/109

Super Senses Random 0.001 0.001 Super Senses MFS 0.001 0.001 Super Senses Cluster Words 0.174 0.365 Super Senses Context Words 0.086 0.188 Making induced senses interpretable Making induced senses interpretable 11.702 sentences , 863 words with avg.polysemy of 3.1 . WSD Model Accuracy Inventory Features Hypers HyperHypers Word Senses Random 0.257 0.610 Word Senses MFS 0.292 0.682 Word Senses Cluster Words 0.291 0.650 Word Senses Context Words 0.308 0.686 May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 59/109

Making induced senses interpretable Making induced senses interpretable 11.702 sentences , 863 words with avg.polysemy of 3.1 . WSD Model Accuracy Inventory Features Hypers HyperHypers Word Senses Random 0.257 0.610 Word Senses MFS 0.292 0.682 Word Senses Cluster Words 0.291 0.650 Word Senses Context Words 0.308 0.686 Super Senses Random 0.001 0.001 Super Senses MFS 0.001 0.001 Super Senses Cluster Words 0.174 0.365 Super Senses Context Words 0.086 0.188 May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 59/109

Linking induced senses to resources May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 60/109

Linking induced senses to resources Linking induced senses to resources Construction of Proto-Conceptualization (PCZ) Graph of Related Words Word Sense Inventory Labeled Word Senses Graph of Related Senses Word Sense Labeling Senses Induce a Graph of Disambiguation Construction of sense with Hypernyms Sem. Related Words Induction of Neighbours feature representations PCZ Linking Proto-Conceptualization to Lexical Resource Part. Linked Senses to the LR Typing of the Unmapped Linking Induced Senses Text Corpus Induced Senses to Senses of the LR Enriched Lexical Resource (LR): Lexical Resource WordNet, BabelNet, ... LREC'16 [Panchenko, 2016], ISWC'16 [Faralli et al., 2016], SENSE@EACL'17 [Panchenko et al., 2017a], NLE'18 [Biemann et al., 2018] May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 61/109

Linking induced senses to resources Linking induced senses to resources Word AdaGram BabelNet AdaGram BoW BabelNet BoW python 2 bn:01713224n perl, php, java, smalltalk, ruby, language, programming, python- lua, tcl, scripting, javascript, ista, python programming, bindings, binding, programming, python3, python2, level, com- coldfusion, actionscript, net, . . . puter, pythonistas, python3000, python 1 bn:01157670n monty, circus, spamalot, python, monty, comedy, monty python, magoo, muppet, snoopy, fea- british, monte, monte python, turette, disney, tunes, tune, clas- troupe, pythonesque, foot, artist, sic, shorts, short, apocalypse, . . . record, surreal, terry, . . . python 3 bn:00046456n spectacled, unicornis, snake, gi- molurus, indian, boa, tigris, ant, caiman, leopard, squirrel, tiger python, rock, tiger, indian crocodile, horned, cat, mole, ele- python, reptile, python molurus, phant, opossum, pheasant, . . . indian rock python, coluber, . . . python 4 bn:01157670n circus, fmy, fmying, dusk, lizard, monty, comedy, monty python, moth, unicorn, pufg, adder, vul- british, monte, monte python, ture, tyrannosaurus, zephyr, bad- troupe, pythonesque, foot, artist, ger, . . . record, surreal, terry, . . . python 1 bn:00473212n monty, circus, spamalot, python, pictures, monty, python monty magoo, muppet, snoopy, fea- pictures, limited, company, turette, disney, tunes, tune, clas- python pictures limited, king- sic, shorts, short, apocalypse, . . . dom, picture, serve, director, . . . python 1 bn:03489893n monty, circus, spamalot, python, fjlm, horror, movie, clabaugh, magoo, muppet, snoopy, fea- richard, monster, century, direct, turette, disney, tunes, tune, clas- snake, python movie, television, sic, shorts, short, apocalypse, . . . giant, natural, language, for-tv, . . . May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 62/109

Linking induced senses to resources Linking induced senses to resources Model Representation of the Sense "disk (medium)" WordNet memory, device, fmoppy, disk, hard, disk, disk, computer, science, computing, diskette, fjxed, disk, fmoppy, magnetic, disc, magnetic, disk, hard, disc, storage, device WordNet + Linked recorder, disk, fmoppy, console, diskette, handset, desktop, iPhone, iPod, HDTV, kit, RAM, Discs, Blu- ray, computer, GB, microchip, site, cartridge, printer, tv, VCR, Disc, player, LCD, software, component, camcorder, cellphone, card, monitor, display, burner, Web, stereo, internet, model, iTunes, turntable, chip, cable, camera, iphone, notebook, device, server, surface, wafer, page, drive, laptop, screen, pc, television, hardware, YouTube, dvr, DVD, product, folder, VCR, radio, phone, circuitry, partition, megabyte, peripheral, format, machine, tuner, website, merchandise, equipment, gb, discs, MP3, hard-drive, piece, video, storage device, memory device, microphone, hd, EP, content, soundtrack, webcam, system, blade, graphic, microprocessor, collection, document, programming, battery, key- board, HD, handheld, CDs, reel, web, material, hard-disk, ep, chart, debut, confjguration, recording, album, broadcast, download, fjxed disk, planet, pda, microfjlm, iPod, videotape, text, cylinder, cpu, canvas, label, sampler, workstation, electrode, magnetic disc, catheter, magnetic disk, Video, mo- bile, cd, song, modem, mouse, tube, set, ipad, signal, substrate, vinyl, music, clip, pad, audio, com- pilation, memory, message, reissue, ram, CD, subsystem, hdd, touchscreen, electronics, demo, shell, sensor, fjle, shelf, processor, cassette, extra, mainframe, motherboard, fmoppy disk, lp, tape, version, kilobyte, pacemaker, browser, Playstation, pager, module, cache, DVD, movie, Windows, cd-rom, e- book, valve, directory, harddrive, smartphone, audiotape, technology, hard disk, show, computing, computer science, Blu-Ray, blu-ray, HDD, HD-DVD, scanner, hard disc, gadget, booklet, copier, play- back, TiVo, controller, fjlter, DVDs, gigabyte, paper, mp3, CPU, dvd-r, pipe, cd-r, playlist, slot, VHS, fjlm, videocassette, interface, adapter, database, manual, book, channel, changer, storage May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 63/109

Linking induced senses to resources Linking induced senses to resources Evaluation of linking accuracy: May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 64/109

Linking induced senses to resources Linking induced senses to resources Evaluation of enriched representations based on WSD: May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 65/109

Shared task on word sense induction May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 66/109

Shared task on word sense induction A shared task on WSI An ACL SIGSLAV sponsored shared task on word sense induction (WSI) for the Russian language. More details : https://russe.nlpub.org/2018/wsi May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 67/109

Contexts where the word occurs, e.g.: ``river bank is a slope beside a body of water'' `` bank is a fjnancial institution that accepts deposits'' ``Oh, the bank was robbed. They took about a million dollars.'' `` bank of Elbe is a good and popular hangout spot complete with good food and fun'' You need to group the contexts by senses : ``river bank is a slope beside a body of water'' `` bank of Elbe is a good and popular hangout spot complete with good food and fun'' `` bank is a fjnancial institution that accepts deposits'' ``Oh, the bank was robbed. They took about a million dollars.'' Shared task on word sense induction A lexical sample WSI task Target word , e.g. ``bank''. May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 68/109

You need to group the contexts by senses : ``river bank is a slope beside a body of water'' `` bank of Elbe is a good and popular hangout spot complete with good food and fun'' `` bank is a fjnancial institution that accepts deposits'' ``Oh, the bank was robbed. They took about a million dollars.'' Shared task on word sense induction A lexical sample WSI task Target word , e.g. ``bank''. Contexts where the word occurs, e.g.: ``river bank is a slope beside a body of water'' `` bank is a fjnancial institution that accepts deposits'' ``Oh, the bank was robbed. They took about a million dollars.'' `` bank of Elbe is a good and popular hangout spot complete with good food and fun'' May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 68/109

Shared task on word sense induction A lexical sample WSI task Target word , e.g. ``bank''. Contexts where the word occurs, e.g.: ``river bank is a slope beside a body of water'' `` bank is a fjnancial institution that accepts deposits'' ``Oh, the bank was robbed. They took about a million dollars.'' `` bank of Elbe is a good and popular hangout spot complete with good food and fun'' You need to group the contexts by senses : ``river bank is a slope beside a body of water'' `` bank of Elbe is a good and popular hangout spot complete with good food and fun'' `` bank is a fjnancial institution that accepts deposits'' ``Oh, the bank was robbed. They took about a million dollars.'' May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 68/109

Shared task on word sense induction Dataset based on Wikipedia May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 69/109

Shared task on word sense induction Dataset based on RNC May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 70/109

Shared task on word sense induction Dataset based on dictionary glosses May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 71/109

Shared task on word sense induction A sample from the wiki-wiki dataset May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 72/109

Shared task on word sense induction A sample from the wiki-wiki dataset May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 73/109

Shared task on word sense induction A sample from the wiki-wiki dataset May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 74/109

Shared task on word sense induction A sample from the bts-rnc dataset May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 75/109

Shared task on word sense induction A sample from the active-dict dataset May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 76/109

Shared task on word sense induction jamsic : sense induction 1 Get the neighbors of a target word, e.g. ``bank'' : lender 1 2 river 3 citybank 4 slope … 5 2 Get similar to ``bank'' and dissimilar to ``lender'' : river 1 2 slope 3 land 4 … 3 Compute distances to ``lender'' and ``river'' . May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 77/109

Induction of semantic frames May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 78/109

Induction of semantic frames FrameNet: frame ``Kidnapping'' May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 79/109

Induction of semantic frames Frame induction as a triclustering ACL'2018 [Ustalov et al., 2018a] Example of a LU tricluster corresponding to the ``Kidnapping'' frame from FrameNet. FrameNet Role Lexical Units (LU) Perpetrator Subject kidnapper, alien, militant FEE Verb snatch, kidnap, abduct Victim Object son, people, soldier, child May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 80/109

Induction of semantic frames SVO triple elements May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 81/109

Induction of semantic frames An SVO triple graph Mayor|lead|city Mayor|lead|city Governor|lead|state Governor|lead|state General|command|Department General|command|Department mayor|lead|city mayor|lead|city president|lead|state president|lead|state President|lead|party President|lead|party President|chair|committee President|chair|committee Chief|lead|department Chief|lead|department General|command|department General|command|department President|lead|company President|lead|company president|lead|government president|lead|government chairman|lead|company chairman|lead|company General|head|Department General|head|Department chief|lead|department chief|lead|department President|chair|Committee President|chair|Committee president|lead|department president|lead|department Chairman|lead|company Chairman|lead|company minister|lead|team minister|lead|team Director|lead|agency Director|lead|agency of fi cer|head|department of fi cer|head|department director|lead|department director|lead|department King|run|company King|run|company General|head|department General|head|department Chairman|lead|Committee Chairman|lead|Committee Director|lead|company Director|lead|company Minister|head|government Minister|head|government Director|head|Department Director|head|Department Director|lead|Department Director|lead|Department minister|head|department minister|head|department Director|lead|department Director|lead|department chairman|lead|committee chairman|lead|committee of fi cer|lead|company of fi cer|lead|company president|chair|committee president|chair|committee leader|head|department leader|head|department director|lead|company director|lead|company president|head|government president|head|government director|head|department director|head|department president|chair|Committee president|chair|Committee leader|head|government leader|head|government leader|head|party leader|head|party boss|lead|company boss|lead|company director|chair|committee director|chair|committee Chairman|chair|Committee Chairman|chair|Committee of fi cer|head|team of fi cer|head|team Chairman|chair|committee Chairman|chair|committee Director|chair|committee Director|chair|committee Minister|chair|committee Minister|chair|committee leader|head|agency leader|head|agency Director|chair|Committee Director|chair|Committee Chairman|run|committee Chairman|run|committee leader|head|team leader|head|team President|head|team President|head|team chairman|head|committee chairman|head|committee director|head|agency director|head|agency minister|head|committee minister|head|committee leader|head|committee leader|head|committee president|head|team president|head|team chairman|run|committee chairman|run|committee director|head|team director|head|team representative|chair|committee representative|chair|committee president|head|committee president|head|committee of fi cer|chair|committee of fi cer|chair|committee director|head|committee director|head|committee Director|head|team Director|head|team Of fi cer|chair|Committee Of fi cer|chair|Committee representative|head|committee representative|head|committee May 28, 2018 From unsupervised induction of linguistic structures to applications in deep learning, A. Panchenko 82/109