http://demo.clab.cs.cmu.edu/algo4nlp19/ - - PowerPoint PPT Presentation

▶
http demo clab cs cmu edu algo4nlp19 https piazza com
SMART_READER_LITE
LIVE PREVIEW

http://demo.clab.cs.cmu.edu/algo4nlp19/ - - PowerPoint PPT Presentation

Apr 04, 2023 127 likes •941 views

http://demo.clab.cs.cmu.edu/algo4nlp19/ https://piazza.com/class/jy617kmo6ub134 Chan Young Park Jong Hyuk Park Slide credit: Noah Smith Slide credit: Noah Smith Slide credit: Noah Smith Slide credit: Noah

slide-1
SLIDE 1
slide-2
SLIDE 2

▪

http://demo.clab.cs.cmu.edu/algo4nlp19/

▪

https://piazza.com/class/jy617kmo6ub134

▪

▪

Chan Young Park ▪ Jong Hyuk Park

▪

▪

slide-3
SLIDE 3

Slide credit: Noah Smith

slide-4
SLIDE 4

Slide credit: Noah Smith

slide-5
SLIDE 5

Slide credit: Noah Smith

slide-6
SLIDE 6

Slide credit: Noah Smith

slide-7
SLIDE 7

Slide credit: Noah Smith

slide-8
SLIDE 8

Slide credit: Noah Smith

slide-9
SLIDE 9
slide-10
SLIDE 10

My legal name is Alexander Perchov.

slide-11
SLIDE 11

My legal name is Alexander Perchov. But all of my many friends dub me Alex, because that is a more flaccid-to-utter version of my legal name.

slide-12
SLIDE 12

My legal name is Alexander Perchov. But all of my many friends dub me Alex, because that is a more flaccid-to-utter version of my legal name. Mother dubs me Alexi-stop-spleening-me!, because I am always spleening her.

slide-13
SLIDE 13

My legal name is Alexander Perchov. But all of my many friends dub me Alex, because that is a more flaccid-to-utter version of my legal name. Mother dubs me Alexi-stop-spleening-me!, because I am always spleening her. If you want to know why I am always spleening her, it is because I am always elsewhere with friends, and disseminating so much currency, and performing so many things that can spleen a mother.

slide-14
SLIDE 14

My legal name is Alexander Perchov. But all of my many friends dub me Alex, because that is a more flaccid-to-utter version of my legal name. Mother dubs me Alexi-stop-spleening-me!, because I am always spleening her. If you want to know why I am always spleening her, it is because I am always elsewhere with friends, and disseminating so much currency, and performing so many things that can spleen a mother. Father used to dub me Shapka, for the fur hat I would don even in the summer month.

slide-15
SLIDE 15

My legal name is Alexander Perchov. But all of my many friends dub me Alex, because that is a more flaccid-to-utter version of my legal name. Mother dubs me Alexi-stop-spleening-me!, because I am always spleening her. If you want to know why I am always spleening her, it is because I am always elsewhere with friends, and disseminating so much currency, and performing so many things that can spleen a mother. Father used to dub me Shapka, for the fur hat I would don even in the summer

  • month. He ceased dubbing me that because I
  • rdered him to cease dubbing me that. It sounded

boyish to me, and I have always thought of myself as very potent and generative.

slide-16
SLIDE 16

▪ ▪ ▪ ▪

slide-17
SLIDE 17

▪

▪ ▪

▪

▪ ▪ ▪ ▪ ▪

slide-18
SLIDE 18

▪

▪ ▪

slide-19
SLIDE 19

disseminating so much currency spending a lot of money

slide-20
SLIDE 20

▪

▪ ▪

slide-21
SLIDE 21

▪

▪

▪

▪ ▪

▪

▪

▪

slide-22
SLIDE 22

▪

s p ee ch l a b

slide-23
SLIDE 23

▪

slide-24
SLIDE 24
slide-25
SLIDE 25
slide-26
SLIDE 26
slide-27
SLIDE 27

▪

slide-28
SLIDE 28

▪ ▪

slide-29
SLIDE 29

▪

slide-30
SLIDE 30

▪

slide-31
SLIDE 31
slide-32
SLIDE 32
slide-33
SLIDE 33

sent transmission: English recovered transmission: French recovered message: English’

slide-34
SLIDE 34
slide-35
SLIDE 35

▪ ▪ ▪ ▪ ▪ handwriting recognition ▪ document summarization ▪ dialog generation ▪ linguistic decipherment ▪

slide-36
SLIDE 36

▪ ▪ ▪ ▪ ▪ ▪

slide-37
SLIDE 37

▪

▪ ▪

slide-38
SLIDE 38

▪ ▪ ▪

slide-39
SLIDE 39

▪ ▪ ▪ ▪

slide-40
SLIDE 40

▪

▪

Given a sequence of n random variables: ▪ We want a sequence probability model

slide-41
SLIDE 41

▪

▪

Given a sequence of n random variables: ▪ We want a sequence probability model ▪ There are |V|n possible sequences

slide-42
SLIDE 42
slide-43
SLIDE 43
slide-44
SLIDE 44

▪ Relax independence assumption:

slide-45
SLIDE 45

▪ Relax independence assumption: ▪ Simplify notation:

slide-46
SLIDE 46

▪ We want probability distribution over sequences of any length

slide-47
SLIDE 47

▪ Probability distribution over sequences of any length ▪ Define always Xn=STOP, where STOP is a special symbol

slide-48
SLIDE 48

▪ Probability distribution over sequences of any length ▪ Define always Xn=STOP, where STOP is a special symbol ▪ Then use a Markov process as before: ▪ We now have probability distribution over all sequences

▪ Intuition: at every step you have probability 𝛽h to stop (conditioned

  • n history) and (1-𝛽h) to keep going
slide-49
SLIDE 49
slide-50
SLIDE 50

▪

▪ ▪ ▪

slide-51
SLIDE 51
slide-52
SLIDE 52
slide-53
SLIDE 53
slide-54
SLIDE 54
slide-55
SLIDE 55

▪ ▪

slide-56
SLIDE 56

▪ ▪ ▪ ▪ ▪ ▪

slide-57
SLIDE 57

▪

▪ ▪

198015222 the first 194623024 the same 168504105 the following 158562063 the world … 14112454 the door

  • 23135851162 the *

Training Counts

slide-58
SLIDE 58

▪

198015222 the first 194623024 the same 168504105 the following 158562063 the world … 14112454 the door

  • 23135851162 the *

197302 close the window 191125 close the door 152500 close the gap 116451 close the thread 87298 close the deal …

  • 3785230 close the *

Bigram Model Trigram Model P(door | the) = 0.0006 P(door | close the) = 0.05

slide-59
SLIDE 59

▪ ▪ ▪ ▪ ▪ ▪

slide-60
SLIDE 60

▪

slide-61
SLIDE 61
slide-62
SLIDE 62

▪ ▪ ▪

slide-63
SLIDE 63

▪

▪ ▪ ⇒

slide-64
SLIDE 64

▪

▪ ▪ ⇒ ▪ ▪ ⇒

slide-65
SLIDE 65

▪ ▪ ▪ ▪ ▪ ▪

slide-66
SLIDE 66

▪ ▪

slide-67
SLIDE 67

▪ ▪ ▪ ▪

Training Data Held-Out Data Test Data

Counts / parameters from here Hyperparameters from here Evaluate here

slide-68
SLIDE 68

▪

▪ ▪

slide-69
SLIDE 69

▪

▪ ▪ ▪ ▪

slide-70
SLIDE 70

▪

▪ ⇒ ▪ ⇒ ▪ ⇒

▪ ▪

▪

slide-71
SLIDE 71

▪ ▪ ▪ ▪

slide-72
SLIDE 72

▪ ▪ ▪ ▪ ▪ ▪

▪

slide-73
SLIDE 73

▪ unk unk

▪

unk ▪

slide-74
SLIDE 74

▪

slide-75
SLIDE 75
slide-76
SLIDE 76

▪ ▪

▪

▪ ▪

▪ ▪

slide-77
SLIDE 77

▪

slide-78
SLIDE 78

▪

slide-79
SLIDE 79

Training Data Held-Out Data Test Data

Counts / parameters from here Hyperparameters from here Evaluate here

slide-80
SLIDE 80

▪ ▪ ▪