Matrix Factorization March 17, 2020 Data Science CSCI 1951A Brown - - PowerPoint PPT Presentation

▶
matrix factorization
SMART_READER_LITE
LIVE PREVIEW

Matrix Factorization March 17, 2020 Data Science CSCI 1951A Brown - - PowerPoint PPT Presentation

Sep 05, 2022 154 likes •1.33k views

Matrix Factorization March 17, 2020 Data Science CSCI 1951A Brown University Instructor: Ellie Pavlick HTAs: Josh Levin, Diane Mutako, Sol Zitter 1 Announcements 2 Today Matrix Factorization with SVD Applications to: Topic

slide-1
SLIDE 1

Matrix Factorization

March 17, 2020 Data Science CSCI 1951A Brown University Instructor: Ellie Pavlick HTAs: Josh Levin, Diane Mutako, Sol Zitter

1

slide-2
SLIDE 2

Announcements

  • …

2

slide-3
SLIDE 3

Today

  • Matrix Factorization with SVD
  • Applications to: Topic Modeling, Recommendation

Systems

3

slide-4
SLIDE 4

Use Cases

  • Matrix Completion:
  • Recommendation—if someone likes/watches/clicks
  • n this, they might also like/watch/click on that
  • Dimensionality Reduction:
  • Embeddings/Denoising—Can I throw away

features without losing performance? Can I throw away noisy features an actually increase performance?

4

slide-5
SLIDE 5

What is dimensionality reduction?

Clicks Recency Reading Level Photo Title: “new” Title: “tax” Title: “this” … 10 1.3 11 1 1 … 1000 1.7 3 1 1 … 1000000 2.4 2 1 1 … 1 5.9 19 …

slide-6
SLIDE 6

What is dimensionality reduction?

Clicks Recency Reading Level Photo Title: “new” Title: “tax” Title: “this” … 10 1.3 11 1 1 … 1000 1.7 3 1 1 … 1000000 2.4 2 1 1 … 1 5.9 19 …

  • ften 1000s or (100s of 1000s) of features
slide-7
SLIDE 7

What is dimensionality reduction?

Clicks Recency Reading Level Photo Title: “new” Title: “tax” Title: “this” … 10 1.3 11 1 1 … 1000 1.7 3 1 1 … 1000000 2.4 2 1 1 … 1 5.9 19 …

many (most) are redundant or useless

slide-8
SLIDE 8

What is dimensionality reduction?

Clicks Recency Reading Level Photo Title: “new” Title: “tax” Title: “this” … 10 1.3 11 1 1 … 1000 1.7 3 1 1 … 1000000 2.4 2 1 1 … 1 5.9 19 …

  • slower to train
  • easier to overfit
  • harder to visualize/interpret (*)
slide-9
SLIDE 9

Principle Component Analysis (PCA)

feature 1 feature 2

slide-10
SLIDE 10

feature 1 feature 2

Principle Component Analysis (PCA)

slide-11
SLIDE 11

feature 1 feature 2

Principle Component Analysis (PCA)

slide-12
SLIDE 12

feature 1 feature 2

Principle Component Analysis (PCA)

Looses some distinction between points

slide-13
SLIDE 13

feature 1 feature 2

Principle Component Analysis (PCA)

slide-14
SLIDE 14

feature 1 feature 2

Principle Component Analysis (PCA)

slide-15
SLIDE 15

feature 1 feature 2

Principle Component Analysis (PCA)

Looses even more distinction between points

slide-16
SLIDE 16

feature 1 feature 2

Principle Component Analysis (PCA)

Choose arbitrary line which preserves as much variance as possible

slide-17
SLIDE 17

feature 1 feature 2

Principle Component Analysis (PCA)

First Principle Component Second Principle Component

slide-18
SLIDE 18

feature 1 feature 2

Principle Component Analysis (PCA)

First Principle Component Second Principle Component “change of basis”

slide-19
SLIDE 19

Principle Component Analysis (PCA)

slide-20
SLIDE 20
  • Eigenvalue Decomposition of covariance matrix
  • Singular Value Decomposition of the data matrix

Principle Component Analysis (PCA)

slide-21
SLIDE 21
  • Eigenvalue Decomposition of covariance matrix
  • Singular Value Decomposition of the data matrix

Principle Component Analysis (PCA)

slide-22
SLIDE 22
  • Eigenvalue Decomposition of covariance matrix
  • Singular Value Decomposition of the data matrix

Principle Component Analysis (PCA)

Technically, PCA != SVD (but in practice these are used interchangeably)

slide-23
SLIDE 23
  • Eigenvalue Decomposition of covariance matrix
  • Singular Value Decomposition of the data matrix

Principle Component Analysis (PCA)

Technically, PCA != SVD (but in practice these are used interchangeably)

slide-24
SLIDE 24

Rank of a matrix

2 1 1 4 3 1 2 2 8 4 4

slide-25
SLIDE 25

Rank of a matrix

2 1 1 4 3 1 2 2 8 4 4

= = = = + + + +

slide-26
SLIDE 26

Rank of a matrix

2 1 1 4 3 1 2 2 8 4 4

= = = = + + + +

Rank = 2

slide-27
SLIDE 27

Rank of a matrix

2 1 1 4 3 1 2 2 8 4 4

= = = = + + + +

Rank = 2 No new signal

slide-28
SLIDE 28

Clicker Question!

slide-29
SLIDE 29

Clicker Question! a) 5 b) 4 c) 3 d) 2 e) 1

What is the rank of this matrix?

the congress parliament US UK doc1 1 1 1 1 doc2 1 1 1 doc3 1 1 1 doc4 1 1 1

slide-30
SLIDE 30

Clicker Question! a) 5 b) 4 c) 3 d) 2 e) 1

What is the rank of this matrix?

the congress parliament US UK doc1 1 1 1 1 doc2 1 1 1 doc3 1 1 1 doc4 1 1 1

slide-31
SLIDE 31

Clicker Question! a) 5 b) 4 c) 3 d) 2 e) 1

What is the rank of this matrix?

the parliament US UK doc1 1 1 1 doc2 1 1 1 doc3 1 1 doc4 1 1 1

slide-32
SLIDE 32

Clicker Question! a) 5 b) 4 c) 3 d) 2 e) 1

What is the rank of this matrix?

the parliament US UK doc1 1 1 1 doc2 1 1 1 doc3 1 1 doc4 1 1 1

slide-33
SLIDE 33

Clicker Question! a) 5 b) 4 c) 3 d) 2 e) 1

What is the rank of this matrix?

parliament US UK doc1 1 1 doc2 1 1 doc3 1 doc4 1 1

slide-34
SLIDE 34

Rank of a matrix

“Low Rank Assumption”: we typically assume that our features contain a large amount of redundant information

the congress parliament US UK doc1 1 1 1 1 doc2 1 1 1 doc3 1 1 1 doc4 1 1 1

slide-35
SLIDE 35

Matrix Arithmetic Refresh

a1 a2 a3 b1 b2 b3

a b

slide-36
SLIDE 36

Matrix Arithmetic Refresh

a1 a2 a3 b1 b2 b3

b a·b = (a1xb1) + (a2xb2) + (a3xb3) a

slide-37
SLIDE 37

Matrix Arithmetic Refresh

a11 a12 a13 a21 a22 a23 a31 a32 a33 b11 b12 b21 b22 b31 b32

A B

slide-38
SLIDE 38

Matrix Arithmetic Refresh

a11 a12 a13 a21 a22 a23 a31 a32 a33 b11 b12 b21 b22 b31 b32

A B 3x3 3x2

slide-39
SLIDE 39

Matrix Arithmetic Refresh

a11 a12 a13 a21 a22 a23 a31 a32 a33 b11 b12 b21 b22 b31 b32

A B 3x3 3x2

slide-40
SLIDE 40

Matrix Arithmetic Refresh

a11 a12 a13 a21 a22 a23 a31 a32 a33 b11 b12 b21 b22 b31 b32

A B 3x3 3x2 AB 3x2

?? ?? ?? ?? ?? ??

slide-41
SLIDE 41

Matrix Arithmetic Refresh

a11 a12 a13 a21 a22 a23 a31 a32 a33 b11 b12 b21 b22 b31 b32

A B mxk kxn AB mxn

?? ?? ?? ?? ?? ??

slide-42
SLIDE 42

Matrix Arithmetic Refresh

A B AB

a1·b1 a2·b1 a2·b1 a2·b2 a3·b1 a3·b2

a1 a2 a3 b1 b2

slide-43
SLIDE 43

Matrix Arithmetic Refresh

A B AB

a1·b1 a2·b1 a2·b1 a2·b2 a3·b1 a3·b2

a1 a2 a3 b1 b2 AB[i][j] =ai·bj

slide-44
SLIDE 44

Clicker Question!

slide-45
SLIDE 45

Clicker Question!

1 2 3 3 4 5 2 4 1 2 3 1

x (a)

13 11 25 25 14 7 6 20 10 10 26 13 14 13 25 13 6

(b) (c)

slide-46
SLIDE 46

Clicker Question!

1 2 3 3 4 5 2 4 1 2 3 1

x (a)

13 11 25 25 14 7 6 20 10 10 26 13 14 13 25 13 6

(b) (c)

slide-47
SLIDE 47

Clicker Question!

1 2 3 3 4 5 2 4 1 2 3 1

x (a)

13 11 25 25 14 7 6 20 10 10 26 13 14 13 25 13 6

(b) (c) (1x2) + (2x1) + (3x3)

slide-48
SLIDE 48

Clicker Question!

1 2 3 3 4 5 2 4 1 2 3 1

x (a)

13 11 25 25 14 7 6 20 10 10 26 13 14 13 25 13 6

(b) (c) 2 + 2 + 9

slide-49
SLIDE 49

Clicker Question!

1 2 3 3 4 5 2 4 1 2 3 1

x (a)

13 11 25 25 14 7 6 20 10 10 26 13 14 13 25 13 6

(b) (c) 4 + 4 + 3

slide-50
SLIDE 50

Clicker Question! x (a)

13 11 25 25 14 7 6 20 10 10 26 13 14 13 25 13 6

(b) (c)

1 2 3 3 4 5 2 4 1 2 3 1

slide-51
SLIDE 51

Singular Value Decomposition (SVD)

M m x n

slide-52
SLIDE 52

Singular Value Decomposition (SVD)

M m x n = U m x m V n x n D m x n x x

slide-53
SLIDE 53

Singular Value Decomposition (SVD)

M m x n = U m x m V n x n D m x n x x Data Matrix

slide-54
SLIDE 54

Singular Value Decomposition (SVD)

M m x n = U m x m V n x n D m x n x x Data Matrix Singular Values of M

slide-55
SLIDE 55

Singular Value Decomposition (SVD)

M m x n = U m x m V n x n D m x n x x Data Matrix Singular Values of M (#non-zero = rank M)

slide-56
SLIDE 56

Singular Value Decomposition (SVD)

M m x n = U m x m V n x n D m x n x x Data Matrix Singular Values of M (#non-zero = rank M) Representation of rows of M in new feature space

slide-57
SLIDE 57

Singular Value Decomposition (SVD)

M m x n = U m x m V n x n D m x n x x Data Matrix Singular Values of M (#non-zero = rank M) Principle Components (new features) Representation of rows of M in new feature space

slide-58
SLIDE 58

Singular Value Decomposition (SVD)

the congr ess parlia ment US UK doc1 1 1 1 1 doc2 1 1 1 doc3 1 1 1 doc4 1 1 1

slide-59
SLIDE 59

Singular Value Decomposition (SVD)

the cong ress parli ame US UK doc1

1 1 1 1

doc2

1 1 1

doc3

1 1 1

doc4

1 1 1

d1 -0.60 -0.39 0.70 0.00 d2 -0.48 0.50 -0.12 -0.71 d3 -0.43 -0.58 -0.69 0.00 d4 -0.48 0.50 -0.12 0.71 3.06 0.00 0.00 0.00 0.00 0.00 1.81 0.00 0.00 0.00 0.00 0.00 0.57 0.00 0.00 0.00 0.00 0.00 0.00 0.00 the cong ress parlia ment US UK

  • 0.65 -0.34 -0.51 -0.34 -0.31

0.02

  • 0.54 0.34 -0.54 0.56
  • 0.42

0.02 0.79 0.02 -0.44

  • 0.63

0.27 0.00 0.37 0.63

  • 0.04

0.73 0.00 -0.68 0.04

U V D

slide-60
SLIDE 60

Singular Value Decomposition (SVD)

the cong ress parli ame US UK doc1

1 1 1 1

doc2

1 1 1

doc3

1 1 1

doc4

1 1 1

d1 -0.60 -0.39 0.70 0.00 d2 -0.48 0.50 -0.12 -0.71 d3 -0.43 -0.58 -0.69 0.00 d4 -0.48 0.50 -0.12 0.71 3.06 0.00 0.00 0.00 0.00 0.00 1.81 0.00 0.00 0.00 0.00 0.00 0.57 0.00 0.00 0.00 0.00 0.00 0.00 0.00 the cong ress parlia ment US UK

  • 0.65 -0.34 -0.51 -0.34 -0.31

0.02

  • 0.54 0.34 -0.54 0.56
  • 0.42

0.02 0.79 0.02 -0.44

  • 0.63

0.27 0.00 0.37 0.63

  • 0.04

0.73 0.00 -0.68 0.04

U V D doc1 in old feature space

slide-61
SLIDE 61

Singular Value Decomposition (SVD)

the cong ress parli ame US UK doc1

1 1 1 1

doc2

1 1 1

doc3

1 1 1

doc4

1 1 1

d1 -0.60 -0.39 0.70 0.00 d2 -0.48 0.50 -0.12 -0.71 d3 -0.43 -0.58 -0.69 0.00 d4 -0.48 0.50 -0.12 0.71 3.06 0.00 0.00 0.00 0.00 0.00 1.81 0.00 0.00 0.00 0.00 0.00 0.57 0.00 0.00 0.00 0.00 0.00 0.00 0.00 the cong ress parlia ment US UK

  • 0.65 -0.34 -0.51 -0.34 -0.31

0.02

  • 0.54 0.34 -0.54 0.56
  • 0.42

0.02 0.79 0.02 -0.44

  • 0.63

0.27 0.00 0.37 0.63

  • 0.04

0.73 0.00 -0.68 0.04

U V D doc1 in new feature space

slide-62
SLIDE 62

Singular Value Decomposition (SVD)

the cong ress parli ame US UK doc1

1 1 1 1

doc2

1 1 1

doc3

1 1 1

doc4

1 1 1

d1 -0.60 -0.39 0.70 0.00 d2 -0.48 0.50 -0.12 -0.71 d3 -0.43 -0.58 -0.69 0.00 d4 -0.48 0.50 -0.12 0.71 3.06 0.00 0.00 0.00 0.00 0.00 1.81 0.00 0.00 0.00 0.00 0.00 0.57 0.00 0.00 0.00 0.00 0.00 0.00 0.00 the cong ress parlia ment US UK

  • 0.65 -0.34 -0.51 -0.34 -0.31

0.02

  • 0.54 0.34 -0.54 0.56
  • 0.42

0.02 0.79 0.02 -0.44

  • 0.63

0.27 0.00 0.37 0.63

  • 0.04

0.73 0.00 -0.68 0.04

U V D weight of component 1 for doc 1

slide-63
SLIDE 63

Singular Value Decomposition (SVD)

the cong ress parli ame US UK doc1

1 1 1 1

doc2

1 1 1

doc3

1 1 1

doc4

1 1 1

d1 -0.60 -0.39 0.70 0.00 d2 -0.48 0.50 -0.12 -0.71 d3 -0.43 -0.58 -0.69 0.00 d4 -0.48 0.50 -0.12 0.71 3.06 0.00 0.00 0.00 0.00 0.00 1.81 0.00 0.00 0.00 0.00 0.00 0.57 0.00 0.00 0.00 0.00 0.00 0.00 0.00 the cong ress parlia ment US UK

  • 0.65 -0.34 -0.51 -0.34 -0.31

0.02

  • 0.54 0.34 -0.54 0.56
  • 0.42

0.02 0.79 0.02 -0.44

  • 0.63

0.27 0.00 0.37 0.63

  • 0.04

0.73 0.00 -0.68 0.04

U V D weight of component 1 over all the data

slide-64
SLIDE 64

Singular Value Decomposition (SVD)

the cong ress parli ame US UK doc1

1 1 1 1

doc2

1 1 1

doc3

1 1 1

doc4

1 1 1

d1 -0.60 -0.39 0.70 0.00 d2 -0.48 0.50 -0.12 -0.71 d3 -0.43 -0.58 -0.69 0.00 d4 -0.48 0.50 -0.12 0.71 3.06 0.00 0.00 0.00 0.00 0.00 1.81 0.00 0.00 0.00 0.00 0.00 0.57 0.00 0.00 0.00 0.00 0.00 0.00 0.00 the cong ress parlia ment US UK

  • 0.65 -0.34 -0.51 -0.34 -0.31

0.02

  • 0.54 0.34 -0.54 0.56
  • 0.42

0.02 0.79 0.02 -0.44

  • 0.63

0.27 0.00 0.37 0.63

  • 0.04

0.73 0.00 -0.68 0.04

U V D component 1

slide-65
SLIDE 65

Singular Value Decomposition (SVD)

the cong ress parli ame US UK doc1

1 1 1 1

doc2

1 1 1

doc3

1 1 1

doc4

1 1 1

d1 -0.60 -0.39 0.70 0.00 d2 -0.48 0.50 -0.12 -0.71 d3 -0.43 -0.58 -0.69 0.00 d4 -0.48 0.50 -0.12 0.71 3.06 0.00 0.00 0.00 0.00 0.00 1.81 0.00 0.00 0.00 0.00 0.00 0.57 0.00 0.00 0.00 0.00 0.00 0.00 0.00 the cong ress parlia ment US UK

  • 0.65 -0.34 -0.51 -0.34 -0.31

0.02

  • 0.54 0.34 -0.54 0.56
  • 0.42

0.02 0.79 0.02 -0.44

  • 0.63

0.27 0.00 0.37 0.63

  • 0.04

0.73 0.00 -0.68 0.04

U V D contribution of “the” to component 1

slide-66
SLIDE 66

Singular Value Decomposition (SVD)

M m x n = U m x m V n x n D m x n x x

slide-67
SLIDE 67

67

slide-68
SLIDE 68

Singular Value Decomposition (SVD)

M m x n = U m x l V l x n D l x l x x

T r u n c a t e d

slide-69
SLIDE 69

Singular Value Decomposition (SVD)

M m x n = U m x l V l x n D l x l x x

T r u n c a t e d

keep only first l components

slide-70
SLIDE 70

Singular Value Decomposition (SVD)

M m x n = U m x l V l x n D l x l x x

T r u n c a t e d

keep only first l components “best l-rank approximation of M”

slide-71
SLIDE 71

Singular Value Decomposition (SVD)

M m x n = U m x l V l x n D l x l x x

T r u n c a t e d

keep only first l components “best l-rank approximation of M” ||M - UDV||2 as small as possible

slide-72
SLIDE 72

72

slide-73
SLIDE 73

Dimensionality Reduction

  • “Low Rank Assumption”: we typically assume that
  • ur features contain a large amount of redundant

information

  • We can throw away a lot of principle components

without losing too much of the signal needed for

  • ur task
slide-74
SLIDE 74

Clicker Question!

slide-75
SLIDE 75

Clicker Question! a) Yes b) No c) Yeah, sure, why not? In practice, is this assumption of low rank valid?

slide-76
SLIDE 76

Matrices IRL

  • Data is noisy, so M is most likely full-rank
  • We assume that M is close to a low rank matrix,

and we approximate the matrix it is close to

  • Viewed as a “de-noised” version of M
  • “Original matrix exhibits redundancy and noise,

low-rank reconstruction exploits the former to remove the latter”*

slide-77
SLIDE 77

Matrices IRL

  • Data is noisy, so M is most likely full-rank
  • We assume that M is close to a low rank matrix,

and we approximate the matrix it is close to

  • Viewed as a “de-noised” version of M
  • “Original matrix exhibits redundancy and noise,

low-rank reconstruction exploits the former to remove the latter”*

slide-78
SLIDE 78

Matrices IRL

  • Data is noisy, so M is most likely full-rank
  • We assume that M is close to a low rank matrix,

and we approximate the matrix it is close to

  • Viewed as a “de-noised” version of M
  • “Original matrix exhibits redundancy and noise,

low-rank reconstruction exploits the former to remove the latter”*

slide-79
SLIDE 79

Matrices IRL

  • Data is noisy, so M is most likely full-rank
  • We assume that M is close to a low rank matrix,

and we approximate the matrix it is close to

  • Viewed as a “de-noised” version of M
  • “Original matrix exhibits redundancy and noise,

low-rank reconstruction exploits the former to remove the latter”*

slide-80
SLIDE 80

Matrices IRL

  • Data is noisy, so M is most likely full-rank
  • We assume that M is close to a low rank matrix,

and we approximate the matrix it is close to

  • Viewed as a “de-noised” version of M
  • “Original matrix exhibits redundancy and noise,

low-rank reconstruction exploits the former to remove the latter”*

*Matrix and Tensor Factorization Methods for Natural Language Processing. (ACL 2015)

slide-81
SLIDE 81

Matrices IRL

  • Data is also often incomplete…missing values, new
  • bservations, etc.
  • Can we use SVD for this?
  • Yes! Though we need to make a few changes…
slide-82
SLIDE 82

Matrices IRL

  • Data is also often incomplete…missing values, new
  • bservations, etc.
  • Can we use SVD for this?
  • Yes! Though we need to make a few changes…
slide-83
SLIDE 83

Matrices IRL

  • Data is also often incomplete…missing values, new
  • bservations, etc.
  • Can we use SVD for this?
  • Yes! Though we need to make a few changes…
slide-84
SLIDE 84

Matrices IRL

  • Data is also often incomplete…missing values, new
  • bservations, etc.
  • Can we use SVD for this?
  • Yes! Though we need to make a few changes…
slide-85
SLIDE 85

Matrix Completion

roma ballad of buster scruggs… mud- bound to all the boys i loved before

  • kja

user1 1 1 user2 1 user3 1 1 user4 1 user5 1

slide-86
SLIDE 86

Matrix Completion

roma ballad of buster scruggs… mud- bound to all the boys i loved before

  • kja

user1 1 1 1 user2 1 user3 1 1 1 user4 1 user5 1 “people also liked…”

slide-87
SLIDE 87

Matrix Completion

slide-88
SLIDE 88

Matrix Completion M ≈ UDV = M’

slide-89
SLIDE 89

Matrix Completion M ≈ UDV = M’

  • riginal

completed

slide-90
SLIDE 90

Matrix Completion M ≈ UDV = M’

  • riginal

completed problems?

slide-91
SLIDE 91

Matrix Completion

Exact SVD assumes M is complete… M = U V D x x

slide-92
SLIDE 92

Matrix Completion

…just gradient descent that MF! M = U V D x x

slide-93
SLIDE 93

MF with Gradient Descent

M = U V D x x

slide-94
SLIDE 94

MF with Gradient Descent

M = U V x

slide-95
SLIDE 95

MF with Gradient Descent

M = U V x Not properly SVD (fewer guarantees, e.g. components not

  • rthonormal) but good enough
slide-96
SLIDE 96

MF with Gradient Descent

M = U V x

min

U,V

X

ij

(Mij − ui · vj)2

<latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit><latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit><latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit><latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit>
slide-97
SLIDE 97

MF with Gradient Descent

M = U V x

min

U,V

X

ij

(Mij − ui · vj)2

<latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit><latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit><latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit><latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit>

But! Only consider cases when Mij is observed!

slide-98
SLIDE 98

Clicker Question!

slide-99
SLIDE 99

Clicker Question!

2 3 2 1 2 1 2

min

U,V

X

ij

(Mij − ui · vj)2

<latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit><latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit><latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit><latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit>

M U V Compute the loss given this setting of U and V…

a) 14 b) 10 c) 6

slide-100
SLIDE 100

Clicker Question!

2 3 2 1 2 1 2

min

U,V

X

ij

(Mij − ui · vj)2

<latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit><latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit><latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit><latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit>

M U V Compute the loss given this setting of U and V…

a) 14 b) 10 c) 6

= x 1 2 2 4 M’

slide-101
SLIDE 101

Clicker Question!

2 3 2 1 2 1 2

min

U,V

X

ij

(Mij − ui · vj)2

<latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit><latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit><latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit><latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit>

M U V Compute the loss given this setting of U and V…

a) 14 b) 10 c) 6

= x 1 2 2 4 M’

slide-102
SLIDE 102

Clicker Question!

2 3 2 1 2 1 2

min

U,V

X

ij

(Mij − ui · vj)2

<latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit><latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit><latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit><latexit sha1_base64="FxlQFTpt3XT/cP7nfIKG0d4whfk=">ACGXicbVDLSgMxFM34rPU16tJNsAgVtMwUQZdFN26ECvYBnXHIpGmbNskMSaZQhv6G3/FjQtFXOrKvzFtZ6GtBy73cM69JPeEMaNKO863tbS8srq2ntvIb25t7+zae/t1FSUSkxqOWCSbIVKEUFqmpGmrEkiIeMNMLB9cRvDIlUNBL3ehQTn6OuoB2KkTZSYDsepyJIa6ewPoaeSniQ0v4YFm9n/QwmAYUebkcaDoP+yUM5sAtOyZkCLhI3IwWQoRrYn147wgknQmOGlGq5Tqz9FElNMSPjvJcoEiM8QF3SMlQgTpSfTi8bw2OjtGEnkqaEhlP190aKuFIjHpJjnRPzXsT8T+vlejOpZ9SESeaCDx7qJMwqCM4iQm2qSRYs5EhCEtq/gpxD0mEtQkzb0Jw509eJPVyXVK7t15oXKVxZEDh+AIFIELkAF3IAqAEMHsEzeAVv1pP1Yr1bH7PRJSvbOQB/YH39ACajnxs=</latexit>

M U V Compute the loss given this setting of U and V…

a) 14 b) 10 c) 6

= x 1 2 2 4 M’

1 + 1 + 4

slide-103
SLIDE 103

103

slide-104
SLIDE 104

Changes I make to the nations.js file do not affect any of the html in after I load the nations.html file When I try to display dots from part 2 on my mac (tried chrome, firefox, and safari), the elements do not appear in the html. Can you elaborate on exactly what the directions are in part 2 step 3, the stencil code does not quite imply what we are supposed to do…

Topic Models

slide-105
SLIDE 105

Topic Models

Changes I make to the nations.js file do not affect any of the html in after I load the nations.html file When I try to display dots from part 2 on my mac (tried chrome, firefox, and safari), the elements do not appear in the html. Can you elaborate on exactly what the directions are in part 2 step 3, the stencil code does not quite imply what we are supposed to do…

instructions: stencil, instructions, part, step, rubric, handin… UI: html, javascript, debug, display, elements… systems: mac, windows, linux, chrome, firefox, os… fillers: I, you, when, the, and, a

slide-106
SLIDE 106

Topic Models

“Latent Semantic Analysis” (LSA)

slide-107
SLIDE 107

Topic Models

“Latent Semantic Analysis” (LSA) words are determined by topic (and are conditionally independent of each other)

slide-108
SLIDE 108

Topic Models

“Latent Semantic Analysis” (LSA) documents are a distribution over topics

slide-109
SLIDE 109

Topic Models

the cong ress parli ame US UK doc1

1 1 1 1

doc2

1 1 1

doc3

1 1 1

doc4

1 1 1

d1 -0.60 -0.39 0.70 0.00 d2 -0.48 0.50 -0.12 -0.71 d3 -0.43 -0.58 -0.69 0.00 d4 -0.48 0.50 -0.12 0.71 3.06 0.00 0.00 0.00 0.00 0.00 1.81 0.00 0.00 0.00 0.00 0.00 0.57 0.00 0.00 0.00 0.00 0.00 0.00 0.00 the cong ress parlia ment US UK

  • 0.65 -0.34 -0.51 -0.34 -0.31

0.02

  • 0.54 0.34 -0.54 0.56
  • 0.42

0.02 0.79 0.02 -0.44

  • 0.63

0.27 0.00 0.37 0.63

  • 0.04

0.73 0.00 -0.68 0.04

U V D

slide-110
SLIDE 110

Topic Models

the cong ress parli ame US UK doc1

1 1 1 1

doc2

1 1 1

doc3

1 1 1

doc4

1 1 1

d1 -0.60 -0.39 0.70 0.00 d2 -0.48 0.50 -0.12 -0.71 d3 -0.43 -0.58 -0.69 0.00 d4 -0.48 0.50 -0.12 0.71 3.06 0.00 0.00 0.00 0.00 0.00 1.81 0.00 0.00 0.00 0.00 0.00 0.57 0.00 0.00 0.00 0.00 0.00 0.00 0.00 the cong ress parlia ment US UK

  • 0.65 -0.34 -0.51 -0.34 -0.31

0.02

  • 0.54 0.34 -0.54 0.56
  • 0.42

0.02 0.79 0.02 -0.44

  • 0.63

0.27 0.00 0.37 0.63

  • 0.04

0.73 0.00 -0.68 0.04

U V D component = “topic”

slide-111
SLIDE 111

Topic Models

the cong ress parli ame US UK doc1

1 1 1 1

doc2

1 1 1

doc3

1 1 1

doc4

1 1 1

d1 -0.60 -0.39 0.70 0.00 d2 -0.48 0.50 -0.12 -0.71 d3 -0.43 -0.58 -0.69 0.00 d4 -0.48 0.50 -0.12 0.71 3.06 0.00 0.00 0.00 0.00 0.00 1.81 0.00 0.00 0.00 0.00 0.00 0.57 0.00 0.00 0.00 0.00 0.00 0.00 0.00 the cong ress parlia ment US UK

  • 0.65 -0.34 -0.51 -0.34 -0.31

0.02

  • 0.54 0.34 -0.54 0.56
  • 0.42

0.02 0.79 0.02 -0.44

  • 0.63

0.27 0.00 0.37 0.63

  • 0.04

0.73 0.00 -0.68 0.04

U V D component = “topic” = distribution over words

slide-112
SLIDE 112

Topic Models

the cong ress parli ame US UK doc1

1 1 1 1

doc2

1 1 1

doc3

1 1 1

doc4

1 1 1

d1 -0.60 -0.39 0.70 0.00 d2 -0.48 0.50 -0.12 -0.71 d3 -0.43 -0.58 -0.69 0.00 d4 -0.48 0.50 -0.12 0.71 3.06 0.00 0.00 0.00 0.00 0.00 1.81 0.00 0.00 0.00 0.00 0.00 0.57 0.00 0.00 0.00 0.00 0.00 0.00 0.00 the cong ress parlia ment US UK

  • 0.65 -0.34 -0.51 -0.34 -0.31

0.02

  • 0.54 0.34 -0.54 0.56
  • 0.42

0.02 0.79 0.02 -0.44

  • 0.63

0.27 0.00 0.37 0.63

  • 0.04

0.73 0.00 -0.68 0.04

U V D document = distribution

  • ver topics
slide-113
SLIDE 113

Topic Models

the congress parliament US UK doc1 1 1 1 1 doc2 1 1 1 doc3 1 1 1 doc4 1 1 1

Factorization of the term-document matrix

slide-114
SLIDE 114

Word Embeddings

the congress parliament US UK the 1 1 1 1 1 congress 1 1 1 parlaiment 1 1 1 1 US 1 1 1 1 UK 1 1 1

Factorization of the term-context matrix More on Thursday!

slide-115
SLIDE 115

Word Embeddings

the

  • 0.60 -0.39 0.70 0.00

congress -0.48 0.50 -0.12 -0.71 parliament -0.43 -0.58 -0.69 0.00 US

  • 0.48 0.50 -0.12 0.71

UK 0.02 0.79 0.02 -0.44

  • 0.65 -0.34 -0.51 -0.34 -0.31

0.02 -0.54 0.34 -0.54 0.56

  • 0.42 0.02 0.79 0.02 -0.44
  • 0.63 0.27 0.00 0.37 0.63
  • 0.04 0.73 0.00 -0.68 0.04

x

the con- gress parlia- ment US UK the 1 1 1 1 1 congress 1 1 1 parlaiment 1 1 1 1 US 1 1 1 1 UK 1 1 1

= Embeddings! More on Thursday!

slide-116
SLIDE 116

Word Embeddings

More on Thursday!

slide-117
SLIDE 117

Useful Resources/ References

  • https://github.com/uclmr/acl2015tutorial/
  • https://web.stanford.edu/~jurafsky/li15/lec3.vector.pdf
  • https://arxiv.org/pdf/1404.1100.pdf
  • https://towardsdatascience.com/pca-and-svd-explained-with-

numpy-5d13b0d2a4d8

  • http://nicolas-hug.com/blog/matrix_facto_3
  • https://machinelearningmastery.com/singular-value-decomposition-

for-machine-learning/

  • http://cocosci.princeton.edu/tom/papers/SteyversGriffiths.pdf