Matrix Completion from Fewer Entries Raghunandan Keshavan, Andrea - - PowerPoint PPT Presentation

▶
matrix completion from fewer entries
SMART_READER_LITE
LIVE PREVIEW

Matrix Completion from Fewer Entries Raghunandan Keshavan, Andrea - - PowerPoint PPT Presentation

May 29, 2023 370 likes •918 views

Matrix Completion from Fewer Entries Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Stanford University March 30, 2009 Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion Outline The problem, a look at the data, and

slide-1
SLIDE 1

Matrix Completion from Fewer Entries

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh

Stanford University

March 30, 2009

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-2
SLIDE 2

Outline

1

The problem, a look at the data, and some results (slides)

2

Proofs (blackboard) arXiv:0901.3150

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-3
SLIDE 3

The problem, a look at the data, and some results

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-4
SLIDE 4

Netflix dataset: A big (!) matrix

3 1 3 4 1 1 5 4 4 4 4 4 4 4 4 4 4 4 4 4 4 3 3 3 3 1 1 1 1 1 5 5 5 2 2 2 2

5 · 105 users 2 · 104 movies 108 ratings M =

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-5
SLIDE 5

A big (!) matrix

?

1 3 4 1 1 5 4 4 4 4 4 4 4 4 4 4 4 4 4 4 3 3 3 3 1 1 1 1 1 5 5 5 2 2 2 2 3

? ? ? ?

5 · 105 users 2 · 104 movies 106 queries M =

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-6
SLIDE 6

You get a prize if. . .

RMSE < 0.8563 ; −) Is this possible?

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-7
SLIDE 7

You get a prize if. . .

RMSE < 0.8563 ; −) Is this possible?

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-8
SLIDE 8

You get a prize if. . .

RMSE < 0.8563 ; −) Is this possible?

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-9
SLIDE 9

A model: Incoherent low-rank matrices

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-10
SLIDE 10

The observations

24312412365126251454231542321542143214324135124424225534242444245231552162561272662262626711515252241 13421532161432614361436514327147171542154437171521726547152481582524858141258141841852423233334448148 24312412365126251454231542321542143214324135124423323212144422555231552162561272662262626711515252241 24312412365126251454231542321542143214324135124424225552315521625612726622621412412212626711515252241 41315426514236152461547‘614542422471‘6567157157‘65147‘615241543154311315464566366531253151353116‘161466 24312412365126251454231542321542143214324135124424225552315521625612726622626267115143434343225252241 24312412365126251454231542321542143214324135124424225552315521625612726622623452352566626711515252241 41315426514236152461547‘614542422471‘6567157157‘65147‘615241543154311315312345334646653151353116‘161466 24312412365126251454231542321542143214324135124424225552315521343466663562561272662262626711515252241 24312412365126251454231542321542143214324135124424223434543453555231552162561272662262626711515252241 41315426514236152461547‘614542422471‘6567157157‘65147‘353534543361524154315431131531253151353116‘161466 24312412365126251454231542321542143214343453453452413512442422555231552162561272662262626711515252241 24312412365126251454231542321542143245345354551432413512442422555423155216256127266226262671151525241 41315426514236152461547‘614542422471‘346567157157‘65147‘61524154315431131531253151353116‘16146453454356 24312412365126251454231542321542143214324135133133111124424225552315521625461272662262626711515252241 24312412365126251454231542321542143334211233321432413512442422555231552162561272662262626711515252241 24312412365126251454231542321542143214324135124424225552315521625612721231‘13132662262626711515252241 41315426514236152461547‘614542422471‘6567157157‘65147‘615241543154311131232333311531253151353116‘161466 24312412365126251454231542321542143214324135124424225552315521625612726622626267115152522443744747441 41315426514236152461547‘614542422471‘6567157157‘65147‘615241543154311343344453551531253151353116‘161466 143265421542715765127651543151221652465236125436541625143615243162534535666461r5261463416452646161611 41315426514236152461547‘614542422471‘6567157157‘65147‘615241543154366363443135131531253151353116‘161466 41315426514236152461547‘614542422471‘6567157157‘65147‘615241543144444345554431131531253151353116‘161466 41315426514236152461547‘614542422471‘6567157157‘65147‘615241545345346664315431131531253151353116‘161466 24312412365126251454231542321542143214324135124424225552315521625446346466661272662262626711515252241 24312412365126251454231542321542143214324135124424225552315521625446436666661272662262626711515252241 24312412365126251454231542321542143214324135124424225552315521625464363423361272662262626711515252241 41315426514236152461547‘614542422471‘6567157157‘65147‘615241543135353453445431131531253151353116‘161466 24312412365126251454231542321542143214324135124424225552315521624534466433561272662262626711515252241 24312412365126251454231542321542343434445514321432413512442422555231552162561272662262626711515252241 24312412365126251454231542434444453332154214321432413512442422555231552162561272662262626711515252241 24312412365126251454231542321542121432413512442422555231552162561272662262626713242442515252233333341 5125125653426356254412545346532671735712351663571237213533333333172671238127638172681871881

nα users n movies M =

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-11
SLIDE 11

The observations

3 1 3 4 1 1 5 4 4 4 4 4 4 4 4 4 4 4 4 4 4 3 3 3 3 1 1 1 1 1 5 5 5 2 2 2 2

nα users n movies nǫ unif. random positions ME =

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-12
SLIDE 12

You need some structure!

nα r r n M = U VT r ≪ n

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-13
SLIDE 13

You need some structure!

nα r r n M = U VT r ≪ n

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-14
SLIDE 14

Unstructured factors

  • A1. Bounded entries

|Mia| ≤ Mmax = µ0 √r .

  • A2. Incoherence

r

  • k=1

U2

ik ≤ µ1 r , r

  • k=1

V2

ak ≤ µ1 r .

[Cand´ es, Recht 2008]

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-15
SLIDE 15

Metric (RMSE)

D(M, ˆ M) ≡    1 n2M2

max

  • i,a

|Mia − ˆ Mia|2   

1/2

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-16
SLIDE 16

Previous work

Theorem (Cand´ es, Recht, 2008) If ǫ ≥ C r n1/5 log n then whp

  • 1. M is unique given the observed entries.
  • 2. M is the unique minimum of a SDP.
  • cf. also [Recht, Fazel, Parrilo 2007]

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-17
SLIDE 17

Previous work

Theorem (Cand´ es, Recht, 2008) If ǫ ≥ C r n1/5 log n then whp

  • 1. M is unique given the observed entries.
  • 2. M is the unique minimum of a SDP.
  • cf. also [Recht, Fazel, Parrilo 2007]

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-18
SLIDE 18

Previous work

Theorem (Cand´ es, Recht, 2008) If ǫ ≥ C r n1/5 log n then whp

  • 1. M is unique given the observed entries.
  • 2. M is the unique minimum of a SDP.
  • cf. also [Recht, Fazel, Parrilo 2007]

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-19
SLIDE 19

Previous work

Theorem (Cand´ es, Recht, 2008) If ǫ ≥ C r n1/5 log n then whp

  • 1. M is unique given the observed entries.
  • 2. M is the unique minimum of a SDP.
  • cf. also [Recht, Fazel, Parrilo 2007]

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-20
SLIDE 20

Previous work

Theorem (Cand´ es, Recht, 2008) If ǫ ≥ C r n1/5 log n then whp

  • 1. M is unique given the observed entries.
  • 2. M is the unique minimum of a SDP.
  • cf. also [Recht, Fazel, Parrilo 2007]

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-21
SLIDE 21

Great, but. . .

  • 1. n1/5 observations for 1 bit of information?
  • 2. RMSE = 0?
  • 3. SDP =

O(n4...6). Substitute n = 105. . .

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-22
SLIDE 22

Great, but. . .

  • 1. n1/5 observations for 1 bit of information?
  • 2. RMSE = 0?
  • 3. SDP =

O(n4...6). Substitute n = 105. . .

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-23
SLIDE 23

Great, but. . .

  • 1. n1/5 observations for 1 bit of information?
  • 2. RMSE = 0?
  • 3. SDP =

O(n4...6). Substitute n = 105. . .

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-24
SLIDE 24

Great, but. . .

  • 1. n1/5 observations for 1 bit of information?
  • 2. RMSE = 0?
  • 3. SDP =

O(n4...6). Substitute n = 105. . .

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-25
SLIDE 25

O(n) entries are enough (practice)

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-26
SLIDE 26

A movie

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-27
SLIDE 27

Rank = 1: Bayes optimal vs. Belief Propagation

0.2 0.4 0.6 0.8 1 2 4 6 8 10 12 14 n=100 n=1000 n=10000

ǫ D

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-28
SLIDE 28

Rank = 2: Belief Propagation

0.2 0.4 0.6 0.8 1 1.2 1.4 2 4 6 8 10 12 n=100 n=1000 n=10000

ǫ D

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-29
SLIDE 29

Rank = 3: Belief Propagation

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 4 6 8 10 12 14 16 18 n=100 n=1000 n=10000

ǫ D

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-30
SLIDE 30

Rank = 4: Belief Propagation

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 5 10 15 20 n=100 n=1000 n=10000

ǫ D

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-31
SLIDE 31

O(n) entries are enough (theory)

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-32
SLIDE 32

Naive spectral algorithm

ME

ia =

Mia if (i, a) ∈ E,

  • therwise.

Projection ME =

n

  • i=1

σixiyT

i ,

σ1 ≥ σ2 ≥ . . . Tr(ME) = n√α ǫ

r

  • i=1

σixiyT

i .

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-33
SLIDE 33

Naive spectral algorithm

ME

ia =

Mia if (i, a) ∈ E,

  • therwise.

Projection ME =

n

  • i=1

σixiyT

i ,

σ1 ≥ σ2 ≥ . . . Tr(ME) = n√α ǫ

r

  • i=1

σixiyT

i .

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-34
SLIDE 34

Troubles and solution

If ǫ = O(1), ‘spurious’ singular values Ω(

  • log n/(log log n)).

Trimming

  • ME

ia =

ME

ia

if deg(i) ≤ 2 Edeg(i), deg(a) ≤ 2 Edeg(a) ,

  • therwise.

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-35
SLIDE 35

Troubles and solution

If ǫ = O(1), ‘spurious’ singular values Ω(

  • log n/(log log n)).

Trimming

  • ME

ia =

ME

ia

if deg(i) ≤ 2 Edeg(i), deg(a) ≤ 2 Edeg(a) ,

  • therwise.

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-36
SLIDE 36

Not-as-naive spectral algorithm

Spectral Matrix Completion( matrix ME ) 1: Trim ME, and let ME be the output; 2: Project ME to Tr( ME); 3: Clean residual errors by gradient descent in the factors.

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-37
SLIDE 37

Theorem (Keshavan, M, Oh, 2009) Assume r ≤ n1/2 and bounded entries. Then 1 nMmax ||M − Tr( ME)||F = RMSE ≤ C

  • r/ǫ.

with probability larger than 1 − exp(−Bn). Theorem (Keshavan, M, Oh, 2009) Assune r = O(1), bounded entries and incoherent factors, with Σmin, Σmax uniformly bounded away from 0 and ∞. If ǫ ≥ C ′ log n then Spectral Matrix Completion returns, whp, the matrix M.

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-38
SLIDE 38

Theorem (Keshavan, M, Oh, 2009) Assume r ≤ n1/2 and bounded entries. Then 1 nMmax ||M − Tr( ME)||F = RMSE ≤ C

  • r/ǫ.

with probability larger than 1 − exp(−Bn). Theorem (Keshavan, M, Oh, 2009) Assune r = O(1), bounded entries and incoherent factors, with Σmin, Σmax uniformly bounded away from 0 and ∞. If ǫ ≥ C ′ log n then Spectral Matrix Completion returns, whp, the matrix M.

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-39
SLIDE 39

A comparison

Theorem (Achlioptas, McSherry 2007) Assume ǫ ≥ (8 log n)4 and bounded entries. Then 1 nMmax ||M − Tr( ME)||F = RMSE ≤ 4

  • r/ǫ.

with probability larger than 1 − exp(−19(log n)4). (For n = 106, (8 log n)4 ≈ 1.5 · 108)

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-40
SLIDE 40

A comparison

Theorem (Achlioptas, McSherry 2007) Assume ǫ ≥ (8 log n)4 and bounded entries. Then 1 nMmax ||M − Tr( ME)||F = RMSE ≤ 4

  • r/ǫ.

with probability larger than 1 − exp(−19(log n)4). (For n = 106, (8 log n)4 ≈ 1.5 · 108)

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-41
SLIDE 41

One more comparison

Theorem (Cand´ es, Tao, March 8, 2009) Assume bounded entries and strongly incoherent factors If ǫ ≥ C r (log n)6 then Semidefinite Programming returns, whp, the matrix M. A2’. Strong incoherence

r

  • k=1

U2

ik

≤ µ1 r ,

  • r
  • k=1

UikUjk

  • ≤

µ1 √r ,

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-42
SLIDE 42

One more comparison

Theorem (Cand´ es, Tao, March 8, 2009) Assume bounded entries and strongly incoherent factors If ǫ ≥ C r (log n)6 then Semidefinite Programming returns, whp, the matrix M. A2’. Strong incoherence

r

  • k=1

U2

ik

≤ µ1 r ,

  • r
  • k=1

UikUjk

  • ≤

µ1 √r ,

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-43
SLIDE 43

One more comparison

Theorem (Cand´ es, Tao, March 8, 2009) Assume bounded entries and strongly incoherent factors If ǫ ≥ C r (log n)6 then Semidefinite Programming returns, whp, the matrix M. A2’. Strong incoherence

r

  • k=1

U2

ik

≤ µ1 r ,

  • r
  • k=1

UikUjk

  • ≤

µ1 √r ,

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-44
SLIDE 44

Our approach: Graph theory

movies users 1 1 nα n i a (i, a) ∈ E ⇔ User a rated movie i.

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-45
SLIDE 45

Our approach: Graph theory

movies users 1 1 nα n i a (i, a) ∈ E ⇔ User a rated movie i.

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-46
SLIDE 46

Back to the data

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-47
SLIDE 47

Random r = 4, n = 10000, ǫ = 12.5

10 20 30 40 50 60 70 80 10 20 30 40

σ1 σ2 σ3 σ4

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-48
SLIDE 48

Netflix data (trimmed)

20 40 60 80 100 120 140 20 40 60 80 100 120 140 Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-49
SLIDE 49

Is Neflix a random low-rank matrix?

Compare for coordinate descent (SimonFunk).

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-50
SLIDE 50

Rank = 3

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 Lowrank Netflix Data Random Data U[-1 1] 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 Lowrank Netflix Data Random Data U[-1 1]

D steps steps fit error

  • pred. error

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-51
SLIDE 51

Rank = 4

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 Lowrank Netflix Data Random Data U[-1 1] 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 Lowrank Netflix Data Random Data U[-1 1]

D steps steps fit error

  • pred. error

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-52
SLIDE 52

Rank = 5

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 Lowrank Netflix Data Random Data U[-1 1] 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 Lowrank Netflix Data Random Data U[-1 1]

D steps steps fit error

  • pred. error

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion

slide-53
SLIDE 53

Proofs (blackboard)

Raghunandan Keshavan, Andrea Montanari and Sewoong Oh Matrix Completion