Bayesian Estimation of Low-rank Matrices Pierre Alquier Journes de - PowerPoint PPT Presentation

Bayesian Estimation of Low-rank Matrices : Introduction Bayesian Reduced Rank Regression Bayesian Matrix Completion Bayesian Estimation of Low-rank Matrices Pierre Alquier Journées de Statistique du Sud, Barcelona 09/06/2014 Pierre Alquier Bayesian Estimation of Low-rank Matrices

Bayesian Estimation of Low-rank Matrices : Introduction Bayesian Reduced Rank Regression Bayesian Matrix Completion Introduction Many problems arising in statistics / signal processing involve estimation / recovery of low-rank matrices : PCA, matrix completion for recommender systems, reduced rank regression / multitask learning, video processing : separation of moving object and static background, quantum statistics, ... Pierre Alquier Bayesian Estimation of Low-rank Matrices

Bayesian Estimation of Low-rank Matrices : Introduction Bayesian Reduced Rank Regression Bayesian Matrix Completion Reduced rank regression Consider m regression models Y j = XB j + ε j with the same regressors X , Y j is a column vector in R n , X is n × p . Pierre Alquier Bayesian Estimation of Low-rank Matrices

Bayesian Estimation of Low-rank Matrices : Introduction Bayesian Reduced Rank Regression Bayesian Matrix Completion Reduced rank regression Consider m regression models Y j = XB j + ε j with the same regressors X , Y j is a column vector in R n , X is n × p .        Y 1  B 1  ε 1 Y m = X B m + . . . . . . . . . ε m    � �� � � �� � � �� � Y B E Pierre Alquier Bayesian Estimation of Low-rank Matrices

Bayesian Estimation of Low-rank Matrices : Introduction Bayesian Reduced Rank Regression Bayesian Matrix Completion Reduced rank regression Consider m regression models Y j = XB j + ε j with the same regressors X , Y j is a column vector in R n , X is n × p .        Y 1  B 1  ε 1 Y m = X B m + . . . . . . . . . ε m    � �� � � �� � � �� � Y B E Y = XB + ε where Y is n × m , X is n × p , B is p × m . Pierre Alquier Bayesian Estimation of Low-rank Matrices

Bayesian Estimation of Low-rank Matrices : Introduction Bayesian Reduced Rank Regression Bayesian Matrix Completion Reduced rank regression Consider m regression models Y j = XB j + ε j with the same regressors X , Y j is a column vector in R n , X is n × p .        Y 1  B 1  ε 1 Y m = X B m + . . . . . . . . . ε m    � �� � � �� � � �� � Y B E Economic theory : Y = XB + ε where Y is n × m , rank ( B ) ≪ min ( p , m ) . X is n × p , B is p × m . Pierre Alquier Bayesian Estimation of Low-rank Matrices

Bayesian Estimation of Low-rank Matrices : Introduction Bayesian Reduced Rank Regression Bayesian Matrix Completion Reduced rank regression Consider m regression models Y j = XB j + ε j with the same regressors X , Y j is a column vector in R n , X is n × p .        Y 1  B 1  ε 1 Y m = X B m + . . . . . . . . . ε m    � �� � � �� � � �� � Y B E Economic theory : Y = XB + ε where Y is n × m , rank ( B ) ≪ min ( p , m ) . X is n × p , Example : B is p × m . M. R. Gibbons & W. Ferson (1985). Testing asset pricing models with changing expectations and an unobserved market portfolio. Journal of Financial Economics 14 217–236. Pierre Alquier Bayesian Estimation of Low-rank Matrices

Bayesian Estimation of Low-rank Matrices : Introduction Bayesian Reduced Rank Regression Bayesian Matrix Completion Pierre Alquier Bayesian Estimation of Low-rank Matrices

Bayesian Estimation of Low-rank Matrices : Introduction Bayesian Reduced Rank Regression Bayesian Matrix Completion StarWars I StarWars IV π . . . Claire 4 ? 3 . . . Nial ? 4 ? . . . Brendon 2 ? 4 . . . Andrew ? 4 ? . . . Adrian 1 ? ? . . . Jason 2 4 5 . . . Pierre 3 5 5 . . . . . . . ... . . . . . . . . Pierre Alquier Bayesian Estimation of Low-rank Matrices

Bayesian Estimation of Low-rank Matrices : Introduction Bayesian Reduced Rank Regression Bayesian Matrix Completion StarWars I StarWars IV π . . . Claire 4 ? 3 . . . Nial ? 4 ? . . . Brendon 2 ? 4 . . . Andrew ? 4 ? . . . Adrian 1 ? ? . . . Jason 2 4 5 . . . Pierre 3 5 5 . . . . . . . ... . . . . . . . . J. Bennett & S. Lanning (2007). The Netflix Prize. Proceedings of KDD Cup and Workshop’07 3–6. Pierre Alquier Bayesian Estimation of Low-rank Matrices

Bayesian Estimation of Low-rank Matrices : Introduction Bayesian Reduced Rank Regression Bayesian Matrix Completion StarWars I StarWars IV π . . . Claire 4 ? 3 . . . Nial ? 4 ? . . . Brendon 2 ? 4 . . . Andrew ? 4 ? . . . Adrian 1 ? ? . . . Jason 2 4 5 . . . Pierre 3 5 5 . . . . . . . ... . . . . . . . . J. Bennett & S. Lanning (2007). The Netflix Prize. Proceedings of KDD Cup and Workshop’07 3–6. Parameter : B = ( B i , j ) 1 ≤ i ≤ p , 1 ≤ j ≤ m where B i , j is the rating of user i to movie j . Pierre Alquier Bayesian Estimation of Low-rank Matrices

Bayesian Estimation of Low-rank Matrices : Introduction Bayesian Reduced Rank Regression Bayesian Matrix Completion StarWars I StarWars IV π . . . Claire 4 ? 3 . . . Nial ? 4 ? . . . Brendon 2 ? 4 . . . Andrew ? 4 ? . . . Adrian 1 ? ? . . . Jason 2 4 5 . . . Pierre 3 5 5 . . . . . . . ... . . . . . . . . J. Bennett & S. Lanning (2007). The Netflix Prize. Proceedings of KDD Cup and Workshop’07 3–6. Parameter : B = ( B i , j ) 1 ≤ i ≤ p , 1 ≤ j ≤ m where B i , j is the rating of user i to movie j . Obs. : Y i , j = B i , j + ε i , j , ( i , j ) ∈ I � { 1 , . . . , n } × { 1 , . . . , p } . Pierre Alquier Bayesian Estimation of Low-rank Matrices

Bayesian Estimation of Low-rank Matrices : Introduction Bayesian Reduced Rank Regression Bayesian Matrix Completion StarWars I StarWars IV π . . . Claire 4 ? 3 . . . Nial ? 4 ? . . . Brendon 2 ? 4 . . . Andrew ? 4 ? . . . Adrian 1 ? ? . . . Jason 2 4 5 . . . Pierre 3 5 5 . . . . . . . ... . . . . . . . . J. Bennett & S. Lanning (2007). The Netflix Prize. Proceedings of KDD Cup and Workshop’07 3–6. Parameter : B = ( B i , j ) 1 ≤ i ≤ p , 1 ≤ j ≤ m where B i , j is the rating of user i to movie j . Obs. : Y i , j = B i , j + ε i , j , ( i , j ) ∈ I � { 1 , . . . , n } × { 1 , . . . , p } . Objective : estimate B by ˆ B , and advertise to i the movies j with ˆ B i , j ≃ 5. Pierre Alquier Bayesian Estimation of Low-rank Matrices

Bayesian Estimation of Low-rank Matrices : Introduction Bayesian Reduced Rank Regression Bayesian Matrix Completion StarWars I StarWars IV π . . . Claire 4 ? 3 . . . Nial ? 4 ? . . . Brendon 2 ? 4 . . . Andrew ? 4 ? . . . Adrian 1 ? ? . . . Jason 2 4 5 . . . Pierre 3 5 5 . . . . . . . ... . . . . . . . . J. Bennett & S. Lanning (2007). The Netflix Prize. Proceedings of KDD Cup and Workshop’07 3–6. Parameter : B = ( B i , j ) 1 ≤ i ≤ p , 1 ≤ j ≤ m where B i , j is the rating of user i to movie j . Obs. : Y i , j = B i , j + ε i , j , ( i , j ) ∈ I � { 1 , . . . , n } × { 1 , . . . , p } . Objective : estimate B by ˆ B , and advertise to i the movies j with ˆ B i , j ≃ 5. Assumption : rank ( B ) ≪ min ( m , p ) . Pierre Alquier Bayesian Estimation of Low-rank Matrices

Bayesian Estimation of Low-rank Matrices : Introduction Bayesian Reduced Rank Regression Bayesian Matrix Completion (Non-Bayesian) estimation Usually : fit the data subject to a constraint, or penalty : rank ( B ) . Feasible in reduced-rank regression, e.g. : F. Bunea, Y. She & M. Wegkamp (2011). Optimal selection of reduced rank estimators of high-dimensional matrices. The Annals of Statistics 39 1282–1309. Pierre Alquier Bayesian Estimation of Low-rank Matrices

Bayesian Estimation of Low-rank Matrices : Introduction Bayesian Reduced Rank Regression Bayesian Matrix Completion (Non-Bayesian) estimation Usually : fit the data subject to a constraint, or penalty : rank ( B ) . Feasible in reduced-rank regression, e.g. : F. Bunea, Y. She & M. Wegkamp (2011). Optimal selection of reduced rank estimators of high-dimensional matrices. The Annals of Statistics 39 1282–1309. �� 2 � � 1 � B � ∗ = Tr B T B the nuclear norm, leads to feasible algorithms in matrix completion, e.g. : E. Candès & Y. Plan (2009). Matrix completion with noise. Proceedings of the IEEE 98 925–936. (among others). Pierre Alquier Bayesian Estimation of Low-rank Matrices

Bayesian Estimation of Low-rank Matrices : Introduction Bayesian Reduced Rank Regression Bayesian Matrix Completion Bayesian estimators - known rank Survey of Bayesian estimation in econometrics when rank ( B ) = k is known : J. Geweke (1996). Bayesian reduced rank regression in econometrics. Journal of Econometrics 75 121–146. Pierre Alquier Bayesian Estimation of Low-rank Matrices