A Tale of Two Theories: A Tale of Two Theories: Reconciling - PowerPoint PPT Presentation

A Tale of Two Theories: A Tale of Two Theories: Reconciling Reconciling random matrix theory and shrinkage estimation random matrix theory and shrinkage estimation as methods for covariance matrix estimation as methods for covariance matrix estimation Brian Rowe Brian Rowe Vice President, Portfolio Analytics Vice President, Portfolio Analytics Bank of America Merrill Lynch Bank of America Merrill Lynch July, 2009 July, 2009

Disclaimer Disclaimer The opinions expressed in this presentation The opinions expressed in this presentation are those of the author alone and do not are those of the author alone and do not necessarily reflect the views of Bank of necessarily reflect the views of Bank of America Merrill Lynch, its subsidiaries, or America Merrill Lynch, its subsidiaries, or affiliates. affiliates. “Bank of America Merrill Lynch” is the marketing name for the global banking and global markets businesses of Bank of America Corporation. Lending, derivatives, and other commercial banking activities are performed globally by banking affiliates of Bank of America Corporation, including Bank of America, N.A., member FDIC. Securities, stra- tegic advisory, and other investment banking activities are performed globally by in- vestment banking affiliates of Bank of America Corporation (“Investment Banking Affili- ates”), including, in the United States, Banc of America Securities LLC and Merrill Lynch, Pierce, Fenner & Smith Incorporated, which are both registered broker-dealers and members of FINRA and SIPC, and, in other jurisdictions, locally registered entities. Investment products offered by Investment Banking Affiliates: Are Not FDIC Insured * May Lose Value * Are Not Bank Guaranteed.

Overview Overview ● Motivation Motivation ● Random Matrix Theory Random Matrix Theory ● Shrinkage Estimation Shrinkage Estimation ● Measuring Effectiveness Measuring Effectiveness ● Kullback-Leibler distance Kullback-Leibler distance ● Financial measures Financial measures ● Reconciliation Reconciliation

Motivation Motivation ● Sample covariance != true covariance matrix Sample covariance != true covariance matrix ● Estimation error is large when !(T >> N) Estimation error is large when !(T >> N) ● Large portfolios Large portfolios ● Monthly time frame Monthly time frame ● Need a good estimate of covariance matrix Need a good estimate of covariance matrix

Approaches Approaches Physics: Random matrix theory ● Eigenvalue Eigenvalue distribution distribution

Approaches Approaches Physics: Random matrix theory ● Eigenvalue Eigenvalue distribution distribution ● Null hypothesis Null hypothesis

Approaches Approaches Physics: Random matrix theory ● Eigenvalue Eigenvalue distribution distribution ● Null hypothesis Null hypothesis ● Remove noise Remove noise component component

Approaches Approaches Statistics: Shrinkage Estimation ● Central limit Central limit theorem theorem

Approaches Approaches Statistics: Shrinkage Estimation ● Central limit Central limit theorem theorem ● Weighted average Weighted average α F + (1- α ) S α F + (1- α ) S

Approaches Approaches Statistics: Shrinkage Estimation ● Central limit Central limit theorem theorem ● Weighted average Weighted average α F + (1- α ) S α F + (1- α ) S ● Reduced Reduced estimation error estimation error

Approaches Approaches Which is Right?

Random Matrix Theory Random Matrix Theory ● Eigenvalue distribution of random matrices Eigenvalue distribution of random matrices is defined by the Marcenko-Pastur limit is defined by the Marcenko-Pastur limit 2   max − min − Q =  2   max / min = σ 2  1 ±  2 1 Q  ● Sample correlation matrices can be filtered Sample correlation matrices can be filtered to remove this noise to remove this noise ● The reconstructed matrix is then used in The reconstructed matrix is then used in portfolio optimization portfolio optimization

Random Matrix Theory Random Matrix Theory Marcenko-Pastur Distributions ● Random matrix with Random matrix with normal distribution; normal distribution; N=1000, T=4000 N=1000, T=4000 ● Random matrix with Random matrix with normal distribution; normal distribution; N=250, T=1000 N=250, T=1000 ● Random matrix with Random matrix with normal distribution; normal distribution; N=50, T=200 N=50, T=200

Random Matrix Theory Random Matrix Theory Marcenko-Pastur Distributions ● Random matrix with Random matrix with normal distribution; normal distribution; N=1000, T=4000 N=1000, T=4000 ● Random matrix with Random matrix with normal distribution; normal distribution; N=250, T=1000 N=250, T=1000 ● Random matrix with Random matrix with normal distribution; normal distribution; N=50, T=200 N=50, T=200

Random Matrix Theory Random Matrix Theory Marcenko-Pastur Distributions ● Random matrix with Random matrix with normal distribution; normal distribution; N=1000, T=4000 N=1000, T=4000 ● Random matrix with Random matrix with normal distribution; normal distribution; N=250, T=1000 N=250, T=1000 ● Random matrix with Random matrix with normal distribution; normal distribution; N=50, T=200 N=50, T=200

Random Matrix Theory Random Matrix Theory Fitting the Null Hypothesis ● Daily S&P 500; N=384, Daily S&P 500; N=384, T=1200 T=1200 ● Daily S&P 500 subset; Daily S&P 500 subset; N=75, T=200 N=75, T=200 ● Shuffled S&P 500; N=75, Shuffled S&P 500; N=75, T=200 T=200 Q = 2.072958 σ = 0.8152044

Random Matrix Theory Random Matrix Theory Fitting the Null Hypothesis ● Daily S&P 500; N=384, Daily S&P 500; N=384, T=1200 T=1200 ● Daily S&P 500 subset; Daily S&P 500 subset; N=75, T=200 N=75, T=200 ● Shuffled S&P 500; Shuffled S&P 500; N=75, T=200 N=75, T=200 Q = 1.768204 σ = 0.6321195

Random Matrix Theory Random Matrix Theory Fitting the Null Hypothesis ● Daily S&P 500; N=384, Daily S&P 500; N=384, T=1200 T=1200 ● Daily S&P 500 subset; Daily S&P 500 subset; N=75, T=200 N=75, T=200 ● Shuffled S&P 500; N=75, Shuffled S&P 500; N=75, T=200 T=200 Q = 2.514132 σ = 1.019011

Shrinkage Estimation Shrinkage Estimation ● James-Stein revealed that a global mean James-Stein revealed that a global mean exists exists ● Shrinking samples toward a global mean Shrinking samples toward a global mean improves accuracy of estimation improves accuracy of estimation ● This can be applied to covariance This can be applied to covariance matrices matrices

Shrinkage Estimation Shrinkage Estimation What is the global mean? ● The true mean is unknown The true mean is unknown ● Many candidates exist for covariance Many candidates exist for covariance ● Identity matrix Identity matrix ● Constant correlation matrix Constant correlation matrix ● Biased estimator (e.g. Barra) Biased estimator (e.g. Barra)

Shrinkage Estimation Shrinkage Estimation Shrinkage Intensity ● Use a single value or Use a single value or calculate per iteration calculate per iteration ● Ledoit & Wolf propose Ledoit & Wolf propose optimal coefficient optimal coefficient =  T =− 

Filtering Correlation Matrices Filtering Correlation Matrices RMT reconstructs Shrinkage estimation RMT reconstructs Shrinkage estimation correlation matrix takes a weighted correlation matrix takes a weighted from the empirical average between the from the empirical average between the correlation matrix by sample covariance correlation matrix by sample covariance replacing all and a global mean replacing all and a global mean eigenvalues in noise using a calculated eigenvalues in noise using a calculated part of spectrum with shrinkage constant part of spectrum with shrinkage constant their mean their mean

Does It Work? Does It Work? ● How do you measure effectiveness? How do you measure effectiveness? ● Again, two approaches Again, two approaches ● Kullback-Leibler distance Kullback-Leibler distance ● Out of sample portfolio returns Out of sample portfolio returns ● Which will you believe? Which will you believe?

Kullback-Leibler Distance Kullback-Leibler Distance ● KL distance measures the entropy KL distance measures the entropy between two probability density functions between two probability density functions ● Not a true distance - but still useful! Not a true distance - but still useful! ● Triangle inequality is not satisfied Triangle inequality is not satisfied ● Not symmetric Not symmetric ● Can measure information content and Can measure information content and stability stability

Kullback-Leibler Distance Kullback-Leibler Distance Theoretical Limit Theoretical Limit

Kullback-Leibler Distance Kullback-Leibler Distance Empirical Results Empirical Results