fast quantification of uncertainty and robustness with
play

Fast Quantification of Uncertainty and Robustness with Variational - PowerPoint PPT Presentation

Nov 30, 2023 •131 likes •1.84k views

Fast Quantification of Uncertainty and Robustness with Variational Bayes Tamara Broderick ITT Career Development Assistant Professor, MIT With: Ryan Giordano, Rachael Meager, Jonathan H. Huggins, Michael I. Jordan Bayesian inference

  1. Variational Bayes • VB approximation • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Liebler (KL) divergence: p ( θ | x ) q ( θ ) KL ( q k p ( ·| x )) • VB practical success • point estimates and prediction • fast 3

  2. Variational Bayes • VB approximation • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Liebler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast 3

  3. Variational Bayes • VB approximation • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Leibler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast 3

  4. Variational Bayes • VB approximation • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Leibler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast 3

  5. Variational Bayes • VB approximation • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Leibler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast 3

  6. Variational Bayes • VB approximation • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Leibler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast 3

  7. Variational Bayes • VB approximation • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Leibler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast 3 [Broderick, Boyd, Wibisono, Wilson, Jordan 2013]

  8. Variational Bayes • VB approximation • Approximation for q ∗ ( θ ) p ( θ | x ) posterior • Minimize Kullback-Leibler (KL) divergence: p ( θ | x ) KL ( q k p ( ·| x )) q ∗ ( θ ) • VB practical success • point estimates and prediction • fast, streaming, distributed 3 [Broderick, Boyd, Wibisono, Wilson, Jordan 2013]

  9. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 J Y q ( θ ) = q ( θ j ) ! j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  10. What about uncertainty? • Variational Bayes ! • Mean-field variational Bayes (MFVB) J Y q ( θ ) = q ( θ j ) ! j =1 • Underestimates variance (sometimes severely) • No covariance estimates 4

  11. What about uncertainty? • Variational Bayes ! • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  12. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  13. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  14. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! q ∗ ( θ ) j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  15. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! q ∗ ( θ ) j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  16. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! q ∗ ( θ ) j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [Bishop 2006]

  17. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! q ∗ ( θ ) j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates 4 [MacKay 2003; Bishop 2006; Wang, Titterington 2004; Turner, Sahani 2011]

  18. What about uncertainty? • Variational Bayes q ( θ ) log q ( θ ) Z KL ( q || p ( ·| x )) = p ( θ | x ) d θ ! θ • Mean-field variational Bayes (MFVB) θ 2 p ( θ | x ) J Y q ( θ ) = q ( θ j ) ! q ∗ ( θ ) j =1 • Underestimates variance (sometimes severely) θ 1 • No covariance estimates [MacKay 2003; Bishop 2006; Wang, Titterington 2004; Turner, Sahani 2011] 4 [Fosdick 2013; Dunson 2014; Bardenet, Doucet, Holmes 2015]

  19. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006]

  20. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006]

  21. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006]

  22. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • True posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006]

  23. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006]

  24. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003]

  25. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006]

  26. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006]

  27. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006] [Bishop 2006]

  28. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006] [Bishop 2006]

  29. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006] [Bishop 2006]

  30. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006] [Bishop 2006]

  31. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006] [Bishop 2006]

  32. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006] [Bishop 2006]

  33. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003] [Bishop 2006] [Bishop 2006]

  34. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003]

  35. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003]

  36. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation  d �� d � Σ = dtC p ( ·| x ) ( t ) � [Bishop 2006] dt T � t =0 5 [see also Opper, Winther 2003]

  37. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003]

  38. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003]

  39. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003]

  40. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003]

  41. Linear response • Cumulant-generating function � mean = d C ( t ) := log E e t T θ � dtC ( t ) � ! � t =0 • Exact posterior covariance vs MFVB covariance d 2 d 2 � � � � Σ := dt T dtC p ( ·| x ) ( t ) V := dt T dtC q ∗ ( t ) ! � � � � t =0 t =0 • “Linear response” p ( θ | x ) ! log p t ( θ ) := log p ( θ | x ) + t T θ − C ( t ) , MFVB q ∗ t q ∗ ( θ ) • The LRVB approximation � � d d =: ˆ � � dt T E p t θ t θ Σ = dt T E q ∗ Σ ≈ � � [Bishop 2006] � � t =0 t =0 5 [see also Opper, Winther 2003]

  42. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t 6

  43. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t 6

  44. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t = ( I − V H ) − 1 V 6

  45. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ 6

  46. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ 6

  47. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ 6

  48. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ • Symmetric and positive definite at local min of KL • The LRVB assumption: 6

  49. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ • Symmetric and positive definite at local min of KL • The LRVB assumption: E p t θ ≈ E q ∗ t θ 6

  50. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ • Symmetric and positive definite at local min of KL p ( θ | x ) • The LRVB assumption: E p t θ ≈ E q ∗ t θ q ∗ ( θ ) [Bishop 2006] 6

  51. LRVB estimator � d • LRVB covariance estimate ˆ � t θ Σ := dt T E q ∗ � � t =0 • Suppose exponential family with mean parametrization q t m t ✓ ∂ 2 KL ◆ − 1 � = ( I − V H ) − 1 V ˆ � Σ = � ∂ m ∂ m T � m = m ∗ • Symmetric and positive definite at local min of KL p ( θ | x ) • The LRVB assumption: E p t θ ≈ E q ∗ t θ q ∗ ( θ ) • LRVB estimate is exact when MFVB gives exact mean (e.g. multivariate normal) [Bishop 2006] 6

  52. Microcredit Experiment • Simplified from Meager (2015) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: ! ! • Priors and hyperpriors: 7

  53. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: ! ! • Priors and hyperpriors: 7

  54. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: ! ! • Priors and hyperpriors: 7

  55. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: ! ! • Priors and hyperpriors: 7

  56. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: ! ! • Priors and hyperpriors: 7

  57. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  58. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  59. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  60. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  61. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  62. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  63. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  64. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  65. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: 7

  66. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: ✓ ◆ ✓✓ ◆ ◆ µ k µ iid ∼ N , C τ k τ 7

  67. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: ✓ ◆ ✓✓ ◆ ◆ µ k µ iid ∼ N , C τ k τ iid σ − 2 ∼ Γ ( a, b ) k 7

  68. Microcredit Experiment • Simplified from Meager (2016) • K microcredit trials (Mexico, Mongolia, Bosnia, India, Morocco, Philippines, Ethiopia) • N k businesses in k th site (~900 to ~17K) 1 if microcredit • Profit of n th business at k th site: profit ! indep N ( µ k + T kn τ k , σ 2 k ) y kn ∼ ! • Priors and hyperpriors: ✓ ◆ ✓✓ ◆ ◆ ✓ ◆ ✓✓ ◆ ◆ µ k µ iid µ iid µ 0 , Λ − 1 ∼ N ∼ N , C τ k τ τ τ 0 iid σ − 2 ∼ Γ ( a, b ) C ∼ Sep&LKJ( η , c, d ) k 7

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Fast Robustness Quantification with Variational Bayes Tamara Broderick ITT Career Development

Fast Robustness Quantification with Variational Bayes Tamara Broderick ITT Career Development

Fast Robustness Quantification with Variational Bayes Tamara Broderick ITT Career Development Assistant Professor, MIT With: Ryan Giordano, Rachael Meager, Jonathan Huggins, Michael I. Jordan Bayesian inference Complex, modular

1.45k views • 144 slides
Uncertainty quantification of Antarctic contribution to sea-level rise using the fast Elementary

Uncertainty quantification of Antarctic contribution to sea-level rise using the fast Elementary

European Geosciences Union General Assembly 2017 Uncertainty quantification of Antarctic contribution to sea-level rise using the fast Elementary Thermomechanical Ice Sheet (f.ETISh) model K. Bulthuis 1 , 2 , M. Arnst 1 , F. Pattyn 2 , L. Favier 2

647 views • 28 slides
Semi-intrusive Uncertainty Quantification for Multiscale models Anna Nikishova 1 Alfons Hoekstra 1

Semi-intrusive Uncertainty Quantification for Multiscale models Anna Nikishova 1 Alfons Hoekstra 1

Motivation Uncertainty Quantification Case Study Semi-intrusive Uncertainty Quantification for Multiscale models Anna Nikishova 1 Alfons Hoekstra 1 1 Computational Science Lab Universities of Amsterdam Quantification of Uncertainty: Improving

268 views • 11 slides
Overview of Uncertainty Quantification Algorithm R&D in the DAKOTA Project Michael S. Eldred

Overview of Uncertainty Quantification Algorithm R&D in the DAKOTA Project Michael S. Eldred

Overview of Uncertainty Quantification Algorithm R&D in the DAKOTA Project Michael S. Eldred Optimization and Uncertainty Quantification Dept. Sandia National Laboratories, Albuquerque, NM NIST UQ Workshop Boulder, CO; August 1-4, 2011

512 views • 33 slides
Metamodels in Uncertainty Quantification and Reliability Analysis S. Marelli and B. Sudret Chair

Metamodels in Uncertainty Quantification and Reliability Analysis S. Marelli and B. Sudret Chair

Metamodels in Uncertainty Quantification and Reliability Analysis S. Marelli and B. Sudret Chair of Risk, Safety and Uncertainty Quantification ETH Z urich CEMRACS Summer School on Numerical methods for stochastic models: control,

1.01k views • 82 slides
The Role of Expert Knowledge in Uncertainty Quantification (Are We Adding More Uncertainty (Are

The Role of Expert Knowledge in Uncertainty Quantification (Are We Adding More Uncertainty (Are

Army Conference on Applied Statistics Santa Fe, NM October 25, 2001 The Role of Expert Knowledge in Uncertainty Quantification (Are We Adding More Uncertainty (Are We Adding More Uncertainty or More Understanding?) or More Understanding?)

374 views • 21 slides
Interval Based Finite Elements for Uncertainty Quantification in Engineering Mechanics Rafi L.

Interval Based Finite Elements for Uncertainty Quantification in Engineering Mechanics Rafi L.

Interval Based Finite Elements for Uncertainty Quantification in Engineering Mechanics Rafi L. Muhanna Center for Reliable Engineering Computing (REC) Georgia Institute of Technology ifip -Working Conference on Uncertainty Quantification in

1.57k views • 52 slides
A Non-parametric Approach for Uncertainty Quantification in Elastodynamics S Adhikari

A Non-parametric Approach for Uncertainty Quantification in Elastodynamics S Adhikari

A Non-parametric Approach for Uncertainty Quantification in Elastodynamics S Adhikari Department of Aerospace Engineering, University of Bristol, Bristol, U.K. Email: S.Adhikari@bristol.ac.uk Uncertainty Quantification p.1/43 Newport, RI,

547 views • 43 slides
46.1 Introduction chapter overview: 46. Introduction and Quantification 47. Representation

46.1 Introduction chapter overview: 46. Introduction and Quantification 47. Representation

Foundations of Artificial Intelligence May 24, 2017 46. Uncertainty: Introduction and Quantification Foundations of Artificial Intelligence 46.1 Introduction 46. Uncertainty: Introduction and Quantification 46.2 Probability Theory Malte

213 views • 8 slides
Center for Uncertainty Quantification Thuwal, KSA, Jan. 6, 2015 1 / 39 Our Contribution This

Center for Uncertainty Quantification Thuwal, KSA, Jan. 6, 2015 1 / 39 Our Contribution This

Hybrid Multilevel Monte Carlo Simulation of Stochastic Reaction Networks Alvaro Moraes joint work with Ra ul Tempone and Pedro Vilanova http://sri-uq.kaust.edu.sa Center for Uncertainty Quantification Thuwal, KSA, Jan. 6, 2015 1 / 39 Our

699 views • 39 slides
Advances concerning multiscale methods and uncertainty quantification in E XA -D UNE P. Bastian,

Advances concerning multiscale methods and uncertainty quantification in E XA -D UNE P. Bastian,

W ESTFLISCHE W ILHELMS -U NIVERSITT M NSTER Advances concerning multiscale methods and uncertainty quantification in E XA -D UNE P. Bastian, C. Engwer, J. Fahlke, M. Geveler, D. living knowledge Gddeke, O. Iliev, O. Ippisch, R. Milk, J.

327 views • 32 slides
Towards Bayesian uncertainty quantification for forest models used in the U.K. GHG inventory for

Towards Bayesian uncertainty quantification for forest models used in the U.K. GHG inventory for

Towards Bayesian uncertainty quantification for forest models used in the U.K. GHG inventory for LULUCF Marcel van Oijen & Amanda Thomson Centre for Ecology and Hydrology Edinburgh, U.K. Contents 1. Current methodology used to make the

299 views • 25 slides
THermodynamic Formalism and Uncertainty Quantification Luc Rey-Bellet University of

THermodynamic Formalism and Uncertainty Quantification Luc Rey-Bellet University of

THermodynamic Formalism and Uncertainty Quantification Luc Rey-Bellet University of Massachusetts Amherst Quantissima III, Venice, August 2019 Work supported by NSF and AFOSR 1 Collaborators on related projects Paul Dupuis (Brown

827 views • 34 slides
Quantification and Visualization of Spatial Uncertainty in Isosurfaces for Parametric and

Quantification and Visualization of Spatial Uncertainty in Isosurfaces for Parametric and

Introduction Related work Problem Description and Approach Results and Conclusion Quantification and Visualization of Spatial Uncertainty in Isosurfaces for Parametric and Nonparametric Noise Models Tushar Athawale Department of Computer

1.32k views • 115 slides
Uncertainty Quantification in Materials Modeling Pablo Seleson Oak Ridge National Laboratory

Uncertainty Quantification in Materials Modeling Pablo Seleson Oak Ridge National Laboratory

Uncertainty Quantification in Materials Modeling Pablo Seleson Oak Ridge National Laboratory Miroslav Stoyanov Oak Ridge National Laboratory Clayton G. Webster Oak Ridge National Laboratory Quantification of Uncertainty: Improving Efficiency

679 views • 43 slides
Bayesian calibration and uncertainty quantification of computer models Application to the

Bayesian calibration and uncertainty quantification of computer models Application to the

Bayesian calibration and uncertainty quantification of computer models Application to the simulation of astrophysical radiative shocks Jean Giorla 1 , J. Garnier 2 , . Falize 1 , B. Loupias 1 1 CEA/DAM/DIF, F-91297, Arpajon, France 2

780 views • 30 slides
The Macroeconomics of Microfinance Francisco Buera 1 Joseph Kaboski 2 Yongseok Shin 3 1 Federal

The Macroeconomics of Microfinance Francisco Buera 1 Joseph Kaboski 2 Yongseok Shin 3 1 Federal

The Macroeconomics of Microfinance Francisco Buera 1 Joseph Kaboski 2 Yongseok Shin 3 1 Federal Reserve Bank of Minneapolis, UCLA & NBER 2 University of Notre Dame & NBER 3 Wash U St. Louis & St. Louis Fed January, 2012 Microfinance

1.33k views • 132 slides
Introduction to Micronance November 2007 () Introduction November 2007 1 / 14 The Nature

Introduction to Micronance November 2007 () Introduction November 2007 1 / 14 The Nature

Introduction to Micronance November 2007 () Introduction November 2007 1 / 14 The Nature of Micronance Microcredit: collection of banking practices built around small loans, typically with no collateral Micronance: also includes

498 views • 14 slides
System Administration HW5 - Micro Computer Center weicc, zjlin Computer Center, CS, NCTU 2

System Administration HW5 - Micro Computer Center weicc, zjlin Computer Center, CS, NCTU 2

System Administration HW5 - Micro Computer Center weicc, zjlin Computer Center, CS, NCTU 2 Bonus Computer Center, CS, NCTU 3 Computer Center, CS, NCTU 4 Computer Center, CS, NCTU Requirements (1) - Overview Machines sahome:

702 views • 22 slides
CSCI 446 ARTIFICIAL INTELLIGENCE FINAL EXAM STUDY OUTLINE Introduction to Artificial

CSCI 446 ARTIFICIAL INTELLIGENCE FINAL EXAM STUDY OUTLINE Introduction to Artificial

CSCI 446 ARTIFICIAL INTELLIGENCE FINAL EXAM STUDY OUTLINE Introduction to Artificial Intelligence I. Definitions of Artificial Intelligence A. Acting Like Humans -- Turing Test B. Thinking Like Humans -- Cognitive Modeling C. Thinking

309 views • 6 slides
2011 FINANCIAL RESULTS KCB INVESTOR AND MEDIA BRIEFING 1 ST MARCH 2012 NAIROBI M R . P E T E R

2011 FINANCIAL RESULTS KCB INVESTOR AND MEDIA BRIEFING 1 ST MARCH 2012 NAIROBI M R . P E T E R

2011 FINANCIAL RESULTS KCB INVESTOR AND MEDIA BRIEFING 1 ST MARCH 2012 NAIROBI M R . P E T E R W . M U T H O K A , M B S , K C B G R O U P C H A I R M A N D R . M A R T I N O D U O R - O T I E N O , C E O - K C B G R O U P M R . J O

724 views • 22 slides
Introduction to Your Introduction to Your Microeconomics Course Microeconomics Course Welcome

Introduction to Your Introduction to Your Microeconomics Course Microeconomics Course Welcome

The Economics Department, UMR Presents: Introduction to Your Introduction to Your Microeconomics Course Microeconomics Course Welcome to what I hope will be a Welcome to what I hope will be a moment of excitement and moment of excitement

493 views • 8 slides
Microeconomic Approaches for Quantifying the Value of Remotely Sensed Information Yusuke

Microeconomic Approaches for Quantifying the Value of Remotely Sensed Information Yusuke

Microeconomic Approaches for Quantifying the Value of Remotely Sensed Information Yusuke Kuwayama Resources for the Future Outline 1. Introduction 2. What is the Value of Information (VOI)? 3. Examples of microeconomic VOI studies 4.

337 views • 11 slides
CSC304 Algorithmic Game Theory & Mechanism Design Nisarg Shah CSC304 - Nisarg Shah 1

CSC304 Algorithmic Game Theory & Mechanism Design Nisarg Shah CSC304 - Nisarg Shah 1

CSC304 Algorithmic Game Theory & Mechanism Design Nisarg Shah CSC304 - Nisarg Shah 1 Introduction Instructor: Nisarg Shah (/~nisarg, nisarg@cs) Guest lectures: Prof. Allan Borodin TA: Tyrone Strangway (/~tyrone, tyrone@cs)

585 views • 29 slides

More recommend