fast uncertainty estimates and bayesian model averaging
play

FAST UNCERTAINTY ESTIMATES AND BAYESIAN MODEL AVERAGING OF DNNS - PowerPoint PPT Presentation

1 FAST UNCERTAINTY ESTIMATES AND BAYESIAN MODEL AVERAGING OF DNNS WESLEY MADDOX JOINT WORK WITH TIMUR GARIPOV , PAVEL IZMAILOV , DMITRY VETROV , ANDREW GORDON WILSON 2 SUMMARY Stochastic Weight Averaging (Izmailov et al, UAI,


  1. � 1 FAST UNCERTAINTY ESTIMATES AND BAYESIAN MODEL AVERAGING OF DNNS WESLEY MADDOX JOINT WORK WITH TIMUR GARIPOV , PAVEL IZMAILOV , DMITRY VETROV , ANDREW GORDON WILSON

  2. � 2 SUMMARY ‣ Stochastic Weight Averaging (Izmailov et al, UAI, 2018) computes first moment of weights given from SGD iterates with a modified learning rate schedule. ‣ We propose to keep the variance as well to form a Gaussian approximation in weight space. ‣ Sample from Gaussian to compute Bayesian model averages and estimate uncertainty. ‣ Theoretically motivated from results on SGD & relation of iterates to Gaussian distribution (Ruppert, 1992 and Mandt et al, 2017).

  3. <latexit sha1_base64="uGLxWtCmfm/tnJsHfMg2JigS3sc=">ACBnicbVDLSgNBEJyNrxhfqx5FGAxCAhJ2RdBj0IsniWgekF3C7GSDJndHWZ6xSXm5MVf8eJBEa9+gzf/xsnjoIkFDUVN91dgRcg+N8W5mFxaXlexqbm19Y3PL3t6p6ThRlFVpLGLVCIhmgkesChwEa0jFSBgIVg/6FyO/fseU5nF0C6lkfki6Ee9wSsBILXtfFjzoMSAPaRF7REoV3+OrghcmR/im2LzTskZA8Td0ryaIpKy/7y2jFNQhYBFUTrputI8AdEAaeCDXNeopktE+6rGloREKm/cH4jSE+NEobd2JlKgI8Vn9PDEiodRoGpjMk0NOz3kj8z2sm0DnzBzySCbCIThZ1EoEhxqNMcJsrRkGkhCquLkV0x5RhIJLmdCcGdfnie145LrlNzrk3z5fBpHFu2hA1RALjpFZXSJKqiKHpEz+gVvVlP1ov1bn1MWjPWdGYX/YH1+QM4sZen</latexit> <latexit sha1_base64="uGLxWtCmfm/tnJsHfMg2JigS3sc=">ACBnicbVDLSgNBEJyNrxhfqx5FGAxCAhJ2RdBj0IsniWgekF3C7GSDJndHWZ6xSXm5MVf8eJBEa9+gzf/xsnjoIkFDUVN91dgRcg+N8W5mFxaXlexqbm19Y3PL3t6p6ThRlFVpLGLVCIhmgkesChwEa0jFSBgIVg/6FyO/fseU5nF0C6lkfki6Ee9wSsBILXtfFjzoMSAPaRF7REoV3+OrghcmR/im2LzTskZA8Td0ryaIpKy/7y2jFNQhYBFUTrputI8AdEAaeCDXNeopktE+6rGloREKm/cH4jSE+NEobd2JlKgI8Vn9PDEiodRoGpjMk0NOz3kj8z2sm0DnzBzySCbCIThZ1EoEhxqNMcJsrRkGkhCquLkV0x5RhIJLmdCcGdfnie145LrlNzrk3z5fBpHFu2hA1RALjpFZXSJKqiKHpEz+gVvVlP1ov1bn1MWjPWdGYX/YH1+QM4sZen</latexit> <latexit sha1_base64="uGLxWtCmfm/tnJsHfMg2JigS3sc=">ACBnicbVDLSgNBEJyNrxhfqx5FGAxCAhJ2RdBj0IsniWgekF3C7GSDJndHWZ6xSXm5MVf8eJBEa9+gzf/xsnjoIkFDUVN91dgRcg+N8W5mFxaXlexqbm19Y3PL3t6p6ThRlFVpLGLVCIhmgkesChwEa0jFSBgIVg/6FyO/fseU5nF0C6lkfki6Ee9wSsBILXtfFjzoMSAPaRF7REoV3+OrghcmR/im2LzTskZA8Td0ryaIpKy/7y2jFNQhYBFUTrputI8AdEAaeCDXNeopktE+6rGloREKm/cH4jSE+NEobd2JlKgI8Vn9PDEiodRoGpjMk0NOz3kj8z2sm0DnzBzySCbCIThZ1EoEhxqNMcJsrRkGkhCquLkV0x5RhIJLmdCcGdfnie145LrlNzrk3z5fBpHFu2hA1RALjpFZXSJKqiKHpEz+gVvVlP1ov1bn1MWjPWdGYX/YH1+QM4sZen</latexit> <latexit sha1_base64="uGLxWtCmfm/tnJsHfMg2JigS3sc=">ACBnicbVDLSgNBEJyNrxhfqx5FGAxCAhJ2RdBj0IsniWgekF3C7GSDJndHWZ6xSXm5MVf8eJBEa9+gzf/xsnjoIkFDUVN91dgRcg+N8W5mFxaXlexqbm19Y3PL3t6p6ThRlFVpLGLVCIhmgkesChwEa0jFSBgIVg/6FyO/fseU5nF0C6lkfki6Ee9wSsBILXtfFjzoMSAPaRF7REoV3+OrghcmR/im2LzTskZA8Td0ryaIpKy/7y2jFNQhYBFUTrputI8AdEAaeCDXNeopktE+6rGloREKm/cH4jSE+NEobd2JlKgI8Vn9PDEiodRoGpjMk0NOz3kj8z2sm0DnzBzySCbCIThZ1EoEhxqNMcJsrRkGkhCquLkV0x5RhIJLmdCcGdfnie145LrlNzrk3z5fBpHFu2hA1RALjpFZXSJKqiKHpEz+gVvVlP1ov1bn1MWjPWdGYX/YH1+QM4sZen</latexit> <latexit sha1_base64="Mf9KAZJ8VrK1OBmaJH2lqNu3TI=">ACMHicbZDLSgMxFIYz3q23qks3wSK0qGVGBF16WagLpYJVoVPLmTS1wcxMSM6IZewjufFRdKOgiFufwrTOwtsPgZ/vnMPJ+QMlhUHXfXYGBoeGR0bHxnMTk1PTM/nZuVMTJ5rxKotlrM8DMFyKiFdRoOTnSnMIA8nPgqvdXv3smsj4ugEO4rXQ7iMREswQIsa+T1V9LHNEW47JeqDUjq+oUcZa6SH25XuCi3u/wAlukx9aXc0gR6ULtJVr1tq5Atu2e2L/jVeZgokU6WRf/CbMUtCHiGTYEzNcxXWU9AomOTdnJ8YroBdwSWvWRtByE097R/cpUuWNGkr1vZFSPv0+0QKoTGdMLCdIWDb/K714H+1WoKtzXoqIpUgj9jXolYiKca0lx5tCs0Zyo41wLSwf6WsDRoY2oxzNgTv98l/zela2XPL3vF6YWsni2OMLJBFUiQe2SBbZJ9USJUwckceyQt5de6dJ+fNef9qHXCymXnyQ87HJ0gRpzc=</latexit> <latexit sha1_base64="Mf9KAZJ8VrK1OBmaJH2lqNu3TI=">ACMHicbZDLSgMxFIYz3q23qks3wSK0qGVGBF16WagLpYJVoVPLmTS1wcxMSM6IZewjufFRdKOgiFufwrTOwtsPgZ/vnMPJ+QMlhUHXfXYGBoeGR0bHxnMTk1PTM/nZuVMTJ5rxKotlrM8DMFyKiFdRoOTnSnMIA8nPgqvdXv3smsj4ugEO4rXQ7iMREswQIsa+T1V9LHNEW47JeqDUjq+oUcZa6SH25XuCi3u/wAlukx9aXc0gR6ULtJVr1tq5Atu2e2L/jVeZgokU6WRf/CbMUtCHiGTYEzNcxXWU9AomOTdnJ8YroBdwSWvWRtByE097R/cpUuWNGkr1vZFSPv0+0QKoTGdMLCdIWDb/K714H+1WoKtzXoqIpUgj9jXolYiKca0lx5tCs0Zyo41wLSwf6WsDRoY2oxzNgTv98l/zela2XPL3vF6YWsni2OMLJBFUiQe2SBbZJ9USJUwckceyQt5de6dJ+fNef9qHXCymXnyQ87HJ0gRpzc=</latexit> <latexit sha1_base64="Mf9KAZJ8VrK1OBmaJH2lqNu3TI=">ACMHicbZDLSgMxFIYz3q23qks3wSK0qGVGBF16WagLpYJVoVPLmTS1wcxMSM6IZewjufFRdKOgiFufwrTOwtsPgZ/vnMPJ+QMlhUHXfXYGBoeGR0bHxnMTk1PTM/nZuVMTJ5rxKotlrM8DMFyKiFdRoOTnSnMIA8nPgqvdXv3smsj4ugEO4rXQ7iMREswQIsa+T1V9LHNEW47JeqDUjq+oUcZa6SH25XuCi3u/wAlukx9aXc0gR6ULtJVr1tq5Atu2e2L/jVeZgokU6WRf/CbMUtCHiGTYEzNcxXWU9AomOTdnJ8YroBdwSWvWRtByE097R/cpUuWNGkr1vZFSPv0+0QKoTGdMLCdIWDb/K714H+1WoKtzXoqIpUgj9jXolYiKca0lx5tCs0Zyo41wLSwf6WsDRoY2oxzNgTv98l/zela2XPL3vF6YWsni2OMLJBFUiQe2SBbZJ9USJUwckceyQt5de6dJ+fNef9qHXCymXnyQ87HJ0gRpzc=</latexit> <latexit sha1_base64="Mf9KAZJ8VrK1OBmaJH2lqNu3TI=">ACMHicbZDLSgMxFIYz3q23qks3wSK0qGVGBF16WagLpYJVoVPLmTS1wcxMSM6IZewjufFRdKOgiFufwrTOwtsPgZ/vnMPJ+QMlhUHXfXYGBoeGR0bHxnMTk1PTM/nZuVMTJ5rxKotlrM8DMFyKiFdRoOTnSnMIA8nPgqvdXv3smsj4ugEO4rXQ7iMREswQIsa+T1V9LHNEW47JeqDUjq+oUcZa6SH25XuCi3u/wAlukx9aXc0gR6ULtJVr1tq5Atu2e2L/jVeZgokU6WRf/CbMUtCHiGTYEzNcxXWU9AomOTdnJ8YroBdwSWvWRtByE097R/cpUuWNGkr1vZFSPv0+0QKoTGdMLCdIWDb/K714H+1WoKtzXoqIpUgj9jXolYiKca0lx5tCs0Zyo41wLSwf6WsDRoY2oxzNgTv98l/zela2XPL3vF6YWsni2OMLJBFUiQe2SBbZJ9USJUwckceyQt5de6dJ+fNef9qHXCymXnyQ87HJ0gRpzc=</latexit> <latexit sha1_base64="tR6QRGiHr0gv/IN6f3L8/n2xj4c=">ACEHicbVDLSgMxFM34rPVdekmWMTWRZkRQTdCUQSXFewD2jpk0rQNzcyE5I4wTPsJbvwVNy4UcevSnX9j2s5CWw9cODnXnLv8aTgGmz721pYXFpeWc2sZdc3Nre2czu7NR1GirIqDUWoGh7RTPCAVYGDYA2pGPE9were4Grs1x+Y0jwM7iCWrO2TXsC7nBIwkps7koX4/ngYF/EFvnYTWhBnwExwqhgrOH0WSy6ubxdsifA8RJSR6lqLi5r1YnpJHPAqCaN10bAnthCjgVLBRthVpJgkdkB5rGhoQn+l2MjlohA+N0sHdUJkKAE/U3xMJ8bWOfc90+gT6etYbi/95zQi65+2EBzICFtDpR91IYAjxOB3c4YpRELEhCpudsW0TxShYDLMmhCc2ZPnSe2k5Ngl5/Y0X75M48igfXSACshBZ6iMblAFVRFj+gZvaI368l6sd6tj2nrgpXO7KE/sD5/AITVm6I=</latexit> <latexit sha1_base64="tR6QRGiHr0gv/IN6f3L8/n2xj4c=">ACEHicbVDLSgMxFM34rPVdekmWMTWRZkRQTdCUQSXFewD2jpk0rQNzcyE5I4wTPsJbvwVNy4UcevSnX9j2s5CWw9cODnXnLv8aTgGmz721pYXFpeWc2sZdc3Nre2czu7NR1GirIqDUWoGh7RTPCAVYGDYA2pGPE9were4Grs1x+Y0jwM7iCWrO2TXsC7nBIwkps7koX4/ngYF/EFvnYTWhBnwExwqhgrOH0WSy6ubxdsifA8RJSR6lqLi5r1YnpJHPAqCaN10bAnthCjgVLBRthVpJgkdkB5rGhoQn+l2MjlohA+N0sHdUJkKAE/U3xMJ8bWOfc90+gT6etYbi/95zQi65+2EBzICFtDpR91IYAjxOB3c4YpRELEhCpudsW0TxShYDLMmhCc2ZPnSe2k5Ngl5/Y0X75M48igfXSACshBZ6iMblAFVRFj+gZvaI368l6sd6tj2nrgpXO7KE/sD5/AITVm6I=</latexit> <latexit sha1_base64="tR6QRGiHr0gv/IN6f3L8/n2xj4c=">ACEHicbVDLSgMxFM34rPVdekmWMTWRZkRQTdCUQSXFewD2jpk0rQNzcyE5I4wTPsJbvwVNy4UcevSnX9j2s5CWw9cODnXnLv8aTgGmz721pYXFpeWc2sZdc3Nre2czu7NR1GirIqDUWoGh7RTPCAVYGDYA2pGPE9were4Grs1x+Y0jwM7iCWrO2TXsC7nBIwkps7koX4/ngYF/EFvnYTWhBnwExwqhgrOH0WSy6ubxdsifA8RJSR6lqLi5r1YnpJHPAqCaN10bAnthCjgVLBRthVpJgkdkB5rGhoQn+l2MjlohA+N0sHdUJkKAE/U3xMJ8bWOfc90+gT6etYbi/95zQi65+2EBzICFtDpR91IYAjxOB3c4YpRELEhCpudsW0TxShYDLMmhCc2ZPnSe2k5Ngl5/Y0X75M48igfXSACshBZ6iMblAFVRFj+gZvaI368l6sd6tj2nrgpXO7KE/sD5/AITVm6I=</latexit> <latexit sha1_base64="tR6QRGiHr0gv/IN6f3L8/n2xj4c=">ACEHicbVDLSgMxFM34rPVdekmWMTWRZkRQTdCUQSXFewD2jpk0rQNzcyE5I4wTPsJbvwVNy4UcevSnX9j2s5CWw9cODnXnLv8aTgGmz721pYXFpeWc2sZdc3Nre2czu7NR1GirIqDUWoGh7RTPCAVYGDYA2pGPE9were4Grs1x+Y0jwM7iCWrO2TXsC7nBIwkps7koX4/ngYF/EFvnYTWhBnwExwqhgrOH0WSy6ubxdsifA8RJSR6lqLi5r1YnpJHPAqCaN10bAnthCjgVLBRthVpJgkdkB5rGhoQn+l2MjlohA+N0sHdUJkKAE/U3xMJ8bWOfc90+gT6etYbi/95zQi65+2EBzICFtDpR91IYAjxOB3c4YpRELEhCpudsW0TxShYDLMmhCc2ZPnSe2k5Ngl5/Y0X75M48igfXSACshBZ6iMblAFVRFj+gZvaI368l6sd6tj2nrgpXO7KE/sD5/AITVm6I=</latexit> � 3 APPROXIMATE BAYESIAN INFERENCE ‣ Why? ‣ Compute intractable integrals p ( y ∗ | y ) = E p ( θ | y ) ( p ( y | θ )) ‣ Uncertainty quantification ‣ How? ‣ Laplace: p ( θ | y ) ≈ N ( θ MAP , ( H ( θ MAP ) + λ I ) − 1 ) ‣ Variational Bayes: p ( θ | y ) ≈ N ( µ, S ) ‣ Markov Chain Monte Carlo

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend


More recommend