Gaussian Mixture Models Instructor: John Thickstun Discussion Board: - - PowerPoint PPT Presentation

gaussian mixture models
SMART_READER_LITE
LIVE PREVIEW

Gaussian Mixture Models Instructor: John Thickstun Discussion Board: - - PowerPoint PPT Presentation

Gaussian Mixture Models Instructor: John Thickstun Discussion Board: Available on Canvas Zoom Link: Available on Canvas Instructor Contact: thickstn@cs.washington.edu Course Webpage: https://courses.cs.washington.edu/courses/cse599i/20au/ CSE


slide-1
SLIDE 1

Gaussian Mixture Models

Discussion Board: Available on Canvas Zoom Link: Available on Canvas Instructor Contact: thickstn@cs.washington.edu Course Webpage: https://courses.cs.washington.edu/courses/cse599i/20au/ CSE 599I: Generative Models University of Washington Lecture 2 Instructor: John Thickstun

slide-2
SLIDE 2

Logistics

CSE 599I: Generative Models University of Washington Lecture 2

  • Lectures will be recorded and posted internally starting today.
  • A better discussion board (Ed) will be up in the next couple days.
  • TA: Sami Davies will be helping out with course infrastructure and

moderating questions during lecture

slide-3
SLIDE 3

Recap: Overfitting

CSE 599I: Generative Models University of Washington Lecture 2

  • Given finite samples from a continuous distribution
  • Estimate the probability of each element of using the MLE?
  • Regularization: restrict our estimator to a parametric family

x1, . . . , xn ∼ p

<latexit sha1_base64="sHutXi237a5jqKGi7cR0t69SFyY=">AB/XicbZDLSgMxFIbP1Fut/GycxMsgotSZkTQZcFNlxXsBdphyKSZNjSTGZKMtA7FV3HjQhG3voc738a0nYW2Hgh8/P85yckfJwp7TjfVmFtfWNzq7hd2tnd2z+wD49aKk4loU0S81h2AqwoZ4I2NdOcdhJcRw2g5GtzO/UClYrG415OEehEeCBYygrWRfPtk7LuVXj/WqjL2BeopFqHEt8tO1ZkXWgU3hzLk1fDtL3MFSMqNOFYqa7rJNrLsNSMcDot9VJFE0xGeEC7BgWOqPKy+fZTdG6UPgpjaY7QaK7+nshwpNQkCkxnhPVQLXsz8T+vm+rwxsuYSFJNBVk8FKYc6RjNokB9JinRfGIAE8nMrogMscREm8BKJgR3+cur0Lqsuobvrsq1eh5HEU7hDC7AhWuoQR0a0AQCj/AMr/BmPVkv1rv1sWgtWPnMfwp6/MHYLuUiQ=</latexit><latexit sha1_base64="sHutXi237a5jqKGi7cR0t69SFyY=">AB/XicbZDLSgMxFIbP1Fut/GycxMsgotSZkTQZcFNlxXsBdphyKSZNjSTGZKMtA7FV3HjQhG3voc738a0nYW2Hgh8/P85yckfJwp7TjfVmFtfWNzq7hd2tnd2z+wD49aKk4loU0S81h2AqwoZ4I2NdOcdhJcRw2g5GtzO/UClYrG415OEehEeCBYygrWRfPtk7LuVXj/WqjL2BeopFqHEt8tO1ZkXWgU3hzLk1fDtL3MFSMqNOFYqa7rJNrLsNSMcDot9VJFE0xGeEC7BgWOqPKy+fZTdG6UPgpjaY7QaK7+nshwpNQkCkxnhPVQLXsz8T+vm+rwxsuYSFJNBVk8FKYc6RjNokB9JinRfGIAE8nMrogMscREm8BKJgR3+cur0Lqsuobvrsq1eh5HEU7hDC7AhWuoQR0a0AQCj/AMr/BmPVkv1rv1sWgtWPnMfwp6/MHYLuUiQ=</latexit><latexit sha1_base64="sHutXi237a5jqKGi7cR0t69SFyY=">AB/XicbZDLSgMxFIbP1Fut/GycxMsgotSZkTQZcFNlxXsBdphyKSZNjSTGZKMtA7FV3HjQhG3voc738a0nYW2Hgh8/P85yckfJwp7TjfVmFtfWNzq7hd2tnd2z+wD49aKk4loU0S81h2AqwoZ4I2NdOcdhJcRw2g5GtzO/UClYrG415OEehEeCBYygrWRfPtk7LuVXj/WqjL2BeopFqHEt8tO1ZkXWgU3hzLk1fDtL3MFSMqNOFYqa7rJNrLsNSMcDot9VJFE0xGeEC7BgWOqPKy+fZTdG6UPgpjaY7QaK7+nshwpNQkCkxnhPVQLXsz8T+vm+rwxsuYSFJNBVk8FKYc6RjNokB9JinRfGIAE8nMrogMscREm8BKJgR3+cur0Lqsuobvrsq1eh5HEU7hDC7AhWuoQR0a0AQCj/AMr/BmPVkv1rv1sWgtWPnMfwp6/MHYLuUiQ=</latexit><latexit sha1_base64="sHutXi237a5jqKGi7cR0t69SFyY=">AB/XicbZDLSgMxFIbP1Fut/GycxMsgotSZkTQZcFNlxXsBdphyKSZNjSTGZKMtA7FV3HjQhG3voc738a0nYW2Hgh8/P85yckfJwp7TjfVmFtfWNzq7hd2tnd2z+wD49aKk4loU0S81h2AqwoZ4I2NdOcdhJcRw2g5GtzO/UClYrG415OEehEeCBYygrWRfPtk7LuVXj/WqjL2BeopFqHEt8tO1ZkXWgU3hzLk1fDtL3MFSMqNOFYqa7rJNrLsNSMcDot9VJFE0xGeEC7BgWOqPKy+fZTdG6UPgpjaY7QaK7+nshwpNQkCkxnhPVQLXsz8T+vm+rwxsuYSFJNBVk8FKYc6RjNokB9JinRfGIAE8nMrogMscREm8BKJgR3+cur0Lqsuobvrsq1eh5HEU7hDC7AhWuoQR0a0AQCj/AMr/BmPVkv1rv1sWgtWPnMfwp6/MHYLuUiQ=</latexit>

X

<latexit sha1_base64="p1VwIcWperjIiohbPjNoyvSeP8=">AB8nicbVBNS8NAFHypX7V+VT16CRbBU0lE0GPBi3iqYGshDWz3bRLN7th90UoT/DiwdFvPprvPlv3LQ5aOvAwjDzHjtvolRwg5737VTW1jc2t6rbtZ3dvf2D+uFR16hMU9ahSijdi4hgkvWQY6C9VLNSBIJ9hNbgr/8Ylpw5V8wGnKwoSMJI85JWiloJ8QHFMi8t5sUG94TW8Od5X4JWlAifag/tUfKpolTCIVxJjA91IMc6KRU8FmtX5mWErohIxYKkCTNhPo8c8+sMnRjpe2T6M7V3xs5SYyZJpGdLCKaZa8Q/OCDOPrMOcyzZBJuvgozoSLyi3ud4dcM4piagmhmtusLh0TSjalmq2BH/5FXSvWj6lt9fNlp3ZR1VOIFTOAcfrqAFt9CGDlBQ8Ayv8Oag8+K8Ox+L0YpT7hzDHzifP5WrkXY=</latexit><latexit sha1_base64="p1VwIcWperjIiohbPjNoyvSeP8=">AB8nicbVBNS8NAFHypX7V+VT16CRbBU0lE0GPBi3iqYGshDWz3bRLN7th90UoT/DiwdFvPprvPlv3LQ5aOvAwjDzHjtvolRwg5737VTW1jc2t6rbtZ3dvf2D+uFR16hMU9ahSijdi4hgkvWQY6C9VLNSBIJ9hNbgr/8Ylpw5V8wGnKwoSMJI85JWiloJ8QHFMi8t5sUG94TW8Od5X4JWlAifag/tUfKpolTCIVxJjA91IMc6KRU8FmtX5mWErohIxYKkCTNhPo8c8+sMnRjpe2T6M7V3xs5SYyZJpGdLCKaZa8Q/OCDOPrMOcyzZBJuvgozoSLyi3ud4dcM4piagmhmtusLh0TSjalmq2BH/5FXSvWj6lt9fNlp3ZR1VOIFTOAcfrqAFt9CGDlBQ8Ayv8Oag8+K8Ox+L0YpT7hzDHzifP5WrkXY=</latexit><latexit sha1_base64="p1VwIcWperjIiohbPjNoyvSeP8=">AB8nicbVBNS8NAFHypX7V+VT16CRbBU0lE0GPBi3iqYGshDWz3bRLN7th90UoT/DiwdFvPprvPlv3LQ5aOvAwjDzHjtvolRwg5737VTW1jc2t6rbtZ3dvf2D+uFR16hMU9ahSijdi4hgkvWQY6C9VLNSBIJ9hNbgr/8Ylpw5V8wGnKwoSMJI85JWiloJ8QHFMi8t5sUG94TW8Od5X4JWlAifag/tUfKpolTCIVxJjA91IMc6KRU8FmtX5mWErohIxYKkCTNhPo8c8+sMnRjpe2T6M7V3xs5SYyZJpGdLCKaZa8Q/OCDOPrMOcyzZBJuvgozoSLyi3ud4dcM4piagmhmtusLh0TSjalmq2BH/5FXSvWj6lt9fNlp3ZR1VOIFTOAcfrqAFt9CGDlBQ8Ayv8Oag8+K8Ox+L0YpT7hzDHzifP5WrkXY=</latexit><latexit sha1_base64="p1VwIcWperjIiohbPjNoyvSeP8=">AB8nicbVBNS8NAFHypX7V+VT16CRbBU0lE0GPBi3iqYGshDWz3bRLN7th90UoT/DiwdFvPprvPlv3LQ5aOvAwjDzHjtvolRwg5737VTW1jc2t6rbtZ3dvf2D+uFR16hMU9ahSijdi4hgkvWQY6C9VLNSBIJ9hNbgr/8Ylpw5V8wGnKwoSMJI85JWiloJ8QHFMi8t5sUG94TW8Od5X4JWlAifag/tUfKpolTCIVxJjA91IMc6KRU8FmtX5mWErohIxYKkCTNhPo8c8+sMnRjpe2T6M7V3xs5SYyZJpGdLCKaZa8Q/OCDOPrMOcyzZBJuvgozoSLyi3ud4dcM4piagmhmtusLh0TSjalmq2BH/5FXSvWj6lt9fNlp3ZR1VOIFTOAcfrqAFt9CGDlBQ8Ayv8Oag8+K8Ox+L0YpT7hzDHzifP5WrkXY=</latexit>

ˆ p(x) = (

1 n

if x ∈ {x1, . . . , xn},

  • therwise.
<latexit sha1_base64="l3SBGHwvNaW/+mNpoZFfDBzUao=">ACW3icbVFda9swFJXdU29bs029tQXsXSlgxDsMdheBoW+lD210LSFKBhZuU5EZdlI15uD8J/c0/bQvzKqpIb1YxcEh3Pu51FWKWkxjv8E4cazedbve3oxc7LV7v9128ubFkbAWNRqtJcZdyCkhrGKFHBVWAF5mCy+z6eKVf/gBjZanPcVnBtOBzLXMpOHoq7Ru24Eirw+Yj/RaxDOZSO+H72TZiueHCJa3TLT2gDKFBR2VO9xvKpKbMNWkyZLMS7bBJNWv32yFjUfwvt8QFmJ/SQjuKGOhZ1zjtD+JRvA76FCQdGJAuTtP+Lz9F1AVoFIpbO0niCqeOG5RCgV+0tlBxc3nMPFQ8wLs1K29aekHz8xoXhr/NI1e7/C8cLaZH5zILjwj7WVuT/tEmN+depk7qEbS4G5TXimJV0bTmTQgUC094MJIvysVC+4tRf8dkTcheXzyU3DxaZR4fPZ5cPS9s6NH9sh7ckgS8oUckRNySsZEkN/kb7AV9IKbcCOMwp271Doat6SBxG+uwWVR7MU</latexit><latexit sha1_base64="l3SBGHwvNaW/+mNpoZFfDBzUao=">ACW3icbVFda9swFJXdU29bs029tQXsXSlgxDsMdheBoW+lD210LSFKBhZuU5EZdlI15uD8J/c0/bQvzKqpIb1YxcEh3Pu51FWKWkxjv8E4cazedbve3oxc7LV7v9128ubFkbAWNRqtJcZdyCkhrGKFHBVWAF5mCy+z6eKVf/gBjZanPcVnBtOBzLXMpOHoq7Ru24Eirw+Yj/RaxDOZSO+H72TZiueHCJa3TLT2gDKFBR2VO9xvKpKbMNWkyZLMS7bBJNWv32yFjUfwvt8QFmJ/SQjuKGOhZ1zjtD+JRvA76FCQdGJAuTtP+Lz9F1AVoFIpbO0niCqeOG5RCgV+0tlBxc3nMPFQ8wLs1K29aekHz8xoXhr/NI1e7/C8cLaZH5zILjwj7WVuT/tEmN+depk7qEbS4G5TXimJV0bTmTQgUC094MJIvysVC+4tRf8dkTcheXzyU3DxaZR4fPZ5cPS9s6NH9sh7ckgS8oUckRNySsZEkN/kb7AV9IKbcCOMwp271Doat6SBxG+uwWVR7MU</latexit><latexit sha1_base64="l3SBGHwvNaW/+mNpoZFfDBzUao=">ACW3icbVFda9swFJXdU29bs029tQXsXSlgxDsMdheBoW+lD210LSFKBhZuU5EZdlI15uD8J/c0/bQvzKqpIb1YxcEh3Pu51FWKWkxjv8E4cazedbve3oxc7LV7v9128ubFkbAWNRqtJcZdyCkhrGKFHBVWAF5mCy+z6eKVf/gBjZanPcVnBtOBzLXMpOHoq7Ru24Eirw+Yj/RaxDOZSO+H72TZiueHCJa3TLT2gDKFBR2VO9xvKpKbMNWkyZLMS7bBJNWv32yFjUfwvt8QFmJ/SQjuKGOhZ1zjtD+JRvA76FCQdGJAuTtP+Lz9F1AVoFIpbO0niCqeOG5RCgV+0tlBxc3nMPFQ8wLs1K29aekHz8xoXhr/NI1e7/C8cLaZH5zILjwj7WVuT/tEmN+depk7qEbS4G5TXimJV0bTmTQgUC094MJIvysVC+4tRf8dkTcheXzyU3DxaZR4fPZ5cPS9s6NH9sh7ckgS8oUckRNySsZEkN/kb7AV9IKbcCOMwp271Doat6SBxG+uwWVR7MU</latexit><latexit sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i7+gsTLXTzQrMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GS7KDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RtRxzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D7odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uat6tDP4I+/zBzjGijg=</latexit><latexit sha1_base64="3+ORFdh+G4boPkokmGh7Jz6zA=">ACUHicbVFNTxsxEPUuhYRtCmvVhAEZWiaJdLuVSqxKVHKhFAiqOV15lNLzelT1bNrL2T/YEB/4KwgmR+OqTLD29N54ZP2eVkhbj+C4INz5sbnW629H3qed3f7n3oUtayNgJEpVmquMW1BSwglKriqDPAiU3CZXZ8u/cu/YKws9TkuKpgUfKZlLgVHL6V9w+YcaXUfKc/I5bBTGonfD/bRiw3XLikdbqlh5QhNOiozOlBQ5nUlLkmTQZsWqIdNKlm7UE7YCyKn2tLnIO5kRbaYcRAT9eN0/5+PIxXoO9Jsib7ZI2ztP/PTxF1ARqF4taOk7jCieMGpVDgF60tVFxc8xmMPdW8ADtxq2xa+s0rU5qXxh+NdKW+vOF4Ye2iyHxlwXFu3pL8X/euMb8ZOKkrmoELZ4G5bWiWNJl0HQqDQhUC0+4MNLvSsWc+0jRf0fkQ0jePvk9uTgeJp7/iUmXfCV75Igk5Af5RX6TMzIigtySh6ATdIP7cCOMnuIKg3VuX8grhL1H1y2yRQ=</latexit><latexit sha1_base64="3+ORFdh+G4boPkokmGh7Jz6zA=">ACUHicbVFNTxsxEPUuhYRtCmvVhAEZWiaJdLuVSqxKVHKhFAiqOV15lNLzelT1bNrL2T/YEB/4KwgmR+OqTLD29N54ZP2eVkhbj+C4INz5sbnW629H3qed3f7n3oUtayNgJEpVmquMW1BSwglKriqDPAiU3CZXZ8u/cu/YKws9TkuKpgUfKZlLgVHL6V9w+YcaXUfKc/I5bBTGonfD/bRiw3XLikdbqlh5QhNOiozOlBQ5nUlLkmTQZsWqIdNKlm7UE7YCyKn2tLnIO5kRbaYcRAT9eN0/5+PIxXoO9Jsib7ZI2ztP/PTxF1ARqF4taOk7jCieMGpVDgF60tVFxc8xmMPdW8ADtxq2xa+s0rU5qXxh+NdKW+vOF4Ye2iyHxlwXFu3pL8X/euMb8ZOKkrmoELZ4G5bWiWNJl0HQqDQhUC0+4MNLvSsWc+0jRf0fkQ0jePvk9uTgeJp7/iUmXfCV75Igk5Af5RX6TMzIigtySh6ATdIP7cCOMnuIKg3VuX8grhL1H1y2yRQ=</latexit><latexit sha1_base64="ZG9INygWT0NU9/Nns64zyJqeGds=">ACW3icbVFda9swFJXdrc3cr3RjT3sRSztaCMHuS/tSKOxl7KmDpS1EwcjKdSIqy0a6bh2E/+Setof9lTElNWxrd0FwOd+HmWVkhbj+EcQbrx4ubnVexVt7+zu7fcPXl/bsjYCxqJUpbnNuAUlNYxRoLbygAvMgU32d3HlX5zD8bKUn/FZQXTgs+1zKXg6Km0b9iCI62OmxN6EbEM5lI74fvZNmK54cIlrdMt/UAZQoOypweNpRJTZlr0mTIZiXaYZNq1h62Q8ai+E9uiQswD9JCO4oY6FnXO0P4lG8DvocJB0YkC6u0v43P0XUBWgUils7SeIKp4blEKBX7S2UHFx+cw8VDzAuzUrb1p6ZFnZjQvjX8a6Zr9u8LxwtplkfnMguPCPtVW5P+0SY35+dRJXdUIWjwOymtFsaQro+lMGhColh5wYaTflYoF95ai/47Im5A8Pfk5uD4dJR5/iQeXnzs7euQdeU+OSULOyCX5RK7ImAjynfwKtoJe8DPcCKNw5zE1DLqaN+SfCN/+BpQHsxA=</latexit><latexit sha1_base64="l3SBGHwvNaW/+mNpoZFfDBzUao=">ACW3icbVFda9swFJXdU29bs029tQXsXSlgxDsMdheBoW+lD210LSFKBhZuU5EZdlI15uD8J/c0/bQvzKqpIb1YxcEh3Pu51FWKWkxjv8E4cazedbve3oxc7LV7v9128ubFkbAWNRqtJcZdyCkhrGKFHBVWAF5mCy+z6eKVf/gBjZanPcVnBtOBzLXMpOHoq7Ru24Eirw+Yj/RaxDOZSO+H72TZiueHCJa3TLT2gDKFBR2VO9xvKpKbMNWkyZLMS7bBJNWv32yFjUfwvt8QFmJ/SQjuKGOhZ1zjtD+JRvA76FCQdGJAuTtP+Lz9F1AVoFIpbO0niCqeOG5RCgV+0tlBxc3nMPFQ8wLs1K29aekHz8xoXhr/NI1e7/C8cLaZH5zILjwj7WVuT/tEmN+depk7qEbS4G5TXimJV0bTmTQgUC094MJIvysVC+4tRf8dkTcheXzyU3DxaZR4fPZ5cPS9s6NH9sh7ckgS8oUckRNySsZEkN/kb7AV9IKbcCOMwp271Doat6SBxG+uwWVR7MU</latexit><latexit sha1_base64="l3SBGHwvNaW/+mNpoZFfDBzUao=">ACW3icbVFda9swFJXdU29bs029tQXsXSlgxDsMdheBoW+lD210LSFKBhZuU5EZdlI15uD8J/c0/bQvzKqpIb1YxcEh3Pu51FWKWkxjv8E4cazedbve3oxc7LV7v9128ubFkbAWNRqtJcZdyCkhrGKFHBVWAF5mCy+z6eKVf/gBjZanPcVnBtOBzLXMpOHoq7Ru24Eirw+Yj/RaxDOZSO+H72TZiueHCJa3TLT2gDKFBR2VO9xvKpKbMNWkyZLMS7bBJNWv32yFjUfwvt8QFmJ/SQjuKGOhZ1zjtD+JRvA76FCQdGJAuTtP+Lz9F1AVoFIpbO0niCqeOG5RCgV+0tlBxc3nMPFQ8wLs1K29aekHz8xoXhr/NI1e7/C8cLaZH5zILjwj7WVuT/tEmN+depk7qEbS4G5TXimJV0bTmTQgUC094MJIvysVC+4tRf8dkTcheXzyU3DxaZR4fPZ5cPS9s6NH9sh7ckgS8oUckRNySsZEkN/kb7AV9IKbcCOMwp271Doat6SBxG+uwWVR7MU</latexit><latexit sha1_base64="l3SBGHwvNaW/+mNpoZFfDBzUao=">ACW3icbVFda9swFJXdU29bs029tQXsXSlgxDsMdheBoW+lD210LSFKBhZuU5EZdlI15uD8J/c0/bQvzKqpIb1YxcEh3Pu51FWKWkxjv8E4cazedbve3oxc7LV7v9128ubFkbAWNRqtJcZdyCkhrGKFHBVWAF5mCy+z6eKVf/gBjZanPcVnBtOBzLXMpOHoq7Ru24Eirw+Yj/RaxDOZSO+H72TZiueHCJa3TLT2gDKFBR2VO9xvKpKbMNWkyZLMS7bBJNWv32yFjUfwvt8QFmJ/SQjuKGOhZ1zjtD+JRvA76FCQdGJAuTtP+Lz9F1AVoFIpbO0niCqeOG5RCgV+0tlBxc3nMPFQ8wLs1K29aekHz8xoXhr/NI1e7/C8cLaZH5zILjwj7WVuT/tEmN+depk7qEbS4G5TXimJV0bTmTQgUC094MJIvysVC+4tRf8dkTcheXzyU3DxaZR4fPZ5cPS9s6NH9sh7ckgS8oUckRNySsZEkN/kb7AV9IKbcCOMwp271Doat6SBxG+uwWVR7MU</latexit><latexit sha1_base64="l3SBGHwvNaW/+mNpoZFfDBzUao=">ACW3icbVFda9swFJXdU29bs029tQXsXSlgxDsMdheBoW+lD210LSFKBhZuU5EZdlI15uD8J/c0/bQvzKqpIb1YxcEh3Pu51FWKWkxjv8E4cazedbve3oxc7LV7v9128ubFkbAWNRqtJcZdyCkhrGKFHBVWAF5mCy+z6eKVf/gBjZanPcVnBtOBzLXMpOHoq7Ru24Eirw+Yj/RaxDOZSO+H72TZiueHCJa3TLT2gDKFBR2VO9xvKpKbMNWkyZLMS7bBJNWv32yFjUfwvt8QFmJ/SQjuKGOhZ1zjtD+JRvA76FCQdGJAuTtP+Lz9F1AVoFIpbO0niCqeOG5RCgV+0tlBxc3nMPFQ8wLs1K29aekHz8xoXhr/NI1e7/C8cLaZH5zILjwj7WVuT/tEmN+depk7qEbS4G5TXimJV0bTmTQgUC094MJIvysVC+4tRf8dkTcheXzyU3DxaZR4fPZ5cPS9s6NH9sh7ckgS8oUckRNySsZEkN/kb7AV9IKbcCOMwp271Doat6SBxG+uwWVR7MU</latexit><latexit sha1_base64="l3SBGHwvNaW/+mNpoZFfDBzUao=">ACW3icbVFda9swFJXdU29bs029tQXsXSlgxDsMdheBoW+lD210LSFKBhZuU5EZdlI15uD8J/c0/bQvzKqpIb1YxcEh3Pu51FWKWkxjv8E4cazedbve3oxc7LV7v9128ubFkbAWNRqtJcZdyCkhrGKFHBVWAF5mCy+z6eKVf/gBjZanPcVnBtOBzLXMpOHoq7Ru24Eirw+Yj/RaxDOZSO+H72TZiueHCJa3TLT2gDKFBR2VO9xvKpKbMNWkyZLMS7bBJNWv32yFjUfwvt8QFmJ/SQjuKGOhZ1zjtD+JRvA76FCQdGJAuTtP+Lz9F1AVoFIpbO0niCqeOG5RCgV+0tlBxc3nMPFQ8wLs1K29aekHz8xoXhr/NI1e7/C8cLaZH5zILjwj7WVuT/tEmN+depk7qEbS4G5TXimJV0bTmTQgUC094MJIvysVC+4tRf8dkTcheXzyU3DxaZR4fPZ5cPS9s6NH9sh7ckgS8oUckRNySsZEkN/kb7AV9IKbcCOMwp271Doat6SBxG+uwWVR7MU</latexit><latexit sha1_base64="l3SBGHwvNaW/+mNpoZFfDBzUao=">ACW3icbVFda9swFJXdU29bs029tQXsXSlgxDsMdheBoW+lD210LSFKBhZuU5EZdlI15uD8J/c0/bQvzKqpIb1YxcEh3Pu51FWKWkxjv8E4cazedbve3oxc7LV7v9128ubFkbAWNRqtJcZdyCkhrGKFHBVWAF5mCy+z6eKVf/gBjZanPcVnBtOBzLXMpOHoq7Ru24Eirw+Yj/RaxDOZSO+H72TZiueHCJa3TLT2gDKFBR2VO9xvKpKbMNWkyZLMS7bBJNWv32yFjUfwvt8QFmJ/SQjuKGOhZ1zjtD+JRvA76FCQdGJAuTtP+Lz9F1AVoFIpbO0niCqeOG5RCgV+0tlBxc3nMPFQ8wLs1K29aekHz8xoXhr/NI1e7/C8cLaZH5zILjwj7WVuT/tEmN+depk7qEbS4G5TXimJV0bTmTQgUC094MJIvysVC+4tRf8dkTcheXzyU3DxaZR4fPZ5cPS9s6NH9sh7ckgS8oUckRNySsZEkN/kb7AV9IKbcCOMwp271Doat6SBxG+uwWVR7MU</latexit>

p(x).

<latexit sha1_base64="PgyPHUXDI8CV8Ds2g+TmPbx7U8=">AB7HicbZBNSwMxEIZn61etX1WPXoJFqJeyK4IeC17EUwW3LbRLyazbWg2uyRZsZT+Bi8eFPHqD/LmvzFt96CtLwQe3pkhM2+YCq6N6347hbX1jc2t4nZpZ3dv/6B8eNTUSaY+iwRiWqHVKPgEn3DjcB2qpDGocBWOLqZ1VuPqDRP5IMZpxjEdCB5xBk1vLT6tN5rVeuDV3LrIKXg4VyNXolb+6/YRlMUrDBNW647mpCSZUGc4ETkvdTGNK2YgOsGNR0h1MJkvOyVn1umTKFH2SUPm7u+JCY21Hseh7YypGerl2sz8r9bJTHQdTLhM4OSLT6KMkFMQmaXkz5XyIwYW6BMcbsrYUOqKDM2n5INwVs+eRWaFzXP8v1lpX6Xx1GEziFKnhwBXW4hQb4wIDM7zCmyOdF+fd+Vi0Fpx85hj+yPn8Aenqjhs=</latexit><latexit sha1_base64="PgyPHUXDI8CV8Ds2g+TmPbx7U8=">AB7HicbZBNSwMxEIZn61etX1WPXoJFqJeyK4IeC17EUwW3LbRLyazbWg2uyRZsZT+Bi8eFPHqD/LmvzFt96CtLwQe3pkhM2+YCq6N6347hbX1jc2t4nZpZ3dv/6B8eNTUSaY+iwRiWqHVKPgEn3DjcB2qpDGocBWOLqZ1VuPqDRP5IMZpxjEdCB5xBk1vLT6tN5rVeuDV3LrIKXg4VyNXolb+6/YRlMUrDBNW647mpCSZUGc4ETkvdTGNK2YgOsGNR0h1MJkvOyVn1umTKFH2SUPm7u+JCY21Hseh7YypGerl2sz8r9bJTHQdTLhM4OSLT6KMkFMQmaXkz5XyIwYW6BMcbsrYUOqKDM2n5INwVs+eRWaFzXP8v1lpX6Xx1GEziFKnhwBXW4hQb4wIDM7zCmyOdF+fd+Vi0Fpx85hj+yPn8Aenqjhs=</latexit><latexit sha1_base64="PgyPHUXDI8CV8Ds2g+TmPbx7U8=">AB7HicbZBNSwMxEIZn61etX1WPXoJFqJeyK4IeC17EUwW3LbRLyazbWg2uyRZsZT+Bi8eFPHqD/LmvzFt96CtLwQe3pkhM2+YCq6N6347hbX1jc2t4nZpZ3dv/6B8eNTUSaY+iwRiWqHVKPgEn3DjcB2qpDGocBWOLqZ1VuPqDRP5IMZpxjEdCB5xBk1vLT6tN5rVeuDV3LrIKXg4VyNXolb+6/YRlMUrDBNW647mpCSZUGc4ETkvdTGNK2YgOsGNR0h1MJkvOyVn1umTKFH2SUPm7u+JCY21Hseh7YypGerl2sz8r9bJTHQdTLhM4OSLT6KMkFMQmaXkz5XyIwYW6BMcbsrYUOqKDM2n5INwVs+eRWaFzXP8v1lpX6Xx1GEziFKnhwBXW4hQb4wIDM7zCmyOdF+fd+Vi0Fpx85hj+yPn8Aenqjhs=</latexit><latexit sha1_base64="PgyPHUXDI8CV8Ds2g+TmPbx7U8=">AB7HicbZBNSwMxEIZn61etX1WPXoJFqJeyK4IeC17EUwW3LbRLyazbWg2uyRZsZT+Bi8eFPHqD/LmvzFt96CtLwQe3pkhM2+YCq6N6347hbX1jc2t4nZpZ3dv/6B8eNTUSaY+iwRiWqHVKPgEn3DjcB2qpDGocBWOLqZ1VuPqDRP5IMZpxjEdCB5xBk1vLT6tN5rVeuDV3LrIKXg4VyNXolb+6/YRlMUrDBNW647mpCSZUGc4ETkvdTGNK2YgOsGNR0h1MJkvOyVn1umTKFH2SUPm7u+JCY21Hseh7YypGerl2sz8r9bJTHQdTLhM4OSLT6KMkFMQmaXkz5XyIwYW6BMcbsrYUOqKDM2n5INwVs+eRWaFzXP8v1lpX6Xx1GEziFKnhwBXW4hQb4wIDM7zCmyOdF+fd+Vi0Fpx85hj+yPn8Aenqjhs=</latexit>
slide-4
SLIDE 4

Maximum Likelihood Estimation

CSE 599I: Generative Models University of Washington Lecture 2

  • Given a parametric family of probability distributions
  • Choose to maximize the likelihood of observations :
  • KL-divergence minimization:

sup

θ

E

x∼p log pθ(x) ≈ sup θ

1 n

n

X

i=1

log pθ(xi).

<latexit sha1_base64="4OhiY2BNOInQOUJpVuWTqnaEyWA=">ACY3icbVFdSyMxFM2Mu37UVWd32QhbBH0pczIwvoiCLQxy5sVejUIZNm2mC+SO5Iu2H+pG+7Yv/w7R2QeseCDmce26Se1IawR2k6WMUr34uL6xudXa/rSzu5d83r9yuraU9akW2t6UxDHBFesDB8FujGVEloJdl3eX8/r1PbOa/UbZoYNJRkrXnFKIEhF8id3tSlymDAgOJcEJtr4xV6W/mfTFH6aOy6xaXAu9Bj/8x5PT3BOjLF6it8cUVlCfdZ41QRZFp6fZ82tWm0u+EmnSNpJ10AvyfZkrTREr0iechHmtaSKaCODfIUgNDTyxwKljTymvHDKF3ZMwGgSoimRv6RUYNPgrKCFfahqUAL9TXHZ5I52ayDM759G61Nhf/VxvUJ0NPVemBqboy0VLTBoPA8cj7hlFMQsEItD2/FdEJCShC+pRVCyFZHfk+uTjtZ4L+ty+6yzg20SH6ho5Rhn6gC9RFPdRHFP2N1qO9KIme4u14Pz54scbRsucLeoP46zP0Zbjc</latexit><latexit sha1_base64="4OhiY2BNOInQOUJpVuWTqnaEyWA=">ACY3icbVFdSyMxFM2Mu37UVWd32QhbBH0pczIwvoiCLQxy5sVejUIZNm2mC+SO5Iu2H+pG+7Yv/w7R2QeseCDmce26Se1IawR2k6WMUr34uL6xudXa/rSzu5d83r9yuraU9akW2t6UxDHBFesDB8FujGVEloJdl3eX8/r1PbOa/UbZoYNJRkrXnFKIEhF8id3tSlymDAgOJcEJtr4xV6W/mfTFH6aOy6xaXAu9Bj/8x5PT3BOjLF6it8cUVlCfdZ41QRZFp6fZ82tWm0u+EmnSNpJ10AvyfZkrTREr0iechHmtaSKaCODfIUgNDTyxwKljTymvHDKF3ZMwGgSoimRv6RUYNPgrKCFfahqUAL9TXHZ5I52ayDM759G61Nhf/VxvUJ0NPVemBqboy0VLTBoPA8cj7hlFMQsEItD2/FdEJCShC+pRVCyFZHfk+uTjtZ4L+ty+6yzg20SH6ho5Rhn6gC9RFPdRHFP2N1qO9KIme4u14Pz54scbRsucLeoP46zP0Zbjc</latexit><latexit sha1_base64="4OhiY2BNOInQOUJpVuWTqnaEyWA=">ACY3icbVFdSyMxFM2Mu37UVWd32QhbBH0pczIwvoiCLQxy5sVejUIZNm2mC+SO5Iu2H+pG+7Yv/w7R2QeseCDmce26Se1IawR2k6WMUr34uL6xudXa/rSzu5d83r9yuraU9akW2t6UxDHBFesDB8FujGVEloJdl3eX8/r1PbOa/UbZoYNJRkrXnFKIEhF8id3tSlymDAgOJcEJtr4xV6W/mfTFH6aOy6xaXAu9Bj/8x5PT3BOjLF6it8cUVlCfdZ41QRZFp6fZ82tWm0u+EmnSNpJ10AvyfZkrTREr0iechHmtaSKaCODfIUgNDTyxwKljTymvHDKF3ZMwGgSoimRv6RUYNPgrKCFfahqUAL9TXHZ5I52ayDM759G61Nhf/VxvUJ0NPVemBqboy0VLTBoPA8cj7hlFMQsEItD2/FdEJCShC+pRVCyFZHfk+uTjtZ4L+ty+6yzg20SH6ho5Rhn6gC9RFPdRHFP2N1qO9KIme4u14Pz54scbRsucLeoP46zP0Zbjc</latexit><latexit sha1_base64="4OhiY2BNOInQOUJpVuWTqnaEyWA=">ACY3icbVFdSyMxFM2Mu37UVWd32QhbBH0pczIwvoiCLQxy5sVejUIZNm2mC+SO5Iu2H+pG+7Yv/w7R2QeseCDmce26Se1IawR2k6WMUr34uL6xudXa/rSzu5d83r9yuraU9akW2t6UxDHBFesDB8FujGVEloJdl3eX8/r1PbOa/UbZoYNJRkrXnFKIEhF8id3tSlymDAgOJcEJtr4xV6W/mfTFH6aOy6xaXAu9Bj/8x5PT3BOjLF6it8cUVlCfdZ41QRZFp6fZ82tWm0u+EmnSNpJ10AvyfZkrTREr0iechHmtaSKaCODfIUgNDTyxwKljTymvHDKF3ZMwGgSoimRv6RUYNPgrKCFfahqUAL9TXHZ5I52ayDM759G61Nhf/VxvUJ0NPVemBqboy0VLTBoPA8cj7hlFMQsEItD2/FdEJCShC+pRVCyFZHfk+uTjtZ4L+ty+6yzg20SH6ho5Rhn6gC9RFPdRHFP2N1qO9KIme4u14Pz54scbRsucLeoP46zP0Zbjc</latexit>

{pθ : θ ∈ Θ}

<latexit sha1_base64="w9KWL0SnXdwREl8JBkPFodnGlO0=">ACXicbZDLSsNAFIYnXmu9RV26GSyCq5KIoLgquBFXFXqDJoTJdNIOnUzCzIlQrdufBU3LhRx6xu482ctFlo6w8DH/85hzPnD1PBNTjOt7Wyura+sVnZqm7v7O7t2weHZ1kirI2TUSieiHRTHDJ2sBsF6qGIlDwbrh+Kaodx+Y0jyRLZikzI/JUPKIUwLGCmzs5WngwYgBwde4BI9L7LUK9KaBXPqzkx4GdwSaqhUM7C/vEFCs5hJoIJo3XedFPycKOBUsGnVyzRLCR2TIesblCRm2s9nl0zxqXEGOEqUeRLwzP09kZNY60kcms6YwEgv1grzv1o/g+jKz7lM2CSzhdFmcCQ4CIWPOCKURATA4Qqbv6K6YgoQsGEVzUhuIsnL0PnvO4avr+oNe7KOCroGJ2gM+SiS9RAt6iJ2oiR/SMXtGb9WS9WO/Wx7x1xSpnjtAfWZ8/f4iZlQ=</latexit><latexit sha1_base64="w9KWL0SnXdwREl8JBkPFodnGlO0=">ACXicbZDLSsNAFIYnXmu9RV26GSyCq5KIoLgquBFXFXqDJoTJdNIOnUzCzIlQrdufBU3LhRx6xu482ctFlo6w8DH/85hzPnD1PBNTjOt7Wyura+sVnZqm7v7O7t2weHZ1kirI2TUSieiHRTHDJ2sBsF6qGIlDwbrh+Kaodx+Y0jyRLZikzI/JUPKIUwLGCmzs5WngwYgBwde4BI9L7LUK9KaBXPqzkx4GdwSaqhUM7C/vEFCs5hJoIJo3XedFPycKOBUsGnVyzRLCR2TIesblCRm2s9nl0zxqXEGOEqUeRLwzP09kZNY60kcms6YwEgv1grzv1o/g+jKz7lM2CSzhdFmcCQ4CIWPOCKURATA4Qqbv6K6YgoQsGEVzUhuIsnL0PnvO4avr+oNe7KOCroGJ2gM+SiS9RAt6iJ2oiR/SMXtGb9WS9WO/Wx7x1xSpnjtAfWZ8/f4iZlQ=</latexit><latexit sha1_base64="w9KWL0SnXdwREl8JBkPFodnGlO0=">ACXicbZDLSsNAFIYnXmu9RV26GSyCq5KIoLgquBFXFXqDJoTJdNIOnUzCzIlQrdufBU3LhRx6xu482ctFlo6w8DH/85hzPnD1PBNTjOt7Wyura+sVnZqm7v7O7t2weHZ1kirI2TUSieiHRTHDJ2sBsF6qGIlDwbrh+Kaodx+Y0jyRLZikzI/JUPKIUwLGCmzs5WngwYgBwde4BI9L7LUK9KaBXPqzkx4GdwSaqhUM7C/vEFCs5hJoIJo3XedFPycKOBUsGnVyzRLCR2TIesblCRm2s9nl0zxqXEGOEqUeRLwzP09kZNY60kcms6YwEgv1grzv1o/g+jKz7lM2CSzhdFmcCQ4CIWPOCKURATA4Qqbv6K6YgoQsGEVzUhuIsnL0PnvO4avr+oNe7KOCroGJ2gM+SiS9RAt6iJ2oiR/SMXtGb9WS9WO/Wx7x1xSpnjtAfWZ8/f4iZlQ=</latexit><latexit sha1_base64="w9KWL0SnXdwREl8JBkPFodnGlO0=">ACXicbZDLSsNAFIYnXmu9RV26GSyCq5KIoLgquBFXFXqDJoTJdNIOnUzCzIlQrdufBU3LhRx6xu482ctFlo6w8DH/85hzPnD1PBNTjOt7Wyura+sVnZqm7v7O7t2weHZ1kirI2TUSieiHRTHDJ2sBsF6qGIlDwbrh+Kaodx+Y0jyRLZikzI/JUPKIUwLGCmzs5WngwYgBwde4BI9L7LUK9KaBXPqzkx4GdwSaqhUM7C/vEFCs5hJoIJo3XedFPycKOBUsGnVyzRLCR2TIesblCRm2s9nl0zxqXEGOEqUeRLwzP09kZNY60kcms6YwEgv1grzv1o/g+jKz7lM2CSzhdFmcCQ4CIWPOCKURATA4Qqbv6K6YgoQsGEVzUhuIsnL0PnvO4avr+oNe7KOCroGJ2gM+SiS9RAt6iJ2oiR/SMXtGb9WS9WO/Wx7x1xSpnjtAfWZ8/f4iZlQ=</latexit>

θ ∈ Θ

<latexit sha1_base64="MVQ7LtFzcFp0ON/xXmsersopjc=">AB+nicbZDLSsNAFIYn9VbrLdWlm8EiuCqJCLosuBFXFXoRmlIm05N26GQSZk6UEvsoblwo4tYncefbOL0stPWHgY/nM584epFAY979sprK1vbG4Vt0s7u3v7B275sGWSTHNo8kQm+j5kBqRQ0ESBEu5TDSwOJbTD0fW03n4AbUSiGjhOoRuzgRKR4Ayt1XPLAQ4BGQ2EokFjij234lW9megq+AuokIXqPfcr6Cc8i0Ehl8yYju+l2M2ZRsElTEpBZiBlfMQG0LGoWAym89On9BT6/RplGj7FNKZ+3siZ7Ex4zi0nTHDoVmuTc3/ap0Mo6tuLlSaISg+XxRlkmJCpznQvtDAUY4tMK6FvZXyIdOMo02rZEPwl7+8Cq3zqm/57qJSu13EUSTH5IScEZ9ckhq5IXSJw8kmfySt6cJ+fFeXc+5q0FZzFzRP7I+fwBmAaTlA=</latexit><latexit sha1_base64="MVQ7LtFzcFp0ON/xXmsersopjc=">AB+nicbZDLSsNAFIYn9VbrLdWlm8EiuCqJCLosuBFXFXoRmlIm05N26GQSZk6UEvsoblwo4tYncefbOL0stPWHgY/nM584epFAY979sprK1vbG4Vt0s7u3v7B275sGWSTHNo8kQm+j5kBqRQ0ESBEu5TDSwOJbTD0fW03n4AbUSiGjhOoRuzgRKR4Ayt1XPLAQ4BGQ2EokFjij234lW9megq+AuokIXqPfcr6Cc8i0Ehl8yYju+l2M2ZRsElTEpBZiBlfMQG0LGoWAym89On9BT6/RplGj7FNKZ+3siZ7Ex4zi0nTHDoVmuTc3/ap0Mo6tuLlSaISg+XxRlkmJCpznQvtDAUY4tMK6FvZXyIdOMo02rZEPwl7+8Cq3zqm/57qJSu13EUSTH5IScEZ9ckhq5IXSJw8kmfySt6cJ+fFeXc+5q0FZzFzRP7I+fwBmAaTlA=</latexit><latexit sha1_base64="MVQ7LtFzcFp0ON/xXmsersopjc=">AB+nicbZDLSsNAFIYn9VbrLdWlm8EiuCqJCLosuBFXFXoRmlIm05N26GQSZk6UEvsoblwo4tYncefbOL0stPWHgY/nM584epFAY979sprK1vbG4Vt0s7u3v7B275sGWSTHNo8kQm+j5kBqRQ0ESBEu5TDSwOJbTD0fW03n4AbUSiGjhOoRuzgRKR4Ayt1XPLAQ4BGQ2EokFjij234lW9megq+AuokIXqPfcr6Cc8i0Ehl8yYju+l2M2ZRsElTEpBZiBlfMQG0LGoWAym89On9BT6/RplGj7FNKZ+3siZ7Ex4zi0nTHDoVmuTc3/ap0Mo6tuLlSaISg+XxRlkmJCpznQvtDAUY4tMK6FvZXyIdOMo02rZEPwl7+8Cq3zqm/57qJSu13EUSTH5IScEZ9ckhq5IXSJw8kmfySt6cJ+fFeXc+5q0FZzFzRP7I+fwBmAaTlA=</latexit><latexit sha1_base64="MVQ7LtFzcFp0ON/xXmsersopjc=">AB+nicbZDLSsNAFIYn9VbrLdWlm8EiuCqJCLosuBFXFXoRmlIm05N26GQSZk6UEvsoblwo4tYncefbOL0stPWHgY/nM584epFAY979sprK1vbG4Vt0s7u3v7B275sGWSTHNo8kQm+j5kBqRQ0ESBEu5TDSwOJbTD0fW03n4AbUSiGjhOoRuzgRKR4Ayt1XPLAQ4BGQ2EokFjij234lW9megq+AuokIXqPfcr6Cc8i0Ehl8yYju+l2M2ZRsElTEpBZiBlfMQG0LGoWAym89On9BT6/RplGj7FNKZ+3siZ7Ex4zi0nTHDoVmuTc3/ap0Mo6tuLlSaISg+XxRlkmJCpznQvtDAUY4tMK6FvZXyIdOMo02rZEPwl7+8Cq3zqm/57qJSu13EUSTH5IScEZ9ckhq5IXSJw8kmfySt6cJ+fFeXc+5q0FZzFzRP7I+fwBmAaTlA=</latexit>

x1, . . . , xn

<latexit sha1_base64="FQOTn5qxI62FOpSgq8xnwEKtWFU=">AB9HicbVDLSsNAFL3xWeur6tJNsAguSklE0GXBjbiqYB/QhjCZTtqhk0mcuSktod/hxoUibv0Yd/6N0zYLbT0wcDjnHu6dEySCa3Scb2tfWNza7uwU9zd2z84LB0dN3WcKsoaNBaxagdEM8ElayBHwdqJYiQKBGsFw9uZ3xoxpXksH3GSMC8ifclDTgkayRv7bqXbi1FXxr70S2Wn6sxhrxI3J2XIUfdLXyZL04hJpIJo3XGdBL2MKORUsGmxm2qWEDokfdYxVJKIaS+bHz21z43Ss8NYmSfRnqu/ExmJtJ5EgZmMCA70sjcT/M6KY3XsZlkiKTdLEoTIWNsT1rwO5xSiKiSGEKm5utemAKELR9FQ0JbjLX14lzcuqa/jDVbl2n9dRgFM4gwtw4RpqcAd1aACFJ3iGV3izRtaL9W59LEbXrDxzAn9gf4ANS6Rvw=</latexit><latexit sha1_base64="FQOTn5qxI62FOpSgq8xnwEKtWFU=">AB9HicbVDLSsNAFL3xWeur6tJNsAguSklE0GXBjbiqYB/QhjCZTtqhk0mcuSktod/hxoUibv0Yd/6N0zYLbT0wcDjnHu6dEySCa3Scb2tfWNza7uwU9zd2z84LB0dN3WcKsoaNBaxagdEM8ElayBHwdqJYiQKBGsFw9uZ3xoxpXksH3GSMC8ifclDTgkayRv7bqXbi1FXxr70S2Wn6sxhrxI3J2XIUfdLXyZL04hJpIJo3XGdBL2MKORUsGmxm2qWEDokfdYxVJKIaS+bHz21z43Ss8NYmSfRnqu/ExmJtJ5EgZmMCA70sjcT/M6KY3XsZlkiKTdLEoTIWNsT1rwO5xSiKiSGEKm5utemAKELR9FQ0JbjLX14lzcuqa/jDVbl2n9dRgFM4gwtw4RpqcAd1aACFJ3iGV3izRtaL9W59LEbXrDxzAn9gf4ANS6Rvw=</latexit><latexit sha1_base64="FQOTn5qxI62FOpSgq8xnwEKtWFU=">AB9HicbVDLSsNAFL3xWeur6tJNsAguSklE0GXBjbiqYB/QhjCZTtqhk0mcuSktod/hxoUibv0Yd/6N0zYLbT0wcDjnHu6dEySCa3Scb2tfWNza7uwU9zd2z84LB0dN3WcKsoaNBaxagdEM8ElayBHwdqJYiQKBGsFw9uZ3xoxpXksH3GSMC8ifclDTgkayRv7bqXbi1FXxr70S2Wn6sxhrxI3J2XIUfdLXyZL04hJpIJo3XGdBL2MKORUsGmxm2qWEDokfdYxVJKIaS+bHz21z43Ss8NYmSfRnqu/ExmJtJ5EgZmMCA70sjcT/M6KY3XsZlkiKTdLEoTIWNsT1rwO5xSiKiSGEKm5utemAKELR9FQ0JbjLX14lzcuqa/jDVbl2n9dRgFM4gwtw4RpqcAd1aACFJ3iGV3izRtaL9W59LEbXrDxzAn9gf4ANS6Rvw=</latexit><latexit sha1_base64="FQOTn5qxI62FOpSgq8xnwEKtWFU=">AB9HicbVDLSsNAFL3xWeur6tJNsAguSklE0GXBjbiqYB/QhjCZTtqhk0mcuSktod/hxoUibv0Yd/6N0zYLbT0wcDjnHu6dEySCa3Scb2tfWNza7uwU9zd2z84LB0dN3WcKsoaNBaxagdEM8ElayBHwdqJYiQKBGsFw9uZ3xoxpXksH3GSMC8ifclDTgkayRv7bqXbi1FXxr70S2Wn6sxhrxI3J2XIUfdLXyZL04hJpIJo3XGdBL2MKORUsGmxm2qWEDokfdYxVJKIaS+bHz21z43Ss8NYmSfRnqu/ExmJtJ5EgZmMCA70sjcT/M6KY3XsZlkiKTdLEoTIWNsT1rwO5xSiKiSGEKm5utemAKELR9FQ0JbjLX14lzcuqa/jDVbl2n9dRgFM4gwtw4RpqcAd1aACFJ3iGV3izRtaL9W59LEbXrDxzAn9gf4ANS6Rvw=</latexit>

E

x∼p log pθ(x) = E x∼p log pθ(x)

p(x) p(x) = H(p) + D(p k pθ) H(p).

<latexit sha1_base64="LHczl6pq909RMCd6/MWgD9lPg=">ADQHicjVLdahQxFM6Mf3X8auX3gSXxS2OZaY3CkUoWKF4VcFuFzbLkslkdkOTSUy64x4Av4BL6Nb+EbeCdeCV6Zma2ltQgemPk+zjlfJud8UyjOjM2yr1F85eq16zfWbia3bt+5u76xea9v5EwTekQkl3pQYEM5q+mRZbTgdIUi4LT4+LkRVM/nlNtmKzf2KWiI4EnNasYwTakxhufkcB2KpVrsSjcS+/HboEME1B5+ARxOYFqjOyUWtxbMHn8H8UqNKYuHM671TzVqsjDnpqCz6GiNVyeZO+bNWD9GEvm0btscbnWw7awNeJvkp6ewdzH9+BAcjejT6iUZCZobQnHxgzTNmRw9oywqlP0MxQhckJntBhoDUW1Ixcu0UPuyFTwkrq8NQWtnzCoeFaUZOA5qlKBq0U9FAIzApx5YuQiUtCpEuTJDRdKxmjKySCsusU0bvZWSGw8vXsZWz0aO1WpmaU1Wd6lmHFoJG9dgyTQli8DwUSzMA4kUx2bIO3SZKgfRoG1vQh8ZyP/wOgoXSwJ2ueOGHOyPX8y4Y0U0g7ORoF5VNm2HvKfqAdjs73SQsqKbviBQC1+UfqXf7Z4f86zsO1VIL7xDvU20D6haDgfnfdl0m/Z3tPDXwclXYBVr4AF4CHogB0/BHjgAh+AIkCiKHkVZlMdf4m/x9/jHqjWOTjX3wYWIf/0GBdgO+g=</latexit><latexit sha1_base64="rYH3AzA1nKGgQH/mBmqvaVUX/wk=">ADQHicjVLdihMxFJ4Z/9bxr6uX3gRL2S6OpdMbhUVYsEIRLyrY3UJTSiaTacMmk5iktTUGn8An8G18C9/AO/FK8ELMtHXZdRE8MPN9nHO+TM75JpOMatNufwmjS5evXL2cz2+cfPW7Tu13btHWswVJgMsmFDGnCaEkGhpGhlIRxDNGjrOTZ1X9eEGUpqJ8bVaSjDmalrSgGBmfmtQ+QY7MTEi7xiyz52b2CXUlAPpwCPIxBTICTQzYlBzuQ+egv9RwEIhbM/onJXVW26O6DXlPngIC2LnC6sdKetDsApebNuaE1q9XarvQ5wkaRbUj/sLX582Pv1sj/ZDT/CXOA5J6XBDGk9StvSjC1ShmJGXAznmkiET9CUjDwtESd6bNdbdKDhMzkohPJPacA6e1ZhEdfVyIlHveJZhWbGK6gEOmHIkKWvJFnGk6X2MpJMFZIzipdJwQySaU3QjDtwPnLmOLJ2NJSzg0p8eYuxZwBI0DlGsipItiwlScIK+rHAXiG/I6N9zaOY9glfmBF+sg35l3/O3DqS0O7XfHSjTpj23TWG9GIAain8ADmVZum7wh8Dw/qnUbsF1St1hwjsr8j9TZ7ukh/qOhaVQ3FnIjogyHtUavYHp3ZdJEedVur5K+/ki2ATO8H94EHQDNLgcXAY9IJ+MAhwGIZ7YTtMo8/R1+hb9H3TGoVbzb3gXEQ/fwP/rxBq</latexit><latexit sha1_base64="rYH3AzA1nKGgQH/mBmqvaVUX/wk=">ADQHicjVLdihMxFJ4Z/9bxr6uX3gRL2S6OpdMbhUVYsEIRLyrY3UJTSiaTacMmk5iktTUGn8An8G18C9/AO/FK8ELMtHXZdRE8MPN9nHO+TM75JpOMatNufwmjS5evXL2cz2+cfPW7Tu13btHWswVJgMsmFDGnCaEkGhpGhlIRxDNGjrOTZ1X9eEGUpqJ8bVaSjDmalrSgGBmfmtQ+QY7MTEi7xiyz52b2CXUlAPpwCPIxBTICTQzYlBzuQ+egv9RwEIhbM/onJXVW26O6DXlPngIC2LnC6sdKetDsApebNuaE1q9XarvQ5wkaRbUj/sLX582Pv1sj/ZDT/CXOA5J6XBDGk9StvSjC1ShmJGXAznmkiET9CUjDwtESd6bNdbdKDhMzkohPJPacA6e1ZhEdfVyIlHveJZhWbGK6gEOmHIkKWvJFnGk6X2MpJMFZIzipdJwQySaU3QjDtwPnLmOLJ2NJSzg0p8eYuxZwBI0DlGsipItiwlScIK+rHAXiG/I6N9zaOY9glfmBF+sg35l3/O3DqS0O7XfHSjTpj23TWG9GIAain8ADmVZum7wh8Dw/qnUbsF1St1hwjsr8j9TZ7ukh/qOhaVQ3FnIjogyHtUavYHp3ZdJEedVur5K+/ki2ATO8H94EHQDNLgcXAY9IJ+MAhwGIZ7YTtMo8/R1+hb9H3TGoVbzb3gXEQ/fwP/rxBq</latexit><latexit sha1_base64="nDX1LPEw9acRWDJ2+4rO3goyV4=">ADQHicjVLdbtMwFE7C3wg/6+CSG4uqohOhSnoD0oQ0iSJNXBWJdpXmqnIcp7Vmx8Z2R4rxI/AEvA1vwRtwh7jlCqct08aExJGS79M53N8zpdcMqpNmn4Lo2vXb9y8tXM7vnP3v3d1t6DsRZLhckICybUJEeaMFqRkaGkYlUBPGckeP89FVTPz4jSlNRvTMrSaYczStaUoyMT81aXyBHZiGkXWOe29fOzWwNeVAOvAMjEHcgbNghjUrfBS/A/ClgqhO0FnbOyecvNEUduQ+eAkirsqBnVrzVgfgnLxfN/RmrXbaS9cBrpJsS9rBNoazvfAzLAReclIZzJDWJ1kqzdQiZShmxMVwqYlE+BTNyYmnFeJET+16iw50fKYApVD+qQxYZy8qLOK6GTnxqFc8b9AseAONQCcMGVL7SpLnPKm1l5FkrpBcUFwnJRPIJI3eCMG0A5cvY8oXU0sruTSkwpu7lEsGjACNa6CgimDVp4grKgfB+AF8js23ts4juGA+IEVGSLfWAz878CpL03sdsW1O+lPbdZb0QnBqCdwQNYNG2afiTwEzxo9zuxX1BFPmDBOaqKP1JnB+eH/Os7FlZCcWchGxNlPKo1egOzv+26Ssb9Xub527R9+GZr5U7wKHgcdIMseB4cBkfBMBgFOAzDJ2EaZtHX6Hv0I/q5aY3CreZhcCmiX78B5p8MhQ=</latexit>
slide-5
SLIDE 5

Generalization

CSE 599I: Generative Models University of Washington Lecture 2

  • Measure performance via likelihood of a test set :
  • Deep learning for classification: send training error to zero
  • Deep learning for generative modeling: send log-likelihood to zero?

xtest

1

, . . . , xtest

m

<latexit sha1_base64="UVzvziUkaKHczZrcQN/9AnGIbt4=">ACynicdVFLixNBEO7M+ljHV1aPXgZDYJVhyeSikMuCOYh4iGCygUyMPT01SbP9GLor68R2bt7EH+NVf4n/xp5kFeNiQfN9VNVXY+sFNxir/ezFRxcu37j5uGt8Padu/fut48eTKxeGwZjpoU204xaEFzBGDkKmJYGqMwEnGXnL5r42QUYy7V6i5sS5pIuFS84o+hdi3a3epciVOgQLNaLJE5zjTbe98pFu9M76W0tukqS9I5fTo4eE8IGS2OWl9IbaWoJAJau0s6ZU4d9QgZwLqMF1bKCk7p0uYeaqoBDt323nqOs9eVRo45/CaOv9W+GotJLiKvZoNzJrEFeygUZgY0F9z4SZ5mMK+tlEC8NLVecVXEhNMW40aPWwtbRfjNYPJ87rso1gmK7Xoq1iFBHzf6inBtgKDaeUGa4HydiK2oQ7/lMAzTIfiBDYyoT8yH/jCS+9DUpVwVOb+o6l/7o5r96R23TCKOk6SPMmzfKPkH5KB51+N/QLUvCBaSmpyn9Lazf8U+R/7hUaSNrl4oJGPRotugPmPx7rqtk0j9JPH/jL/mK7OyQPCKPyTFJyDNySl6SERkTRr6Qb+Q7+RG8DkywCdwuNWhdah6SPQs+/wLxBeO3</latexit><latexit sha1_base64="cZ7pKCTa2WetLTKJYD2J1QbFRGQ=">ACynicdVFLixNBEO7M+ljHV3Y9ehkMgVWGJZOLQi4Lm4OIhwgmG8jE0NTkzTbj6G7spvYzs2b+GO86u/w4H/wR9iTrGJcLGi+j6r6quRlYJb7HR+NIK9Gzdv3d6/E969d/Bw+bB4cjqpWEwZFpoM86oBcEVDJGjgHFpgMpMwFl2flrHzy7AWK7VW1yXMJV0rnjBGUXvmjXbq3cpwgodgsVqlsRprtHGu145a7Y6x52NRdJckVaJ896e5Pvhz8Hs4PGZ1+ILSUoZIJaO0k6JU4dNciZgCpMlxZKys7pHCaeKirBTt1mnipqe08eFdr4pzDaeP9WOCqtpLiIPdq1zGrEhayhFthYUN+9j8RZJuOV9TKI54aWC85WcSE0xbjWo9bCVtFuM1i8mDquyiWCYteiqWIUEf1/qKcG2Ao1p5QZrgfJ2ILaihDv+UwDNM+IENDKhPzPv+MJL70NilXBU5v1hVk+7UHVXuaeXaYRS1krSX5nWa5e8h/ZD2Wt126Bek4JpKanKf0sr1/9T5H/uFRpIyuXihEY9Gg26A+Y/Hu62TUPU48f+Mv+YpsbZ8Jk/IEUnIc3JCXpIBGRJGPpEv5Cv5FrwOTLAO3DY1aFxpHpEdCz7+Aq2c5YU=</latexit><latexit sha1_base64="cZ7pKCTa2WetLTKJYD2J1QbFRGQ=">ACynicdVFLixNBEO7M+ljHV3Y9ehkMgVWGJZOLQi4Lm4OIhwgmG8jE0NTkzTbj6G7spvYzs2b+GO86u/w4H/wR9iTrGJcLGi+j6r6quRlYJb7HR+NIK9Gzdv3d6/E969d/Bw+bB4cjqpWEwZFpoM86oBcEVDJGjgHFpgMpMwFl2flrHzy7AWK7VW1yXMJV0rnjBGUXvmjXbq3cpwgodgsVqlsRprtHGu145a7Y6x52NRdJckVaJ896e5Pvhz8Hs4PGZ1+ILSUoZIJaO0k6JU4dNciZgCpMlxZKys7pHCaeKirBTt1mnipqe08eFdr4pzDaeP9WOCqtpLiIPdq1zGrEhayhFthYUN+9j8RZJuOV9TKI54aWC85WcSE0xbjWo9bCVtFuM1i8mDquyiWCYteiqWIUEf1/qKcG2Ao1p5QZrgfJ2ILaihDv+UwDNM+IENDKhPzPv+MJL70NilXBU5v1hVk+7UHVXuaeXaYRS1krSX5nWa5e8h/ZD2Wt126Bek4JpKanKf0sr1/9T5H/uFRpIyuXihEY9Gg26A+Y/Hu62TUPU48f+Mv+YpsbZ8Jk/IEUnIc3JCXpIBGRJGPpEv5Cv5FrwOTLAO3DY1aFxpHpEdCz7+Aq2c5YU=</latexit><latexit sha1_base64="CrImFfmRg+MVksCpnRPvgCF5Xc=">ACynicdVFLixNBEO6Mr3V8ZfXoZTAEVhiWmVxW2MuCOYh4iGCygUwMPT01SbP9GLpr1sR2bt7EH+NVf4n/xp5sFONiQfN9VNVXY+8EtxikvzsBDdu3rp95+BueO/+g4ePuoePJ1bXhsGYaHNKcWBFcwRo4CpUBKnMB5/nFyzZ+fgnGcq3e4aCuaRLxUvOKHrXotfv8Q1ugQLDaLNM4KjTbe98pFt5cJ1uLrpN0R3pkZ6PFYerL8RqCQqZoNbO0qTCuaMGORPQhFltoaLsgi5h5qmiEuzcbedpor73FGpjX8Ko63b4Wj0kqKq9ij3ci8RVzJFlqBjQX13ftInOcyXlsvg3hpaLXibB2XQlOMWz1qLWwT7TeD5Yu546qERS76qWsRYQ6avcXFdwAQ7HxhDLD/TgRW1FDGfoth2GYDcEPbGBEfWIx9IeR3IemLuOqLPjlupkN5u6oc8b1w+jqJdmp1nRpln+EbJP2Wlv0A/9ghR8YFpKqorf0sYN/xT53z8uU9rIxmViAgY9mi36A6b/nus6mQyOU8/fJr2z17tTHpCn5Bk5Iik5IWfkFRmRMWHkC/lGvpMfwZvABJvAXaUGnZ3mCdmz4PMvylDirw=</latexit>

1 m

m

X

i=1

log pθ(xtest

i

).

<latexit sha1_base64="jyFjMZCSmOqiSu39n6vtB46Yzxw=">AC3HicdVFNixNBEO2Muq7jV1aPXhpDIAtDyOSywiIsmIN4imCygXQSenp6kma7p4fumjWxnZs32as3wR/hP/CqP8B/Y0+yiutiwfAeVfV6ql4lhRQWer2fjeDGzVt7t/fvhHfv3X/wsHnwaGx1aRgfMS21mSTUcilyPgIBk8Kw6lKJD9Nzl7U9dNzbqzQ+RvYFHym6DIXmWAUfGrRPCKZoczFlVMVsaVaOPE8ruYKE6mXuFgQWHGgnfWcAF+DA26hWojD7qLZ6nV728DXSXxJWifDr3tfELDxUHjgqSalYrnwCS1dhr3Cpg5akAwyauQlJYXlJ3RJZ96mlPF7cxtN6xw2dSnGnjvxzwNvu3wlFlFYV5NFuVFIjrFQNtcBGkvr5fSVKEhWtrZfxaGlosRJsHWVSU4hqPWgtbYWvDgPZs5kTeVECz9lulqyUGDSuHcWpMJyB3HhCmRF+HcxW1LsK3vcwDMmA+4UNH1LfmA78qZTwpYkjIs9Scb6upv2Z61TusHLtEONWTI5JWrdZ8Y6T9+S41W+H3qCcv2VaKZqnv6WVG/x5H/cSTXRlWOyDE34NFs0R8w/vdc18m43409f+0v+QrtYh89QU9RB8XoCJ2gl2iIRoihz+gb+o5+BPgQ/AxuNi1Bo1LzWN0JYJPvwBtgetq</latexit><latexit sha1_base64="MBAMutdQHbjeikEtL/ZfJ3Xnl0=">AC3HicdVFNixNBEO2Muq7jV1aPXhpDIAvDkslhUVYMAfxFMFkA+k9PT0JM12Tw/dNWtiO3jxJnv1pj9C/Ade9Qf4b+xJVnFdLBjeo6peT9WrpJDCQrf7sxFcu35j5+burfD2nbv37jf3HoysLg3jQ6alNuOEWi5FzocgQPJxYThVieQnyemzun5yxo0VOn8F64JPFV3kIhOMgk/Nm4ckM5S5uHKqIrZUcyextVMYSL1AhdzAksOtLOaEeArcMAtVHOxfzBvtroH3U3gqyS+IK3jwdedz+LAfzvcY5STUrFc+BSWrtJO4WMHXUgGCSVyEpLS8oO6ULPvE0p4rbqdtsWOG2z6Q408Z/OeBN9m+Fo8oqCsvIo12rpEZYqhpqgY0k9fP7SpQkKlpZL+PRwtBiKdgqyqSmENV60FraCl8eBrInUyfyogSes+0sWSkxaFw7ilNhOAO59oQyI/w6mC2pdxW872EYkj73Cxs+oL4x7ftTKeFLY0dEnqXibFVNelPXqdx+5dohxq2YHJG0brPiDSdvyVGr1w69QTl/zbRSNE9/SyvX/PI/7jSK6NqhyRI27Ao9mgP2D87muklHvIPb8pb/kC7SNXfQIPUYdFKNDdIyeowEaIoY+oW/oO/oRzIL3wYfgfNsaNC40D9GlCD7+Arhf7QE=</latexit><latexit sha1_base64="MBAMutdQHbjeikEtL/ZfJ3Xnl0=">AC3HicdVFNixNBEO2Muq7jV1aPXhpDIAvDkslhUVYMAfxFMFkA+k9PT0JM12Tw/dNWtiO3jxJnv1pj9C/Ade9Qf4b+xJVnFdLBjeo6peT9WrpJDCQrf7sxFcu35j5+burfD2nbv37jf3HoysLg3jQ6alNuOEWi5FzocgQPJxYThVieQnyemzun5yxo0VOn8F64JPFV3kIhOMgk/Nm4ckM5S5uHKqIrZUcyextVMYSL1AhdzAksOtLOaEeArcMAtVHOxfzBvtroH3U3gqyS+IK3jwdedz+LAfzvcY5STUrFc+BSWrtJO4WMHXUgGCSVyEpLS8oO6ULPvE0p4rbqdtsWOG2z6Q408Z/OeBN9m+Fo8oqCsvIo12rpEZYqhpqgY0k9fP7SpQkKlpZL+PRwtBiKdgqyqSmENV60FraCl8eBrInUyfyogSes+0sWSkxaFw7ilNhOAO59oQyI/w6mC2pdxW872EYkj73Cxs+oL4x7ftTKeFLY0dEnqXibFVNelPXqdx+5dohxq2YHJG0brPiDSdvyVGr1w69QTl/zbRSNE9/SyvX/PI/7jSK6NqhyRI27Ao9mgP2D87muklHvIPb8pb/kC7SNXfQIPUYdFKNDdIyeowEaIoY+oW/oO/oRzIL3wYfgfNsaNC40D9GlCD7+Arhf7QE=</latexit><latexit sha1_base64="kKHrMh21LmqUvUN2oYluOGFfuU=">AC3HicdVFNb9NAEN2Yr2I+msKRy4oUipZVZxLkSqkSuSAOAWJpJGyibVer5NVd73W7rgkL5xQ71y49dwhR/Av2GdBkSpGMl6TzPz1jNv0lIKC/3+z1Zw6/adu/f27ocPHj56vN8+eDKxujKMj5mW2kxTarkUBR+DAMmnpeFUpZKfpevmvrZBTdW6OIdbEo+V3RZiFwCj6VtI9Jbihzce1UTWylEidexvVCYSL1EpcJgRUH2lsvCPA1OAW6kQcHiXtTv+ovw18k8Q70kG7GCUHrUuSaVYpXgCT1NpZ3C9h7qgBwSvQ1JZXlJ2Tpd85mlBFbdzt92wxl2fyXCujf8KwNvs3wpHlVUVpFHu1Fpg7BSDTQCG0nq5/eVKE1VtLZexqOloeVKsHWUS0havSgtbQ1vj4M5C/mThRlBbxgV7PklcSgceMozoThDOTGE8qM8OtgtqLeVfC+h2FIhtwvbPiI+sZs6E+lhC9NHRFnomLdT0bzF2vdoe164Yd2JyQrKmzYoPnHwkJ51BN/QGFfw90rRIvstrd3wzyP/+48jhTaqdkROuAGPZov+gPG/57pJoOj2PO3/c7pm90p9Az9Bz1UIyO0Sl6jUZojBj6ir6h7+hHsAg+BZ+Dy6vWoLXTPEXIvjyC2+f6YA=</latexit>

E

x∼p − log pθ(x) ≥ H(p),

<latexit sha1_base64="L7iL2UoAq3/A8CQyaYhbyR2ha2s=">AC3icdVFNj9MwEHXD1xK+unDkYlGt1Eph1fQCYi+VKFLFqUi0W6muKseZtNbacbDdJcXkzAUQV278Gq5w5cZPwWkXxLJiJOs9zcyz/WaSQnBju90fjeDS5StXr+1dD2/cvHX7TnP/7sSotWYwZkoPU2oAcFzGFtuBUwLDVQmAo6Tk6d1/fgUtOEqf2k3BcwlXeY84xan1o0nxBJ7UoVbotJ4p5V1cKVxHCJiwo/JEItcbEgdgWtsOJkt4hYftohMtmq3uYXcb+CKJz0irP+y/4kQGi32Gx9IqthaQm6ZoMbM4m5h545qy5mAKiRrAwVlJ3QJM09zKsHM3dZkhQ98JsWZ0v7kFm+zfysclab2EHk0G5nUaFeyhlpgIkEtlL4SJYmMSuNlEC01LVaclVEmFLVRrbdKCVPh85+x2eO543mxtpCz3V+ytcBW4XqoOUamBUbTyjT3NvBbEU1ZdaPgxDMgBvWMOI+sZ04LcluS9NHeF5lvLTspr15q5duU7lDkKMWzE5ImndZvgbIG/JUat3EPoB5fCaKSlpnv6WVm7w5L/veNIrSsHBET0Naj3qJfYPzvui6Se8w9vyF3+RztIs9dB89QG0Uo0eoj4ZohMaIoS/oK/qGvgc0eBd8D7tWoPGmeYeOhfB5185BuY</latexit><latexit sha1_base64="7HVvLwsGlyQ0sBPZrw6kT5Glebo=">AC3icdVE9jxMxEHWr2P5uByUNBZRpERaTtlQgLgmEkGKqIJEcifFUeT1ThLr7PViO8cGszUNIFq6+zW0NJQ8zPwJgfiODGS9Z5m5tl+M0kuLGdzvdacOnylavXdq6HN27eur1b37szNmqlGYyYEkofJdSA4BmMLcCjnINVCYCDpPjp1X98AS04Sp7adc5TCVdZHzOGbU+Nas/IZLapcrdBpPEPSvLmSuI4RLnJX5AhFrgfEbsEixtFW1MFvAKD1p5O5rVG539zibwRKfkUZv0Hv/4/Thz+Fsr/aBpIqtJGSWCWrMJO7kduqotpwJKEOyMpBTdkwXMPE0oxLM1G1MlrjpMymeK+1PZvEm+7fCUWkqD5FHs5ZJhXYpK6gEJhLUQuErUZLIqDBeBtFC03zJWRHNhaI2qvRWKWFKfP4zdv546niWryxkbPuX+Upgq3A1VJxyDcyKtSeUae7tYLakmjLrRx+GIemDN6xhSH1j2vfbktyXjhzh2TzlJ0U56U5dq3Tt0jVDjBsxOSBp1Wb4GyBvyUGj2wz9gDJ4zZSUNEt/S0vX/3PJ/95xJFNalo6IMWjrUW/QLzD+d10Xybi7H3v+wm/yOdrGDrqH7qMWitEj1EMDNEQjxNAp+oK+om8BDd4FH4NP29agdqa5i85F8PkXyentUQ=</latexit><latexit sha1_base64="7HVvLwsGlyQ0sBPZrw6kT5Glebo=">AC3icdVE9jxMxEHWr2P5uByUNBZRpERaTtlQgLgmEkGKqIJEcifFUeT1ThLr7PViO8cGszUNIFq6+zW0NJQ8zPwJgfiODGS9Z5m5tl+M0kuLGdzvdacOnylavXdq6HN27eur1b37szNmqlGYyYEkofJdSA4BmMLcCjnINVCYCDpPjp1X98AS04Sp7adc5TCVdZHzOGbU+Nas/IZLapcrdBpPEPSvLmSuI4RLnJX5AhFrgfEbsEixtFW1MFvAKD1p5O5rVG539zibwRKfkUZv0Hv/4/Thz+Fsr/aBpIqtJGSWCWrMJO7kduqotpwJKEOyMpBTdkwXMPE0oxLM1G1MlrjpMymeK+1PZvEm+7fCUWkqD5FHs5ZJhXYpK6gEJhLUQuErUZLIqDBeBtFC03zJWRHNhaI2qvRWKWFKfP4zdv546niWryxkbPuX+Upgq3A1VJxyDcyKtSeUae7tYLakmjLrRx+GIemDN6xhSH1j2vfbktyXjhzh2TzlJ0U56U5dq3Tt0jVDjBsxOSBp1Wb4GyBvyUGj2wz9gDJ4zZSUNEt/S0vX/3PJ/95xJFNalo6IMWjrUW/QLzD+d10Xybi7H3v+wm/yOdrGDrqH7qMWitEj1EMDNEQjxNAp+oK+om8BDd4FH4NP29agdqa5i85F8PkXyentUQ=</latexit><latexit sha1_base64="SA0STz2gk6DaxtgZcHexUTLskAo=">AC3icdVFNj9MwEHXD1xK+unDkYlFVaqWwSnoBsZeVKNKU5Fot1JdVY4za2142C7S4rJmROIKzd+DVe48m9wugWxrBjJek8z82y/mbQU3Ng4/tkKrly9dv3G3s3w1u07d+19+9PjFprBmOmhNLTlBoQvICx5VbAtNRAZSrgJD193tRPzkAbrorXdlPCXNJlwXPOqPWpRfsZkdSuVOm2mKbuRV0vXEUMl7is8WMi1BKXC2JXYGmv6mOyhDf4uFf2o0W7Ex/E28CXSbIjHbSL0WK/9ZFkiq0lFJYJaswsiUs7d1RbzgTUIVkbKCk7pUuYeVpQCWbutiZr3PWZDOdK+1NYvM3+rXBUmsZD5NFsZNqgXckGoGJBLVQ+UqUpjKqjJdBtNS0XHFWRblQ1EaN3iolTI0vfsbmT+eOF+XaQsHO/5KvBbYKN0PFGdfArNh4Qpnm3g5mK6ops370YRiSIXjDGkbUN2ZDvy3JfWnqC/yjJ9V9Wwd73a9WvXDTHuJOSQZE2b4e+AvCeHnUE39AMq4C1TUtIi+y2t3fDPJf97x5FCaVk7IiagrUe9Rb/A5N91XSaTwUHi+au4c/Ryt8o9BA9Qj2UoCfoCB2jERojhr6ib+g7+hHQ4EPwKfh83hq0dpoH6EIEX34BocfpZA=</latexit>

pθ(x) ≈ (

1 n

if x ∈ {x1, . . . , xn},

  • therwise.
<latexit sha1_base64="TgL7BksiyswGP5KA7u6MJBNLCc=">ACY3icbVFda9RAFJ1ErTVG6tvIgxulQrLkohQHwu+iE8V3Laws4TJ7M3u0MlkmLnRbIf8Sd98X/4ew2oLZeGDic+3HuPVMaJR1m2Y8ovnP3s793QfJ3sNHj/fTJwdnrmtgKloVGMvSu5ASQ1TlKjgwljgdangvLz8sMmfwXrZKO/4NrAvOZLSspOAaqSK9MwXAFyI+6N5RxY2zTJayEpdRehLmuT1hlufB573VPX1OG0KGnsqKHWVSU+a7Ih+zRYNu3BWa9Yf9mLEk+1PbBAH7TroJwkDvRgGF+kom2TboLdBPoARGeK0SL8HFdHWoFEo7twszwzOPbcohYKwaOvAcHJlzALUPMa3NxvPerpq8AsaNXY8DTSLft3h+e1c+u6DJU1x5W7mduQ/8vNWqzez73UpkXQ4lqoahXFhm4MpwtpQaBaB8CFlWFXKlY8WIrhW5JgQn7z5Nvg7O0kD/jzu9HJp8GOXfKcvCRHJCfH5IR8JKdkSgT5Ge1E+1Ea/Yr34oP42XVpHA09T8k/Eb/4DRbgtjY=</latexit><latexit sha1_base64="TgL7BksiyswGP5KA7u6MJBNLCc=">ACY3icbVFda9RAFJ1ErTVG6tvIgxulQrLkohQHwu+iE8V3Laws4TJ7M3u0MlkmLnRbIf8Sd98X/4ew2oLZeGDic+3HuPVMaJR1m2Y8ovnP3s793QfJ3sNHj/fTJwdnrmtgKloVGMvSu5ASQ1TlKjgwljgdangvLz8sMmfwXrZKO/4NrAvOZLSspOAaqSK9MwXAFyI+6N5RxY2zTJayEpdRehLmuT1hlufB573VPX1OG0KGnsqKHWVSU+a7Ih+zRYNu3BWa9Yf9mLEk+1PbBAH7TroJwkDvRgGF+kom2TboLdBPoARGeK0SL8HFdHWoFEo7twszwzOPbcohYKwaOvAcHJlzALUPMa3NxvPerpq8AsaNXY8DTSLft3h+e1c+u6DJU1x5W7mduQ/8vNWqzez73UpkXQ4lqoahXFhm4MpwtpQaBaB8CFlWFXKlY8WIrhW5JgQn7z5Nvg7O0kD/jzu9HJp8GOXfKcvCRHJCfH5IR8JKdkSgT5Ge1E+1Ea/Yr34oP42XVpHA09T8k/Eb/4DRbgtjY=</latexit><latexit sha1_base64="TgL7BksiyswGP5KA7u6MJBNLCc=">ACY3icbVFda9RAFJ1ErTVG6tvIgxulQrLkohQHwu+iE8V3Laws4TJ7M3u0MlkmLnRbIf8Sd98X/4ew2oLZeGDic+3HuPVMaJR1m2Y8ovnP3s793QfJ3sNHj/fTJwdnrmtgKloVGMvSu5ASQ1TlKjgwljgdangvLz8sMmfwXrZKO/4NrAvOZLSspOAaqSK9MwXAFyI+6N5RxY2zTJayEpdRehLmuT1hlufB573VPX1OG0KGnsqKHWVSU+a7Ih+zRYNu3BWa9Yf9mLEk+1PbBAH7TroJwkDvRgGF+kom2TboLdBPoARGeK0SL8HFdHWoFEo7twszwzOPbcohYKwaOvAcHJlzALUPMa3NxvPerpq8AsaNXY8DTSLft3h+e1c+u6DJU1x5W7mduQ/8vNWqzez73UpkXQ4lqoahXFhm4MpwtpQaBaB8CFlWFXKlY8WIrhW5JgQn7z5Nvg7O0kD/jzu9HJp8GOXfKcvCRHJCfH5IR8JKdkSgT5Ge1E+1Ea/Yr34oP42XVpHA09T8k/Eb/4DRbgtjY=</latexit><latexit sha1_base64="TgL7BksiyswGP5KA7u6MJBNLCc=">ACY3icbVFda9RAFJ1ErTVG6tvIgxulQrLkohQHwu+iE8V3Laws4TJ7M3u0MlkmLnRbIf8Sd98X/4ew2oLZeGDic+3HuPVMaJR1m2Y8ovnP3s793QfJ3sNHj/fTJwdnrmtgKloVGMvSu5ASQ1TlKjgwljgdangvLz8sMmfwXrZKO/4NrAvOZLSspOAaqSK9MwXAFyI+6N5RxY2zTJayEpdRehLmuT1hlufB573VPX1OG0KGnsqKHWVSU+a7Ih+zRYNu3BWa9Yf9mLEk+1PbBAH7TroJwkDvRgGF+kom2TboLdBPoARGeK0SL8HFdHWoFEo7twszwzOPbcohYKwaOvAcHJlzALUPMa3NxvPerpq8AsaNXY8DTSLft3h+e1c+u6DJU1x5W7mduQ/8vNWqzez73UpkXQ4lqoahXFhm4MpwtpQaBaB8CFlWFXKlY8WIrhW5JgQn7z5Nvg7O0kD/jzu9HJp8GOXfKcvCRHJCfH5IR8JKdkSgT5Ge1E+1Ea/Yr34oP42XVpHA09T8k/Eb/4DRbgtjY=</latexit>
slide-6
SLIDE 6

Gaussian Mixture Models

CSE 599I: Generative Models University of Washington Lecture 2

  • K-Gaussian mixture model over data .
  • Each data point belongs to a latent cluster .
  • Each cluster is drawn from a Gaussian with mean and variance :

x ∈ Rd

<latexit sha1_base64="2iHd9J1Pz0Js3raher6L4B19pus=">AB+3icbVDLSsNAFL2pr1pfsS7dDBbBVUlE0GXBjbiqYh/QxDKZTtqhk0mYmUhLyK+4caGIW3/EnX/jtM1CWw8MHM65l3vmBAlnSjvOt1VaW9/Y3CpvV3Z29/YP7MNqW8WpJLRFYh7LboAV5UzQlma024iKY4CTjvB+Hrmd56oVCwWD3qaUD/CQ8FCRrA2Ut+uTpDHBPIirEdBkN3nj4O+XPqzhxolbgFqUGBZt/+8gYxSMqNOFYqZ7rJNrPsNSMcJpXvFTRBJMxHtKeoQJHVPnZPHuOTo0yQGEszRMazdXfGxmOlJpGgZmcZVTL3kz8z+ulOrzyMyaSVFNBFofClCMdo1kRaMAkJZpPDcFEMpMVkRGWmGhTV8WU4C5/eZW0z+u4XcXtcZtUcZjuEzsCFS2jADTShBQm8Ayv8Gbl1ov1bn0sRktWsXMEf2B9/gCTB5Qo</latexit><latexit sha1_base64="2iHd9J1Pz0Js3raher6L4B19pus=">AB+3icbVDLSsNAFL2pr1pfsS7dDBbBVUlE0GXBjbiqYh/QxDKZTtqhk0mYmUhLyK+4caGIW3/EnX/jtM1CWw8MHM65l3vmBAlnSjvOt1VaW9/Y3CpvV3Z29/YP7MNqW8WpJLRFYh7LboAV5UzQlma024iKY4CTjvB+Hrmd56oVCwWD3qaUD/CQ8FCRrA2Ut+uTpDHBPIirEdBkN3nj4O+XPqzhxolbgFqUGBZt/+8gYxSMqNOFYqZ7rJNrPsNSMcJpXvFTRBJMxHtKeoQJHVPnZPHuOTo0yQGEszRMazdXfGxmOlJpGgZmcZVTL3kz8z+ulOrzyMyaSVFNBFofClCMdo1kRaMAkJZpPDcFEMpMVkRGWmGhTV8WU4C5/eZW0z+u4XcXtcZtUcZjuEzsCFS2jADTShBQm8Ayv8Gbl1ov1bn0sRktWsXMEf2B9/gCTB5Qo</latexit><latexit sha1_base64="2iHd9J1Pz0Js3raher6L4B19pus=">AB+3icbVDLSsNAFL2pr1pfsS7dDBbBVUlE0GXBjbiqYh/QxDKZTtqhk0mYmUhLyK+4caGIW3/EnX/jtM1CWw8MHM65l3vmBAlnSjvOt1VaW9/Y3CpvV3Z29/YP7MNqW8WpJLRFYh7LboAV5UzQlma024iKY4CTjvB+Hrmd56oVCwWD3qaUD/CQ8FCRrA2Ut+uTpDHBPIirEdBkN3nj4O+XPqzhxolbgFqUGBZt/+8gYxSMqNOFYqZ7rJNrPsNSMcJpXvFTRBJMxHtKeoQJHVPnZPHuOTo0yQGEszRMazdXfGxmOlJpGgZmcZVTL3kz8z+ulOrzyMyaSVFNBFofClCMdo1kRaMAkJZpPDcFEMpMVkRGWmGhTV8WU4C5/eZW0z+u4XcXtcZtUcZjuEzsCFS2jADTShBQm8Ayv8Gbl1ov1bn0sRktWsXMEf2B9/gCTB5Qo</latexit><latexit sha1_base64="2iHd9J1Pz0Js3raher6L4B19pus=">AB+3icbVDLSsNAFL2pr1pfsS7dDBbBVUlE0GXBjbiqYh/QxDKZTtqhk0mYmUhLyK+4caGIW3/EnX/jtM1CWw8MHM65l3vmBAlnSjvOt1VaW9/Y3CpvV3Z29/YP7MNqW8WpJLRFYh7LboAV5UzQlma024iKY4CTjvB+Hrmd56oVCwWD3qaUD/CQ8FCRrA2Ut+uTpDHBPIirEdBkN3nj4O+XPqzhxolbgFqUGBZt/+8gYxSMqNOFYqZ7rJNrPsNSMcJpXvFTRBJMxHtKeoQJHVPnZPHuOTo0yQGEszRMazdXfGxmOlJpGgZmcZVTL3kz8z+ulOrzyMyaSVFNBFofClCMdo1kRaMAkJZpPDcFEMpMVkRGWmGhTV8WU4C5/eZW0z+u4XcXtcZtUcZjuEzsCFS2jADTShBQm8Ayv8Gbl1ov1bn0sRktWsXMEf2B9/gCTB5Qo</latexit>

z ∈ {1, . . . , K}

<latexit sha1_base64="sFxzT+ysOg0JgCPVrW1N4AVh9Lo=">AB/HicbVDLSsNAFJ34rPUV7dLNYBFclJKIoMuCG9FNBfuAJpTJdNIOnUzCzI0Q/0VNy4UceuHuPNvnLZaOuBgcM593DvnCARXIPjfFsrq2vrG5ulrfL2zu7evn1w2NZxqihr0VjEqhsQzQSXrAUcBOsmipEoEKwTjK+mfueBKc1jeQ9ZwvyIDCUPOSVgpL5decQel9jL3Zo3iEHXbr1J3646dWcGvEzcglRgWbf/jJZmkZMAhVE657rJODnRAGngk3KXqpZQuiYDFnPUEkipv18dvwEnxhlgMNYmScBz9TfiZxEWmdRYCYjAiO96E3F/7xeCuGln3OZpMAknS8KU4EhxtMm8IArRkFkhCquLkV0xFRhILpq2xKcBe/vEzaZ3X8LvzauOmqKOEjtAxOkUukANdI2aqIUoytAzekVv1pP1Yr1bH/PRFavIVNAfWJ8/YpeT+A=</latexit><latexit sha1_base64="sFxzT+ysOg0JgCPVrW1N4AVh9Lo=">AB/HicbVDLSsNAFJ34rPUV7dLNYBFclJKIoMuCG9FNBfuAJpTJdNIOnUzCzI0Q/0VNy4UceuHuPNvnLZaOuBgcM593DvnCARXIPjfFsrq2vrG5ulrfL2zu7evn1w2NZxqihr0VjEqhsQzQSXrAUcBOsmipEoEKwTjK+mfueBKc1jeQ9ZwvyIDCUPOSVgpL5decQel9jL3Zo3iEHXbr1J3646dWcGvEzcglRgWbf/jJZmkZMAhVE657rJODnRAGngk3KXqpZQuiYDFnPUEkipv18dvwEnxhlgMNYmScBz9TfiZxEWmdRYCYjAiO96E3F/7xeCuGln3OZpMAknS8KU4EhxtMm8IArRkFkhCquLkV0xFRhILpq2xKcBe/vEzaZ3X8LvzauOmqKOEjtAxOkUukANdI2aqIUoytAzekVv1pP1Yr1bH/PRFavIVNAfWJ8/YpeT+A=</latexit><latexit sha1_base64="sFxzT+ysOg0JgCPVrW1N4AVh9Lo=">AB/HicbVDLSsNAFJ34rPUV7dLNYBFclJKIoMuCG9FNBfuAJpTJdNIOnUzCzI0Q/0VNy4UceuHuPNvnLZaOuBgcM593DvnCARXIPjfFsrq2vrG5ulrfL2zu7evn1w2NZxqihr0VjEqhsQzQSXrAUcBOsmipEoEKwTjK+mfueBKc1jeQ9ZwvyIDCUPOSVgpL5decQel9jL3Zo3iEHXbr1J3646dWcGvEzcglRgWbf/jJZmkZMAhVE657rJODnRAGngk3KXqpZQuiYDFnPUEkipv18dvwEnxhlgMNYmScBz9TfiZxEWmdRYCYjAiO96E3F/7xeCuGln3OZpMAknS8KU4EhxtMm8IArRkFkhCquLkV0xFRhILpq2xKcBe/vEzaZ3X8LvzauOmqKOEjtAxOkUukANdI2aqIUoytAzekVv1pP1Yr1bH/PRFavIVNAfWJ8/YpeT+A=</latexit><latexit sha1_base64="sFxzT+ysOg0JgCPVrW1N4AVh9Lo=">AB/HicbVDLSsNAFJ34rPUV7dLNYBFclJKIoMuCG9FNBfuAJpTJdNIOnUzCzI0Q/0VNy4UceuHuPNvnLZaOuBgcM593DvnCARXIPjfFsrq2vrG5ulrfL2zu7evn1w2NZxqihr0VjEqhsQzQSXrAUcBOsmipEoEKwTjK+mfueBKc1jeQ9ZwvyIDCUPOSVgpL5decQel9jL3Zo3iEHXbr1J3646dWcGvEzcglRgWbf/jJZmkZMAhVE657rJODnRAGngk3KXqpZQuiYDFnPUEkipv18dvwEnxhlgMNYmScBz9TfiZxEWmdRYCYjAiO96E3F/7xeCuGln3OZpMAknS8KU4EhxtMm8IArRkFkhCquLkV0xFRhILpq2xKcBe/vEzaZ3X8LvzauOmqKOEjtAxOkUukANdI2aqIUoytAzekVv1pP1Yr1bH/PRFavIVNAfWJ8/YpeT+A=</latexit>

x

<latexit sha1_base64="cZOxaYaxnsygdTcbG7e+ELDjna0=">AB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY8CKeWrAf0Iay2U7atZtN2N2IJfQXePGgiFd/kjf/jds2B219YeHhnRl25g0SwbVx3W+nsLa+sblV3C7t7O7tH5QPj1o6ThXDJotFrDoB1Si4xKbhRmAnUijQGA7GN/M6u1HVJrH8t5MEvQjOpQ85IwazWe+uWKW3XnIqvg5VCBXPV+as3iFkaoTRMUK27npsYP6PKcCZwWuqlGhPKxnSIXYuSRqj9bL7olJxZ0DCWNknDZm7vycyGmk9iQLbGVEz0su1mflfrZua8NrPuExSg5ItPgpTQUxMZleTAVfIjJhYoExuythI6oMzabkg3BWz5FVoXVc9y47JSu8vjKMIJnMI5eHAFNbiFOjSBAcIzvMKb8+C8O/Ox6K14OQzx/BHzucP6KONBA=</latexit><latexit sha1_base64="cZOxaYaxnsygdTcbG7e+ELDjna0=">AB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY8CKeWrAf0Iay2U7atZtN2N2IJfQXePGgiFd/kjf/jds2B219YeHhnRl25g0SwbVx3W+nsLa+sblV3C7t7O7tH5QPj1o6ThXDJotFrDoB1Si4xKbhRmAnUijQGA7GN/M6u1HVJrH8t5MEvQjOpQ85IwazWe+uWKW3XnIqvg5VCBXPV+as3iFkaoTRMUK27npsYP6PKcCZwWuqlGhPKxnSIXYuSRqj9bL7olJxZ0DCWNknDZm7vycyGmk9iQLbGVEz0su1mflfrZua8NrPuExSg5ItPgpTQUxMZleTAVfIjJhYoExuythI6oMzabkg3BWz5FVoXVc9y47JSu8vjKMIJnMI5eHAFNbiFOjSBAcIzvMKb8+C8O/Ox6K14OQzx/BHzucP6KONBA=</latexit><latexit sha1_base64="cZOxaYaxnsygdTcbG7e+ELDjna0=">AB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY8CKeWrAf0Iay2U7atZtN2N2IJfQXePGgiFd/kjf/jds2B219YeHhnRl25g0SwbVx3W+nsLa+sblV3C7t7O7tH5QPj1o6ThXDJotFrDoB1Si4xKbhRmAnUijQGA7GN/M6u1HVJrH8t5MEvQjOpQ85IwazWe+uWKW3XnIqvg5VCBXPV+as3iFkaoTRMUK27npsYP6PKcCZwWuqlGhPKxnSIXYuSRqj9bL7olJxZ0DCWNknDZm7vycyGmk9iQLbGVEz0su1mflfrZua8NrPuExSg5ItPgpTQUxMZleTAVfIjJhYoExuythI6oMzabkg3BWz5FVoXVc9y47JSu8vjKMIJnMI5eHAFNbiFOjSBAcIzvMKb8+C8O/Ox6K14OQzx/BHzucP6KONBA=</latexit><latexit sha1_base64="cZOxaYaxnsygdTcbG7e+ELDjna0=">AB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY8CKeWrAf0Iay2U7atZtN2N2IJfQXePGgiFd/kjf/jds2B219YeHhnRl25g0SwbVx3W+nsLa+sblV3C7t7O7tH5QPj1o6ThXDJotFrDoB1Si4xKbhRmAnUijQGA7GN/M6u1HVJrH8t5MEvQjOpQ85IwazWe+uWKW3XnIqvg5VCBXPV+as3iFkaoTRMUK27npsYP6PKcCZwWuqlGhPKxnSIXYuSRqj9bL7olJxZ0DCWNknDZm7vycyGmk9iQLbGVEz0su1mflfrZua8NrPuExSg5ItPgpTQUxMZleTAVfIjJhYoExuythI6oMzabkg3BWz5FVoXVc9y47JSu8vjKMIJnMI5eHAFNbiFOjSBAcIzvMKb8+C8O/Ox6K14OQzx/BHzucP6KONBA=</latexit>

µk

<latexit sha1_base64="E1mYO8TCc4J794h/gLjB+0xZ3M=">AB7XicbZDLSgMxFIbP1Fut1GXboJFcFVmRNBlwY24qmAv0A4lk2ba2FyGJCOUoe/gxoUibn0fd76NaTsLbf0h8PGfc8g5f5xyZmwQfHultfWNza3ydmVnd2/wD8ahmVaUKbRHGlOzE2lDNJm5ZTjupljEnLbj8c2s3n6i2jAlH+wkpZHAQ8kSRrB1Vgv1RNYf9/1qUAvmQqsQFlCFQo2+/9UbKJIJKi3h2JhuGKQ2yrG2jHA6rfQyQ1NMxnhIuw4lFtRE+XzbKTpzgAlSrsnLZq7vydyLIyZiNh1CmxHZrk2M/+rdTObXEc5k2lmqSLj5KMI6vQ7HQ0YJoSycOMNHM7YrICGtMrAuo4kIl09ehdZFLXR8f1mt3xVxlOETuEcQriCOtxCA5pA4BGe4RXePOW9eO/ex6K15BUzx/BH3ucPNMeO5g=</latexit><latexit sha1_base64="E1mYO8TCc4J794h/gLjB+0xZ3M=">AB7XicbZDLSgMxFIbP1Fut1GXboJFcFVmRNBlwY24qmAv0A4lk2ba2FyGJCOUoe/gxoUibn0fd76NaTsLbf0h8PGfc8g5f5xyZmwQfHultfWNza3ydmVnd2/wD8ahmVaUKbRHGlOzE2lDNJm5ZTjupljEnLbj8c2s3n6i2jAlH+wkpZHAQ8kSRrB1Vgv1RNYf9/1qUAvmQqsQFlCFQo2+/9UbKJIJKi3h2JhuGKQ2yrG2jHA6rfQyQ1NMxnhIuw4lFtRE+XzbKTpzgAlSrsnLZq7vydyLIyZiNh1CmxHZrk2M/+rdTObXEc5k2lmqSLj5KMI6vQ7HQ0YJoSycOMNHM7YrICGtMrAuo4kIl09ehdZFLXR8f1mt3xVxlOETuEcQriCOtxCA5pA4BGe4RXePOW9eO/ex6K15BUzx/BH3ucPNMeO5g=</latexit><latexit sha1_base64="E1mYO8TCc4J794h/gLjB+0xZ3M=">AB7XicbZDLSgMxFIbP1Fut1GXboJFcFVmRNBlwY24qmAv0A4lk2ba2FyGJCOUoe/gxoUibn0fd76NaTsLbf0h8PGfc8g5f5xyZmwQfHultfWNza3ydmVnd2/wD8ahmVaUKbRHGlOzE2lDNJm5ZTjupljEnLbj8c2s3n6i2jAlH+wkpZHAQ8kSRrB1Vgv1RNYf9/1qUAvmQqsQFlCFQo2+/9UbKJIJKi3h2JhuGKQ2yrG2jHA6rfQyQ1NMxnhIuw4lFtRE+XzbKTpzgAlSrsnLZq7vydyLIyZiNh1CmxHZrk2M/+rdTObXEc5k2lmqSLj5KMI6vQ7HQ0YJoSycOMNHM7YrICGtMrAuo4kIl09ehdZFLXR8f1mt3xVxlOETuEcQriCOtxCA5pA4BGe4RXePOW9eO/ex6K15BUzx/BH3ucPNMeO5g=</latexit><latexit sha1_base64="E1mYO8TCc4J794h/gLjB+0xZ3M=">AB7XicbZDLSgMxFIbP1Fut1GXboJFcFVmRNBlwY24qmAv0A4lk2ba2FyGJCOUoe/gxoUibn0fd76NaTsLbf0h8PGfc8g5f5xyZmwQfHultfWNza3ydmVnd2/wD8ahmVaUKbRHGlOzE2lDNJm5ZTjupljEnLbj8c2s3n6i2jAlH+wkpZHAQ8kSRrB1Vgv1RNYf9/1qUAvmQqsQFlCFQo2+/9UbKJIJKi3h2JhuGKQ2yrG2jHA6rfQyQ1NMxnhIuw4lFtRE+XzbKTpzgAlSrsnLZq7vydyLIyZiNh1CmxHZrk2M/+rdTObXEc5k2lmqSLj5KMI6vQ7HQ0YJoSycOMNHM7YrICGtMrAuo4kIl09ehdZFLXR8f1mt3xVxlOETuEcQriCOtxCA5pA4BGe4RXePOW9eO/ex6K15BUzx/BH3ucPNMeO5g=</latexit>

Σk

<latexit sha1_base64="aTiChw9kzEpyv5FJz6fBv+9EtR4=">AB73icbZBNSwMxEIZn61etX1WPXoJF8FR2RdBjwYt4qmhroV3KbJptQ5PsmSFUvonvHhQxKt/x5v/xrTdg7a+EHh4Z4bMvFEquLG+/+0VlbX1jeKm6Wt7Z3dvfL+QdMkmasQROR6FaEhgmuWMNyK1gr1QxlJNhDNLya1h+emDY8Ufd2lLJQYl/xmFO0zmp17nhfYnfYLVf8qj8TWYghwrkqnfLX51eQjPJlKUCjWkHfmrDMWrLqWCTUiczLEU6xD5rO1QomQnHs30n5MQ5PRIn2j1lycz9PTFGacxIRq5Toh2YxdrU/K/Wzmx8GY65SjPLFJ1/FGeC2IRMjyc9rhm1YuQAqeZuV0IHqJFaF1HJhRAsnrwMzbNq4Pj2vFK7yeMowhEcwykEcAE1uIY6NICgGd4hTfv0Xvx3r2PeWvBy2cO4Y+8zx/u14/p</latexit><latexit sha1_base64="aTiChw9kzEpyv5FJz6fBv+9EtR4=">AB73icbZBNSwMxEIZn61etX1WPXoJF8FR2RdBjwYt4qmhroV3KbJptQ5PsmSFUvonvHhQxKt/x5v/xrTdg7a+EHh4Z4bMvFEquLG+/+0VlbX1jeKm6Wt7Z3dvfL+QdMkmasQROR6FaEhgmuWMNyK1gr1QxlJNhDNLya1h+emDY8Ufd2lLJQYl/xmFO0zmp17nhfYnfYLVf8qj8TWYghwrkqnfLX51eQjPJlKUCjWkHfmrDMWrLqWCTUiczLEU6xD5rO1QomQnHs30n5MQ5PRIn2j1lycz9PTFGacxIRq5Toh2YxdrU/K/Wzmx8GY65SjPLFJ1/FGeC2IRMjyc9rhm1YuQAqeZuV0IHqJFaF1HJhRAsnrwMzbNq4Pj2vFK7yeMowhEcwykEcAE1uIY6NICgGd4hTfv0Xvx3r2PeWvBy2cO4Y+8zx/u14/p</latexit><latexit sha1_base64="aTiChw9kzEpyv5FJz6fBv+9EtR4=">AB73icbZBNSwMxEIZn61etX1WPXoJF8FR2RdBjwYt4qmhroV3KbJptQ5PsmSFUvonvHhQxKt/x5v/xrTdg7a+EHh4Z4bMvFEquLG+/+0VlbX1jeKm6Wt7Z3dvfL+QdMkmasQROR6FaEhgmuWMNyK1gr1QxlJNhDNLya1h+emDY8Ufd2lLJQYl/xmFO0zmp17nhfYnfYLVf8qj8TWYghwrkqnfLX51eQjPJlKUCjWkHfmrDMWrLqWCTUiczLEU6xD5rO1QomQnHs30n5MQ5PRIn2j1lycz9PTFGacxIRq5Toh2YxdrU/K/Wzmx8GY65SjPLFJ1/FGeC2IRMjyc9rhm1YuQAqeZuV0IHqJFaF1HJhRAsnrwMzbNq4Pj2vFK7yeMowhEcwykEcAE1uIY6NICgGd4hTfv0Xvx3r2PeWvBy2cO4Y+8zx/u14/p</latexit><latexit sha1_base64="aTiChw9kzEpyv5FJz6fBv+9EtR4=">AB73icbZBNSwMxEIZn61etX1WPXoJF8FR2RdBjwYt4qmhroV3KbJptQ5PsmSFUvonvHhQxKt/x5v/xrTdg7a+EHh4Z4bMvFEquLG+/+0VlbX1jeKm6Wt7Z3dvfL+QdMkmasQROR6FaEhgmuWMNyK1gr1QxlJNhDNLya1h+emDY8Ufd2lLJQYl/xmFO0zmp17nhfYnfYLVf8qj8TWYghwrkqnfLX51eQjPJlKUCjWkHfmrDMWrLqWCTUiczLEU6xD5rO1QomQnHs30n5MQ5PRIn2j1lycz9PTFGacxIRq5Toh2YxdrU/K/Wzmx8GY65SjPLFJ1/FGeC2IRMjyc9rhm1YuQAqeZuV0IHqJFaF1HJhRAsnrwMzbNq4Pj2vFK7yeMowhEcwykEcAE1uIY6NICgGd4hTfv0Xvx3r2PeWvBy2cO4Y+8zx/u14/p</latexit>

1. z ∼ Categoricalπ(K), π ∈ ∆K−1, 2. x ∼ N(µz, Σz), µ ∈ RK×d, Σ ∈ RK×d×d.

<latexit sha1_base64="sHLRgD0vK0OXgjRL6G8G9df8E=">ADnHicdVJdb9MwFE1WPkb46uARCVlUqzoUqZMAmkvk7YHxDQYH+sm5lI5jtas+PIdkZW47/Fz+Cdf8NWwZjmqXoHl2fe3zvPUkLwY3t9X6FK40bN2/dXr0T3b13/8HD5tqjgVGlpuyQKqH0cUoMEzxnh5ZbwY4LzYhMBTtKT3fq+6Mzpg1X+Wd7XrChJOcjzklFlKj5s920sVTUxDK3EspPZohbLhE2LKuh1i2URpYAs/wgXv7G3EqI3aABHmOcK7TFjy1e29SHyMcdTuXxarlmKS2ClouHe+g2U5msX4E59IMpot5WS5kKt5aeo+epDElktmUOZjtGBfT7mgdkfNVq/bmx90FSRL0AqW52C0Fv7AmaKlZLmlghzkvQKO3REW04F8xEuDYOBTsmEnQDMCbw0dPFe7QOmQyNlYvt2ie/bfCEWnqhmOI5lymdbRTWYe6wMQCFlzBTZymMq7m4snmhRTqt4LBSxcV1vlRLGo8vN2PHroeN5UVqW0Uv41Igq1BtNMq4ZtSKcwCEag7jIDolmlALv0MUgXUwr2b7IP+YJpYpZ87TDRsuvIw/wTHNfrLPCgmIHjXHLQOHbgxzjZ5U/6Q9dx7sN79YjhFoJ3sJZTN8xvB3vNXqr0ewyZx9o0pKkmd/Sr3bvRC57h2Hc6Wld1gMmLYQ9TyC08n/vl4Fg343Afxhs7X9dun5avAkeBZ0giR4FWwHb4KD4DCg4Wb4JaRh1nja2G3sNfYX1JVwWfM4uHQag9+aoipR</latexit><latexit sha1_base64="sHLRgD0vK0OXgjRL6G8G9df8E=">ADnHicdVJdb9MwFE1WPkb46uARCVlUqzoUqZMAmkvk7YHxDQYH+sm5lI5jtas+PIdkZW47/Fz+Cdf8NWwZjmqXoHl2fe3zvPUkLwY3t9X6FK40bN2/dXr0T3b13/8HD5tqjgVGlpuyQKqH0cUoMEzxnh5ZbwY4LzYhMBTtKT3fq+6Mzpg1X+Wd7XrChJOcjzklFlKj5s920sVTUxDK3EspPZohbLhE2LKuh1i2URpYAs/wgXv7G3EqI3aABHmOcK7TFjy1e29SHyMcdTuXxarlmKS2ClouHe+g2U5msX4E59IMpot5WS5kKt5aeo+epDElktmUOZjtGBfT7mgdkfNVq/bmx90FSRL0AqW52C0Fv7AmaKlZLmlghzkvQKO3REW04F8xEuDYOBTsmEnQDMCbw0dPFe7QOmQyNlYvt2ie/bfCEWnqhmOI5lymdbRTWYe6wMQCFlzBTZymMq7m4snmhRTqt4LBSxcV1vlRLGo8vN2PHroeN5UVqW0Uv41Igq1BtNMq4ZtSKcwCEag7jIDolmlALv0MUgXUwr2b7IP+YJpYpZ87TDRsuvIw/wTHNfrLPCgmIHjXHLQOHbgxzjZ5U/6Q9dx7sN79YjhFoJ3sJZTN8xvB3vNXqr0ewyZx9o0pKkmd/Sr3bvRC57h2Hc6Wld1gMmLYQ9TyC08n/vl4Fg343Afxhs7X9dun5avAkeBZ0giR4FWwHb4KD4DCg4Wb4JaRh1nja2G3sNfYX1JVwWfM4uHQag9+aoipR</latexit><latexit sha1_base64="sHLRgD0vK0OXgjRL6G8G9df8E=">ADnHicdVJdb9MwFE1WPkb46uARCVlUqzoUqZMAmkvk7YHxDQYH+sm5lI5jtas+PIdkZW47/Fz+Cdf8NWwZjmqXoHl2fe3zvPUkLwY3t9X6FK40bN2/dXr0T3b13/8HD5tqjgVGlpuyQKqH0cUoMEzxnh5ZbwY4LzYhMBTtKT3fq+6Mzpg1X+Wd7XrChJOcjzklFlKj5s920sVTUxDK3EspPZohbLhE2LKuh1i2URpYAs/wgXv7G3EqI3aABHmOcK7TFjy1e29SHyMcdTuXxarlmKS2ClouHe+g2U5msX4E59IMpot5WS5kKt5aeo+epDElktmUOZjtGBfT7mgdkfNVq/bmx90FSRL0AqW52C0Fv7AmaKlZLmlghzkvQKO3REW04F8xEuDYOBTsmEnQDMCbw0dPFe7QOmQyNlYvt2ie/bfCEWnqhmOI5lymdbRTWYe6wMQCFlzBTZymMq7m4snmhRTqt4LBSxcV1vlRLGo8vN2PHroeN5UVqW0Uv41Igq1BtNMq4ZtSKcwCEag7jIDolmlALv0MUgXUwr2b7IP+YJpYpZ87TDRsuvIw/wTHNfrLPCgmIHjXHLQOHbgxzjZ5U/6Q9dx7sN79YjhFoJ3sJZTN8xvB3vNXqr0ewyZx9o0pKkmd/Sr3bvRC57h2Hc6Wld1gMmLYQ9TyC08n/vl4Fg343Afxhs7X9dun5avAkeBZ0giR4FWwHb4KD4DCg4Wb4JaRh1nja2G3sNfYX1JVwWfM4uHQag9+aoipR</latexit><latexit sha1_base64="sHLRgD0vK0OXgjRL6G8G9df8E=">ADnHicdVJdb9MwFE1WPkb46uARCVlUqzoUqZMAmkvk7YHxDQYH+sm5lI5jtas+PIdkZW47/Fz+Cdf8NWwZjmqXoHl2fe3zvPUkLwY3t9X6FK40bN2/dXr0T3b13/8HD5tqjgVGlpuyQKqH0cUoMEzxnh5ZbwY4LzYhMBTtKT3fq+6Mzpg1X+Wd7XrChJOcjzklFlKj5s920sVTUxDK3EspPZohbLhE2LKuh1i2URpYAs/wgXv7G3EqI3aABHmOcK7TFjy1e29SHyMcdTuXxarlmKS2ClouHe+g2U5msX4E59IMpot5WS5kKt5aeo+epDElktmUOZjtGBfT7mgdkfNVq/bmx90FSRL0AqW52C0Fv7AmaKlZLmlghzkvQKO3REW04F8xEuDYOBTsmEnQDMCbw0dPFe7QOmQyNlYvt2ie/bfCEWnqhmOI5lymdbRTWYe6wMQCFlzBTZymMq7m4snmhRTqt4LBSxcV1vlRLGo8vN2PHroeN5UVqW0Uv41Igq1BtNMq4ZtSKcwCEag7jIDolmlALv0MUgXUwr2b7IP+YJpYpZ87TDRsuvIw/wTHNfrLPCgmIHjXHLQOHbgxzjZ5U/6Q9dx7sN79YjhFoJ3sJZTN8xvB3vNXqr0ewyZx9o0pKkmd/Sr3bvRC57h2Hc6Wld1gMmLYQ9TyC08n/vl4Fg343Afxhs7X9dun5avAkeBZ0giR4FWwHb4KD4DCg4Wb4JaRh1nja2G3sNfYX1JVwWfM4uHQag9+aoipR</latexit>
slide-7
SLIDE 7

Gaussian Mixture Models

CSE 599I: Generative Models University of Washington Lecture 2

  • Generative model:
  • Likelihood:

p(x) = Z

Z

p(x, z) dz = Z

Z

p(x|z)p(z) dz =

K

X

k=1

πkp(x|z = k) =

K

X

k=1

πkN(x; µk, Σk).

<latexit sha1_base64="5dRKxMlNhbPuwesmUj1qs/Dh8HQ=">AChnicbVHRatswFJXdrc2yrU3Xx72IhYUYQrDTlgxGIbCXscHo6NKURZmRFSURlmQhyaOJ60/ZT+1tfzM5CWNtc0FwOdc6ercRHFmbBj+8fy9J0/3D2rP6s9fvDw8ahy/ujZrgkdkoxn+ibBhnIm6dAy+mN0hSLhNRkn6o9NFPqg3L5De7VHQi8FyGSPYOipu/FLt2wC2LiBi0sZIYLsgmBfS+iEzipAnekK7lbvVoFqbxwI1VuVy+QiLtKLqPzxGSLF4nTjc1Ia7Nb/3fmlbN+RyKP0w6YnOB4zToxo1m2A3XBR+DaAuaYFuXceM3mYkF1RawrEx4yhUdlJgbRnhtKyj3FCFSYrndOygxIKaSbGOsYRvHTOFs0y7Iy1cs/93FgYsxSJc1ZTm4daRe7SxrmdvZsUTKrcUk2D81yDm0Gq53AKdOUWL50ABPN3KyQLDGxLrN1V0I0cMvPwbXvW7k8Nez5uDTNo4aeA3egDaIQB8MwEdwCYaAeHte4PW8U7/md/1zv7+x+t625wTcK3/wF0Lsvyw=</latexit><latexit sha1_base64="5dRKxMlNhbPuwesmUj1qs/Dh8HQ=">AChnicbVHRatswFJXdrc2yrU3Xx72IhYUYQrDTlgxGIbCXscHo6NKURZmRFSURlmQhyaOJ60/ZT+1tfzM5CWNtc0FwOdc6ercRHFmbBj+8fy9J0/3D2rP6s9fvDw8ahy/ujZrgkdkoxn+ibBhnIm6dAy+mN0hSLhNRkn6o9NFPqg3L5De7VHQi8FyGSPYOipu/FLt2wC2LiBi0sZIYLsgmBfS+iEzipAnekK7lbvVoFqbxwI1VuVy+QiLtKLqPzxGSLF4nTjc1Ia7Nb/3fmlbN+RyKP0w6YnOB4zToxo1m2A3XBR+DaAuaYFuXceM3mYkF1RawrEx4yhUdlJgbRnhtKyj3FCFSYrndOygxIKaSbGOsYRvHTOFs0y7Iy1cs/93FgYsxSJc1ZTm4daRe7SxrmdvZsUTKrcUk2D81yDm0Gq53AKdOUWL50ABPN3KyQLDGxLrN1V0I0cMvPwbXvW7k8Nez5uDTNo4aeA3egDaIQB8MwEdwCYaAeHte4PW8U7/md/1zv7+x+t625wTcK3/wF0Lsvyw=</latexit><latexit sha1_base64="5dRKxMlNhbPuwesmUj1qs/Dh8HQ=">AChnicbVHRatswFJXdrc2yrU3Xx72IhYUYQrDTlgxGIbCXscHo6NKURZmRFSURlmQhyaOJ60/ZT+1tfzM5CWNtc0FwOdc6ercRHFmbBj+8fy9J0/3D2rP6s9fvDw8ahy/ujZrgkdkoxn+ibBhnIm6dAy+mN0hSLhNRkn6o9NFPqg3L5De7VHQi8FyGSPYOipu/FLt2wC2LiBi0sZIYLsgmBfS+iEzipAnekK7lbvVoFqbxwI1VuVy+QiLtKLqPzxGSLF4nTjc1Ia7Nb/3fmlbN+RyKP0w6YnOB4zToxo1m2A3XBR+DaAuaYFuXceM3mYkF1RawrEx4yhUdlJgbRnhtKyj3FCFSYrndOygxIKaSbGOsYRvHTOFs0y7Iy1cs/93FgYsxSJc1ZTm4daRe7SxrmdvZsUTKrcUk2D81yDm0Gq53AKdOUWL50ABPN3KyQLDGxLrN1V0I0cMvPwbXvW7k8Nez5uDTNo4aeA3egDaIQB8MwEdwCYaAeHte4PW8U7/md/1zv7+x+t625wTcK3/wF0Lsvyw=</latexit><latexit sha1_base64="5dRKxMlNhbPuwesmUj1qs/Dh8HQ=">AChnicbVHRatswFJXdrc2yrU3Xx72IhYUYQrDTlgxGIbCXscHo6NKURZmRFSURlmQhyaOJ60/ZT+1tfzM5CWNtc0FwOdc6ercRHFmbBj+8fy9J0/3D2rP6s9fvDw8ahy/ujZrgkdkoxn+ibBhnIm6dAy+mN0hSLhNRkn6o9NFPqg3L5De7VHQi8FyGSPYOipu/FLt2wC2LiBi0sZIYLsgmBfS+iEzipAnekK7lbvVoFqbxwI1VuVy+QiLtKLqPzxGSLF4nTjc1Ia7Nb/3fmlbN+RyKP0w6YnOB4zToxo1m2A3XBR+DaAuaYFuXceM3mYkF1RawrEx4yhUdlJgbRnhtKyj3FCFSYrndOygxIKaSbGOsYRvHTOFs0y7Iy1cs/93FgYsxSJc1ZTm4daRe7SxrmdvZsUTKrcUk2D81yDm0Gq53AKdOUWL50ABPN3KyQLDGxLrN1V0I0cMvPwbXvW7k8Nez5uDTNo4aeA3egDaIQB8MwEdwCYaAeHte4PW8U7/md/1zv7+x+t625wTcK3/wF0Lsvyw=</latexit>

1. z ∼ Categoricalπ(K), π ∈ ∆K−1, 2. x ∼ N(µz, Σz), µ ∈ RK×d, Σ ∈ RK×d×d.

<latexit sha1_base64="sHLRgD0vK0OXgjRL6G8G9df8E=">ADnHicdVJdb9MwFE1WPkb46uARCVlUqzoUqZMAmkvk7YHxDQYH+sm5lI5jtas+PIdkZW47/Fz+Cdf8NWwZjmqXoHl2fe3zvPUkLwY3t9X6FK40bN2/dXr0T3b13/8HD5tqjgVGlpuyQKqH0cUoMEzxnh5ZbwY4LzYhMBTtKT3fq+6Mzpg1X+Wd7XrChJOcjzklFlKj5s920sVTUxDK3EspPZohbLhE2LKuh1i2URpYAs/wgXv7G3EqI3aABHmOcK7TFjy1e29SHyMcdTuXxarlmKS2ClouHe+g2U5msX4E59IMpot5WS5kKt5aeo+epDElktmUOZjtGBfT7mgdkfNVq/bmx90FSRL0AqW52C0Fv7AmaKlZLmlghzkvQKO3REW04F8xEuDYOBTsmEnQDMCbw0dPFe7QOmQyNlYvt2ie/bfCEWnqhmOI5lymdbRTWYe6wMQCFlzBTZymMq7m4snmhRTqt4LBSxcV1vlRLGo8vN2PHroeN5UVqW0Uv41Igq1BtNMq4ZtSKcwCEag7jIDolmlALv0MUgXUwr2b7IP+YJpYpZ87TDRsuvIw/wTHNfrLPCgmIHjXHLQOHbgxzjZ5U/6Q9dx7sN79YjhFoJ3sJZTN8xvB3vNXqr0ewyZx9o0pKkmd/Sr3bvRC57h2Hc6Wld1gMmLYQ9TyC08n/vl4Fg343Afxhs7X9dun5avAkeBZ0giR4FWwHb4KD4DCg4Wb4JaRh1nja2G3sNfYX1JVwWfM4uHQag9+aoipR</latexit><latexit sha1_base64="sHLRgD0vK0OXgjRL6G8G9df8E=">ADnHicdVJdb9MwFE1WPkb46uARCVlUqzoUqZMAmkvk7YHxDQYH+sm5lI5jtas+PIdkZW47/Fz+Cdf8NWwZjmqXoHl2fe3zvPUkLwY3t9X6FK40bN2/dXr0T3b13/8HD5tqjgVGlpuyQKqH0cUoMEzxnh5ZbwY4LzYhMBTtKT3fq+6Mzpg1X+Wd7XrChJOcjzklFlKj5s920sVTUxDK3EspPZohbLhE2LKuh1i2URpYAs/wgXv7G3EqI3aABHmOcK7TFjy1e29SHyMcdTuXxarlmKS2ClouHe+g2U5msX4E59IMpot5WS5kKt5aeo+epDElktmUOZjtGBfT7mgdkfNVq/bmx90FSRL0AqW52C0Fv7AmaKlZLmlghzkvQKO3REW04F8xEuDYOBTsmEnQDMCbw0dPFe7QOmQyNlYvt2ie/bfCEWnqhmOI5lymdbRTWYe6wMQCFlzBTZymMq7m4snmhRTqt4LBSxcV1vlRLGo8vN2PHroeN5UVqW0Uv41Igq1BtNMq4ZtSKcwCEag7jIDolmlALv0MUgXUwr2b7IP+YJpYpZ87TDRsuvIw/wTHNfrLPCgmIHjXHLQOHbgxzjZ5U/6Q9dx7sN79YjhFoJ3sJZTN8xvB3vNXqr0ewyZx9o0pKkmd/Sr3bvRC57h2Hc6Wld1gMmLYQ9TyC08n/vl4Fg343Afxhs7X9dun5avAkeBZ0giR4FWwHb4KD4DCg4Wb4JaRh1nja2G3sNfYX1JVwWfM4uHQag9+aoipR</latexit><latexit sha1_base64="sHLRgD0vK0OXgjRL6G8G9df8E=">ADnHicdVJdb9MwFE1WPkb46uARCVlUqzoUqZMAmkvk7YHxDQYH+sm5lI5jtas+PIdkZW47/Fz+Cdf8NWwZjmqXoHl2fe3zvPUkLwY3t9X6FK40bN2/dXr0T3b13/8HD5tqjgVGlpuyQKqH0cUoMEzxnh5ZbwY4LzYhMBTtKT3fq+6Mzpg1X+Wd7XrChJOcjzklFlKj5s920sVTUxDK3EspPZohbLhE2LKuh1i2URpYAs/wgXv7G3EqI3aABHmOcK7TFjy1e29SHyMcdTuXxarlmKS2ClouHe+g2U5msX4E59IMpot5WS5kKt5aeo+epDElktmUOZjtGBfT7mgdkfNVq/bmx90FSRL0AqW52C0Fv7AmaKlZLmlghzkvQKO3REW04F8xEuDYOBTsmEnQDMCbw0dPFe7QOmQyNlYvt2ie/bfCEWnqhmOI5lymdbRTWYe6wMQCFlzBTZymMq7m4snmhRTqt4LBSxcV1vlRLGo8vN2PHroeN5UVqW0Uv41Igq1BtNMq4ZtSKcwCEag7jIDolmlALv0MUgXUwr2b7IP+YJpYpZ87TDRsuvIw/wTHNfrLPCgmIHjXHLQOHbgxzjZ5U/6Q9dx7sN79YjhFoJ3sJZTN8xvB3vNXqr0ewyZx9o0pKkmd/Sr3bvRC57h2Hc6Wld1gMmLYQ9TyC08n/vl4Fg343Afxhs7X9dun5avAkeBZ0giR4FWwHb4KD4DCg4Wb4JaRh1nja2G3sNfYX1JVwWfM4uHQag9+aoipR</latexit><latexit sha1_base64="sHLRgD0vK0OXgjRL6G8G9df8E=">ADnHicdVJdb9MwFE1WPkb46uARCVlUqzoUqZMAmkvk7YHxDQYH+sm5lI5jtas+PIdkZW47/Fz+Cdf8NWwZjmqXoHl2fe3zvPUkLwY3t9X6FK40bN2/dXr0T3b13/8HD5tqjgVGlpuyQKqH0cUoMEzxnh5ZbwY4LzYhMBTtKT3fq+6Mzpg1X+Wd7XrChJOcjzklFlKj5s920sVTUxDK3EspPZohbLhE2LKuh1i2URpYAs/wgXv7G3EqI3aABHmOcK7TFjy1e29SHyMcdTuXxarlmKS2ClouHe+g2U5msX4E59IMpot5WS5kKt5aeo+epDElktmUOZjtGBfT7mgdkfNVq/bmx90FSRL0AqW52C0Fv7AmaKlZLmlghzkvQKO3REW04F8xEuDYOBTsmEnQDMCbw0dPFe7QOmQyNlYvt2ie/bfCEWnqhmOI5lymdbRTWYe6wMQCFlzBTZymMq7m4snmhRTqt4LBSxcV1vlRLGo8vN2PHroeN5UVqW0Uv41Igq1BtNMq4ZtSKcwCEag7jIDolmlALv0MUgXUwr2b7IP+YJpYpZ87TDRsuvIw/wTHNfrLPCgmIHjXHLQOHbgxzjZ5U/6Q9dx7sN79YjhFoJ3sJZTN8xvB3vNXqr0ewyZx9o0pKkmd/Sr3bvRC57h2Hc6Wld1gMmLYQ9TyC08n/vl4Fg343Afxhs7X9dun5avAkeBZ0giR4FWwHb4KD4DCg4Wb4JaRh1nja2G3sNfYX1JVwWfM4uHQag9+aoipR</latexit>
slide-8
SLIDE 8
  • Generative model:
  • The maximum likelihood estimator (parameters in red):
  • No analytical solution; non-convex optimization problem

Maximize the Likelihood

CSE 599I: Generative Models University of Washington Lecture 2

1. z ∼ Categoricalπ(K), π ∈ ∆K−1, 2. x ∼ N(µz, Σz), µ ∈ RK×d, Σ ∈ RK×d×d.

<latexit sha1_base64="0ZIi5qClM5ibVvQcWEs0nQ9kPX4=">ADw3icdVLbhMxEN1tuJTl0hYebGIGqVoibIBCaS+VGqREBVQLr1IcYi8Xiexaq9XtlO2MX7gFf6Hf+FDeGd2GyihdCRrjsZzsx4nBaCG9vt/giXGleuXru+fCO6ev2nZXVtbsHRk01ZftUCaWPUmKY4Dnbt9wKdlRoRmQq2GF6vF3dH54wbjKP9jTg0kGed8xCmxEBqu/mwlHTwxBaHMPZbSoxnChkuELSut2yaWjZWGbOGHuODt3Y0YtVDL4bqy0yzEaY5wjvMGHJR7f7KPE+xjhq9RaVy7myJHYCgu61b2M5Hc5i/J6PJRnO/qMtp2faFSdN3TsP+thyQzKoApaSK5lLs8/53WGq81up1sbugiSOWgGc9sbroXfcaboVLcUkGM6Sfdwg4c0ZTwXyEp4bBqMdkzPoAcwKlBq5uzqN1iGRopDSc3KI6+jfDEWmqjmPw5lSmlbcTWbmKYGIBeyjhJk5TGZf1m8ZjTYoJp2U8EorYuOJbpYTxaLEZO3o2cDwvpbl9KyX0VQgq1D1H1DGNaNWnAIgVHMYB9EJ0YRa+DVRBFuFeTV7BfJvCqaJVfqhw0TDU5ce5h/juELnmXsEFDP4DFxy0DhysJBRxk9K3+8NXNu7De/WI4SaCd7EWZVm+Izhz3iz2VuP4CVz9okqKUme/aZ6t/NH5LI6DudKS+wOGDagte1h0n/+71IjodRLAb580t17Od74c3A8eBO0gCZ4GW8GLYC/YD2jYD7+EX8NvjeN4Zu2LPUpXDOuRcsWMP/Are8PMk=</latexit><latexit sha1_base64="0ZIi5qClM5ibVvQcWEs0nQ9kPX4=">ADw3icdVLbhMxEN1tuJTl0hYebGIGqVoibIBCaS+VGqREBVQLr1IcYi8Xiexaq9XtlO2MX7gFf6Hf+FDeGd2GyihdCRrjsZzsx4nBaCG9vt/giXGleuXru+fCO6ev2nZXVtbsHRk01ZftUCaWPUmKY4Dnbt9wKdlRoRmQq2GF6vF3dH54wbjKP9jTg0kGed8xCmxEBqu/mwlHTwxBaHMPZbSoxnChkuELSut2yaWjZWGbOGHuODt3Y0YtVDL4bqy0yzEaY5wjvMGHJR7f7KPE+xjhq9RaVy7myJHYCgu61b2M5Hc5i/J6PJRnO/qMtp2faFSdN3TsP+thyQzKoApaSK5lLs8/53WGq81up1sbugiSOWgGc9sbroXfcaboVLcUkGM6Sfdwg4c0ZTwXyEp4bBqMdkzPoAcwKlBq5uzqN1iGRopDSc3KI6+jfDEWmqjmPw5lSmlbcTWbmKYGIBeyjhJk5TGZf1m8ZjTYoJp2U8EorYuOJbpYTxaLEZO3o2cDwvpbl9KyX0VQgq1D1H1DGNaNWnAIgVHMYB9EJ0YRa+DVRBFuFeTV7BfJvCqaJVfqhw0TDU5ce5h/juELnmXsEFDP4DFxy0DhysJBRxk9K3+8NXNu7De/WI4SaCd7EWZVm+Izhz3iz2VuP4CVz9okqKUme/aZ6t/NH5LI6DudKS+wOGDagte1h0n/+71IjodRLAb580t17Od74c3A8eBO0gCZ4GW8GLYC/YD2jYD7+EX8NvjeN4Zu2LPUpXDOuRcsWMP/Are8PMk=</latexit><latexit sha1_base64="0ZIi5qClM5ibVvQcWEs0nQ9kPX4=">ADw3icdVLbhMxEN1tuJTl0hYebGIGqVoibIBCaS+VGqREBVQLr1IcYi8Xiexaq9XtlO2MX7gFf6Hf+FDeGd2GyihdCRrjsZzsx4nBaCG9vt/giXGleuXru+fCO6ev2nZXVtbsHRk01ZftUCaWPUmKY4Dnbt9wKdlRoRmQq2GF6vF3dH54wbjKP9jTg0kGed8xCmxEBqu/mwlHTwxBaHMPZbSoxnChkuELSut2yaWjZWGbOGHuODt3Y0YtVDL4bqy0yzEaY5wjvMGHJR7f7KPE+xjhq9RaVy7myJHYCgu61b2M5Hc5i/J6PJRnO/qMtp2faFSdN3TsP+thyQzKoApaSK5lLs8/53WGq81up1sbugiSOWgGc9sbroXfcaboVLcUkGM6Sfdwg4c0ZTwXyEp4bBqMdkzPoAcwKlBq5uzqN1iGRopDSc3KI6+jfDEWmqjmPw5lSmlbcTWbmKYGIBeyjhJk5TGZf1m8ZjTYoJp2U8EorYuOJbpYTxaLEZO3o2cDwvpbl9KyX0VQgq1D1H1DGNaNWnAIgVHMYB9EJ0YRa+DVRBFuFeTV7BfJvCqaJVfqhw0TDU5ce5h/juELnmXsEFDP4DFxy0DhysJBRxk9K3+8NXNu7De/WI4SaCd7EWZVm+Izhz3iz2VuP4CVz9okqKUme/aZ6t/NH5LI6DudKS+wOGDagte1h0n/+71IjodRLAb580t17Od74c3A8eBO0gCZ4GW8GLYC/YD2jYD7+EX8NvjeN4Zu2LPUpXDOuRcsWMP/Are8PMk=</latexit><latexit sha1_base64="0ZIi5qClM5ibVvQcWEs0nQ9kPX4=">ADw3icdVLbhMxEN1tuJTl0hYebGIGqVoibIBCaS+VGqREBVQLr1IcYi8Xiexaq9XtlO2MX7gFf6Hf+FDeGd2GyihdCRrjsZzsx4nBaCG9vt/giXGleuXru+fCO6ev2nZXVtbsHRk01ZftUCaWPUmKY4Dnbt9wKdlRoRmQq2GF6vF3dH54wbjKP9jTg0kGed8xCmxEBqu/mwlHTwxBaHMPZbSoxnChkuELSut2yaWjZWGbOGHuODt3Y0YtVDL4bqy0yzEaY5wjvMGHJR7f7KPE+xjhq9RaVy7myJHYCgu61b2M5Hc5i/J6PJRnO/qMtp2faFSdN3TsP+thyQzKoApaSK5lLs8/53WGq81up1sbugiSOWgGc9sbroXfcaboVLcUkGM6Sfdwg4c0ZTwXyEp4bBqMdkzPoAcwKlBq5uzqN1iGRopDSc3KI6+jfDEWmqjmPw5lSmlbcTWbmKYGIBeyjhJk5TGZf1m8ZjTYoJp2U8EorYuOJbpYTxaLEZO3o2cDwvpbl9KyX0VQgq1D1H1DGNaNWnAIgVHMYB9EJ0YRa+DVRBFuFeTV7BfJvCqaJVfqhw0TDU5ce5h/juELnmXsEFDP4DFxy0DhysJBRxk9K3+8NXNu7De/WI4SaCd7EWZVm+Izhz3iz2VuP4CVz9okqKUme/aZ6t/NH5LI6DudKS+wOGDagte1h0n/+71IjodRLAb580t17Od74c3A8eBO0gCZ4GW8GLYC/YD2jYD7+EX8NvjeN4Zu2LPUpXDOuRcsWMP/Are8PMk=</latexit>

sup

θ

1 n

n

X

i=1

log pθ(xi) = sup

θ

1 n

n

X

i=1

log

K

X

k=1

πkN(xi; µk, Σk).

<latexit sha1_base64="TrGTnKDeIBp3KVMaEj80TuQ5ybg=">ADoniclVLZbhMxFJ1pWErYWnjkxSJUatGoykRIKpKlcgDKgJSlS5SHQaPx5OYeBnZnpJi/Ft8CS/8DXeSsISKByxZ9/j6nGNfX+eV4NZ1u9/jldaVq9eur95o37x1+87dtfV7x1bXhrIjqoU2pzmxTHDFjhx3gp1WhGZC3aST140+yfnzFiu1Tt3UbGhJCPFS06Jg1S29g3buso8njl5w4qA3Zg5EhAuDaE+DV4F4MjM890vFcICz1CVTanbU4zvoV20f+6NCbz9aRZv0JL2opnE5BK4saUCP8mNMfsLFkDZRkKXIR5JAdms7W+t0t7uzgS6DdAE60WIMsvX4Ky40rSVTjgpi7VnardzQE+M4FSy0cW1ZReiEjNgZQEUks0M/OzygDcgUqNQGpnJolv1T4Ym0TS0JRHsh8ya6sWxCI7CJI5NYSfJc5lMLchYMjKkGnM6TUqhiUsavdNa2ICWL+PKZ0PVU7puj8LmUtkNOoaTcquGHUiQsAhBoO5SA6JtATB5+i3cZ9BvUa9hrs31bMEKfNY4+JgaecBqh/hJMG/WYOCDgWfhwkoPHqcdclQU/n4az3tBvBr8V/EYboU6Kd3DR0Cz/zPAXvNPpbThJRX7RLWURBU/pcH3f5n86xyPlTYyeCyOmXEQzSxCp9O/+3oZHPe2U8AHTzp7+4uer0YPofRZpRGT6O96GU0iI4iGj+P8Q8/th61NpvHbQO59SVeKG5Hy2NFv4B+JE1mg=</latexit><latexit sha1_base64="TrGTnKDeIBp3KVMaEj80TuQ5ybg=">ADoniclVLZbhMxFJ1pWErYWnjkxSJUatGoykRIKpKlcgDKgJSlS5SHQaPx5OYeBnZnpJi/Ft8CS/8DXeSsISKByxZ9/j6nGNfX+eV4NZ1u9/jldaVq9eur95o37x1+87dtfV7x1bXhrIjqoU2pzmxTHDFjhx3gp1WhGZC3aST140+yfnzFiu1Tt3UbGhJCPFS06Jg1S29g3buso8njl5w4qA3Zg5EhAuDaE+DV4F4MjM890vFcICz1CVTanbU4zvoV20f+6NCbz9aRZv0JL2opnE5BK4saUCP8mNMfsLFkDZRkKXIR5JAdms7W+t0t7uzgS6DdAE60WIMsvX4Ky40rSVTjgpi7VnardzQE+M4FSy0cW1ZReiEjNgZQEUks0M/OzygDcgUqNQGpnJolv1T4Ym0TS0JRHsh8ya6sWxCI7CJI5NYSfJc5lMLchYMjKkGnM6TUqhiUsavdNa2ICWL+PKZ0PVU7puj8LmUtkNOoaTcquGHUiQsAhBoO5SA6JtATB5+i3cZ9BvUa9hrs31bMEKfNY4+JgaecBqh/hJMG/WYOCDgWfhwkoPHqcdclQU/n4az3tBvBr8V/EYboU6Kd3DR0Cz/zPAXvNPpbThJRX7RLWURBU/pcH3f5n86xyPlTYyeCyOmXEQzSxCp9O/+3oZHPe2U8AHTzp7+4uer0YPofRZpRGT6O96GU0iI4iGj+P8Q8/th61NpvHbQO59SVeKG5Hy2NFv4B+JE1mg=</latexit><latexit sha1_base64="TrGTnKDeIBp3KVMaEj80TuQ5ybg=">ADoniclVLZbhMxFJ1pWErYWnjkxSJUatGoykRIKpKlcgDKgJSlS5SHQaPx5OYeBnZnpJi/Ft8CS/8DXeSsISKByxZ9/j6nGNfX+eV4NZ1u9/jldaVq9eur95o37x1+87dtfV7x1bXhrIjqoU2pzmxTHDFjhx3gp1WhGZC3aST140+yfnzFiu1Tt3UbGhJCPFS06Jg1S29g3buso8njl5w4qA3Zg5EhAuDaE+DV4F4MjM890vFcICz1CVTanbU4zvoV20f+6NCbz9aRZv0JL2opnE5BK4saUCP8mNMfsLFkDZRkKXIR5JAdms7W+t0t7uzgS6DdAE60WIMsvX4Ky40rSVTjgpi7VnardzQE+M4FSy0cW1ZReiEjNgZQEUks0M/OzygDcgUqNQGpnJolv1T4Ym0TS0JRHsh8ya6sWxCI7CJI5NYSfJc5lMLchYMjKkGnM6TUqhiUsavdNa2ICWL+PKZ0PVU7puj8LmUtkNOoaTcquGHUiQsAhBoO5SA6JtATB5+i3cZ9BvUa9hrs31bMEKfNY4+JgaecBqh/hJMG/WYOCDgWfhwkoPHqcdclQU/n4az3tBvBr8V/EYboU6Kd3DR0Cz/zPAXvNPpbThJRX7RLWURBU/pcH3f5n86xyPlTYyeCyOmXEQzSxCp9O/+3oZHPe2U8AHTzp7+4uer0YPofRZpRGT6O96GU0iI4iGj+P8Q8/th61NpvHbQO59SVeKG5Hy2NFv4B+JE1mg=</latexit><latexit sha1_base64="lm0MUkOzo5hm7fNgs9x04ULkiGs=">ACvnicdVLbhMxFHWGVxkKtGs2I6JIBY2qmWxA6gaJLNhUBImkeJRdcdzJ7Hqx8h2SlJ3foAt/8K/8Dd40vAoFVeyztG9Ptf34bIR3Los+9GL7t1/8PDR3uP4yX789Nnzg/2p1SvDcMK0GZWgkXBFU4cdwJnjUGQpcCz8uJ9Fz+7RGO5Vp/dpsFCwkLxmjNwTU+P+hnx9nWkrsk35E+2dn5Ye87rTRbSVSOCbB2nmeNKzwYx5nANqYriw2wC1jgPFAFEm3ht3W2ySB4qTWJhzlkq3b4UHaSW4ZRrQbmTZoVvKDjqBTQU4XIdIWpYyXdsgw3RhoFlytk5rocGlnd5pLWyb3C7G1W8Lz1WzcqjYTS31SiROJ91ckobZE5sAgFmeGgnYUswFyYXhzTEYZ+DZ6G9B8bNOC0e0pmIWEdRv6X9C0Y39ujiFkrEZhM5KHDNPuaorfrlu58PCH7X+VesHcZL0c3pCq+6a5VdIr+lJfziIwyQVfmFaSlDVL2nrR7+T/O8dT5U2svVUTNG4gGaLYdH5v2u9S6bD4zwTxnZIy/IS3JEcvKGvCMfyJhMCMV+Uq+RaeRja5uPkTU2/2MQ3LoufAi3gmA=</latexit><latexit sha1_base64="Vq6G3z5yj3l1yO6l7zoX6BkJhAY=">ADl3iclVLbhMxEPU2XEotOWVF4tQqUWrKpsXEFUlJPqAQECr0otUh8Xr9SYmvqxsb0kx/i2+hBf+htmkXELFA5asOR7POePxTFL4Xy/z1Z6ly7fuPm8q3u7ZU7d1fX1leOnWks40fMSGNPC+q4FJofeElP60tp6qQ/KSYPG/vT865dcLod/6i5kNFR1pUglEPrnztG3FNnQcyUwqWl5H4Mfc0YlJZykIWg4Qo/IgdrP4XmMizQjX+Txsc5qLbyL/1elFZmfJ+35FV7g1iKfAFVRP2ZUhjexTbOzEKIaCEkXIdipCh4t7bztV5/uz9b+CrILkEPXa79fD35SkrDGsW1Z5I6d5b1az8M1HrBJI9d0jheUzahI34GUFPF3TDMke8AZ4SV8bC1h7PvH8yAlWurSUF6y5U0Vo/Vq1pCS6V1PMp3KRFodKpAxpPR5bWY8GmaSUN9WnL98ZIF/HiY3z1ZBiErhvPNZu/pWok9ga37calsJx5eQGAMiugHMzGFHriYSi6XbLHoV7LX4P825pb6o19FAi18JXTCPWPSNqi35H7FBTLPRg4JUDjNBChq1KcT+PZYBg2Y9iKYaOLcS8jO6Rsw5z4zMkXstMbHThJzX/xIxSVJc/qTHs/RL5V5AtLEqBiKPufVg7cxCp7O/+3oVHA+2M8AHfbSM7qMHaBNl6DF6hl6gfXSEWPI0+ZCI5GPnYedl52A+E0vJ5XDcQwurc/gDjsc0gw=</latexit><latexit sha1_base64="Vq6G3z5yj3l1yO6l7zoX6BkJhAY=">ADl3iclVLbhMxEPU2XEotOWVF4tQqUWrKpsXEFUlJPqAQECr0otUh8Xr9SYmvqxsb0kx/i2+hBf+htmkXELFA5asOR7POePxTFL4Xy/z1Z6ly7fuPm8q3u7ZU7d1fX1leOnWks40fMSGNPC+q4FJofeElP60tp6qQ/KSYPG/vT865dcLod/6i5kNFR1pUglEPrnztG3FNnQcyUwqWl5H4Mfc0YlJZykIWg4Qo/IgdrP4XmMizQjX+Txsc5qLbyL/1elFZmfJ+35FV7g1iKfAFVRP2ZUhjexTbOzEKIaCEkXIdipCh4t7bztV5/uz9b+CrILkEPXa79fD35SkrDGsW1Z5I6d5b1az8M1HrBJI9d0jheUzahI34GUFPF3TDMke8AZ4SV8bC1h7PvH8yAlWurSUF6y5U0Vo/Vq1pCS6V1PMp3KRFodKpAxpPR5bWY8GmaSUN9WnL98ZIF/HiY3z1ZBiErhvPNZu/pWok9ga37calsJx5eQGAMiugHMzGFHriYSi6XbLHoV7LX4P825pb6o19FAi18JXTCPWPSNqi35H7FBTLPRg4JUDjNBChq1KcT+PZYBg2Y9iKYaOLcS8jO6Rsw5z4zMkXstMbHThJzX/xIxSVJc/qTHs/RL5V5AtLEqBiKPufVg7cxCp7O/+3oVHA+2M8AHfbSM7qMHaBNl6DF6hl6gfXSEWPI0+ZCI5GPnYedl52A+E0vJ5XDcQwurc/gDjsc0gw=</latexit><latexit sha1_base64="IEODs2waPsEchMze79Xav0QAo6Y=">ADoniclVLbhMxFJ1peJTwaAtLNhahUotGVSYbEFWlSmSBioBUpQ+pDoPH40lM/BjZnpJi/Ft8CRv+hjtJeISKBZase3x9zrGvr/NKcOu63e/xSuva9Rs3V2+1b9+5e29tfeP+idW1oeyYaqHNWU4sE1yxY8edYGeVYUTmgp3mkxfN/ukFM5Zr9c5dVmwoyUjxklPiIJWtf8O2rjKPZ07esCJgN2aOBIRLQ6hPg1cBODLzfC8N7xXCQo9Qlc1pW9OMb6M9L8ujcl8PWnWr9CStuLZBKSuDElwr8JzTG7SxRZAyVZSh3xkSQ3d7J1jvdne5soKsgXYBOtBiDbCP+igtNa8mUo4JYe52Kzf0xDhOBQtXFtWETohI3YOUBHJ7NDPDg9oEzIFKrWBqRyaZf9UeCJtU0sC0V7KvIluLJvQCGwiGNT2EnyXCZTCzKWjAypxpxOk1Jo4pJG7QWNqDly7jy2dBzVdWOKTq/S1kL5DRq2o0Kbh14hIAoYZDOYiOCfTEwadot3GfQb2GvQb7txUzxGnzxGNi4CmnAeof4aRBv5kDAo5FHz6c5OBx5jFXZcEvpuG8N/RbwW8Hv9lGqJPiXVw0NMs/M/wF73Z6m214ScU+US0lUcVPafD9Xyb/OsdjpY0MHosTZhxEM4vQ6fTvl4FJ72dFPBht7N/sOj5avQwehRtRWn0NqPXkaD6Di8fP4Q8zj63HrYPWYetoTl2JF5oH0dJo4R/3UTW</latexit><latexit sha1_base64="TrGTnKDeIBp3KVMaEj80TuQ5ybg=">ADoniclVLZbhMxFJ1pWErYWnjkxSJUatGoykRIKpKlcgDKgJSlS5SHQaPx5OYeBnZnpJi/Ft8CS/8DXeSsISKByxZ9/j6nGNfX+eV4NZ1u9/jldaVq9eur95o37x1+87dtfV7x1bXhrIjqoU2pzmxTHDFjhx3gp1WhGZC3aST140+yfnzFiu1Tt3UbGhJCPFS06Jg1S29g3buso8njl5w4qA3Zg5EhAuDaE+DV4F4MjM890vFcICz1CVTanbU4zvoV20f+6NCbz9aRZv0JL2opnE5BK4saUCP8mNMfsLFkDZRkKXIR5JAdms7W+t0t7uzgS6DdAE60WIMsvX4Ky40rSVTjgpi7VnardzQE+M4FSy0cW1ZReiEjNgZQEUks0M/OzygDcgUqNQGpnJolv1T4Ym0TS0JRHsh8ya6sWxCI7CJI5NYSfJc5lMLchYMjKkGnM6TUqhiUsavdNa2ICWL+PKZ0PVU7puj8LmUtkNOoaTcquGHUiQsAhBoO5SA6JtATB5+i3cZ9BvUa9hrs31bMEKfNY4+JgaecBqh/hJMG/WYOCDgWfhwkoPHqcdclQU/n4az3tBvBr8V/EYboU6Kd3DR0Cz/zPAXvNPpbThJRX7RLWURBU/pcH3f5n86xyPlTYyeCyOmXEQzSxCp9O/+3oZHPe2U8AHTzp7+4uer0YPofRZpRGT6O96GU0iI4iGj+P8Q8/th61NpvHbQO59SVeKG5Hy2NFv4B+JE1mg=</latexit><latexit sha1_base64="TrGTnKDeIBp3KVMaEj80TuQ5ybg=">ADoniclVLZbhMxFJ1pWErYWnjkxSJUatGoykRIKpKlcgDKgJSlS5SHQaPx5OYeBnZnpJi/Ft8CS/8DXeSsISKByxZ9/j6nGNfX+eV4NZ1u9/jldaVq9eur95o37x1+87dtfV7x1bXhrIjqoU2pzmxTHDFjhx3gp1WhGZC3aST140+yfnzFiu1Tt3UbGhJCPFS06Jg1S29g3buso8njl5w4qA3Zg5EhAuDaE+DV4F4MjM890vFcICz1CVTanbU4zvoV20f+6NCbz9aRZv0JL2opnE5BK4saUCP8mNMfsLFkDZRkKXIR5JAdms7W+t0t7uzgS6DdAE60WIMsvX4Ky40rSVTjgpi7VnardzQE+M4FSy0cW1ZReiEjNgZQEUks0M/OzygDcgUqNQGpnJolv1T4Ym0TS0JRHsh8ya6sWxCI7CJI5NYSfJc5lMLchYMjKkGnM6TUqhiUsavdNa2ICWL+PKZ0PVU7puj8LmUtkNOoaTcquGHUiQsAhBoO5SA6JtATB5+i3cZ9BvUa9hrs31bMEKfNY4+JgaecBqh/hJMG/WYOCDgWfhwkoPHqcdclQU/n4az3tBvBr8V/EYboU6Kd3DR0Cz/zPAXvNPpbThJRX7RLWURBU/pcH3f5n86xyPlTYyeCyOmXEQzSxCp9O/+3oZHPe2U8AHTzp7+4uer0YPofRZpRGT6O96GU0iI4iGj+P8Q8/th61NpvHbQO59SVeKG5Hy2NFv4B+JE1mg=</latexit><latexit sha1_base64="TrGTnKDeIBp3KVMaEj80TuQ5ybg=">ADoniclVLZbhMxFJ1pWErYWnjkxSJUatGoykRIKpKlcgDKgJSlS5SHQaPx5OYeBnZnpJi/Ft8CS/8DXeSsISKByxZ9/j6nGNfX+eV4NZ1u9/jldaVq9eur95o37x1+87dtfV7x1bXhrIjqoU2pzmxTHDFjhx3gp1WhGZC3aST140+yfnzFiu1Tt3UbGhJCPFS06Jg1S29g3buso8njl5w4qA3Zg5EhAuDaE+DV4F4MjM890vFcICz1CVTanbU4zvoV20f+6NCbz9aRZv0JL2opnE5BK4saUCP8mNMfsLFkDZRkKXIR5JAdms7W+t0t7uzgS6DdAE60WIMsvX4Ky40rSVTjgpi7VnardzQE+M4FSy0cW1ZReiEjNgZQEUks0M/OzygDcgUqNQGpnJolv1T4Ym0TS0JRHsh8ya6sWxCI7CJI5NYSfJc5lMLchYMjKkGnM6TUqhiUsavdNa2ICWL+PKZ0PVU7puj8LmUtkNOoaTcquGHUiQsAhBoO5SA6JtATB5+i3cZ9BvUa9hrs31bMEKfNY4+JgaecBqh/hJMG/WYOCDgWfhwkoPHqcdclQU/n4az3tBvBr8V/EYboU6Kd3DR0Cz/zPAXvNPpbThJRX7RLWURBU/pcH3f5n86xyPlTYyeCyOmXEQzSxCp9O/+3oZHPe2U8AHTzp7+4uer0YPofRZpRGT6O96GU0iI4iGj+P8Q8/th61NpvHbQO59SVeKG5Hy2NFv4B+JE1mg=</latexit><latexit sha1_base64="TrGTnKDeIBp3KVMaEj80TuQ5ybg=">ADoniclVLZbhMxFJ1pWErYWnjkxSJUatGoykRIKpKlcgDKgJSlS5SHQaPx5OYeBnZnpJi/Ft8CS/8DXeSsISKByxZ9/j6nGNfX+eV4NZ1u9/jldaVq9eur95o37x1+87dtfV7x1bXhrIjqoU2pzmxTHDFjhx3gp1WhGZC3aST140+yfnzFiu1Tt3UbGhJCPFS06Jg1S29g3buso8njl5w4qA3Zg5EhAuDaE+DV4F4MjM890vFcICz1CVTanbU4zvoV20f+6NCbz9aRZv0JL2opnE5BK4saUCP8mNMfsLFkDZRkKXIR5JAdms7W+t0t7uzgS6DdAE60WIMsvX4Ky40rSVTjgpi7VnardzQE+M4FSy0cW1ZReiEjNgZQEUks0M/OzygDcgUqNQGpnJolv1T4Ym0TS0JRHsh8ya6sWxCI7CJI5NYSfJc5lMLchYMjKkGnM6TUqhiUsavdNa2ICWL+PKZ0PVU7puj8LmUtkNOoaTcquGHUiQsAhBoO5SA6JtATB5+i3cZ9BvUa9hrs31bMEKfNY4+JgaecBqh/hJMG/WYOCDgWfhwkoPHqcdclQU/n4az3tBvBr8V/EYboU6Kd3DR0Cz/zPAXvNPpbThJRX7RLWURBU/pcH3f5n86xyPlTYyeCyOmXEQzSxCp9O/+3oZHPe2U8AHTzp7+4uer0YPofRZpRGT6O96GU0iI4iGj+P8Q8/th61NpvHbQO59SVeKG5Hy2NFv4B+JE1mg=</latexit><latexit sha1_base64="TrGTnKDeIBp3KVMaEj80TuQ5ybg=">ADoniclVLZbhMxFJ1pWErYWnjkxSJUatGoykRIKpKlcgDKgJSlS5SHQaPx5OYeBnZnpJi/Ft8CS/8DXeSsISKByxZ9/j6nGNfX+eV4NZ1u9/jldaVq9eur95o37x1+87dtfV7x1bXhrIjqoU2pzmxTHDFjhx3gp1WhGZC3aST140+yfnzFiu1Tt3UbGhJCPFS06Jg1S29g3buso8njl5w4qA3Zg5EhAuDaE+DV4F4MjM890vFcICz1CVTanbU4zvoV20f+6NCbz9aRZv0JL2opnE5BK4saUCP8mNMfsLFkDZRkKXIR5JAdms7W+t0t7uzgS6DdAE60WIMsvX4Ky40rSVTjgpi7VnardzQE+M4FSy0cW1ZReiEjNgZQEUks0M/OzygDcgUqNQGpnJolv1T4Ym0TS0JRHsh8ya6sWxCI7CJI5NYSfJc5lMLchYMjKkGnM6TUqhiUsavdNa2ICWL+PKZ0PVU7puj8LmUtkNOoaTcquGHUiQsAhBoO5SA6JtATB5+i3cZ9BvUa9hrs31bMEKfNY4+JgaecBqh/hJMG/WYOCDgWfhwkoPHqcdclQU/n4az3tBvBr8V/EYboU6Kd3DR0Cz/zPAXvNPpbThJRX7RLWURBU/pcH3f5n86xyPlTYyeCyOmXEQzSxCp9O/+3oZHPe2U8AHTzp7+4uer0YPofRZpRGT6O96GU0iI4iGj+P8Q8/th61NpvHbQO59SVeKG5Hy2NFv4B+JE1mg=</latexit><latexit sha1_base64="TrGTnKDeIBp3KVMaEj80TuQ5ybg=">ADoniclVLZbhMxFJ1pWErYWnjkxSJUatGoykRIKpKlcgDKgJSlS5SHQaPx5OYeBnZnpJi/Ft8CS/8DXeSsISKByxZ9/j6nGNfX+eV4NZ1u9/jldaVq9eur95o37x1+87dtfV7x1bXhrIjqoU2pzmxTHDFjhx3gp1WhGZC3aST140+yfnzFiu1Tt3UbGhJCPFS06Jg1S29g3buso8njl5w4qA3Zg5EhAuDaE+DV4F4MjM890vFcICz1CVTanbU4zvoV20f+6NCbz9aRZv0JL2opnE5BK4saUCP8mNMfsLFkDZRkKXIR5JAdms7W+t0t7uzgS6DdAE60WIMsvX4Ky40rSVTjgpi7VnardzQE+M4FSy0cW1ZReiEjNgZQEUks0M/OzygDcgUqNQGpnJolv1T4Ym0TS0JRHsh8ya6sWxCI7CJI5NYSfJc5lMLchYMjKkGnM6TUqhiUsavdNa2ICWL+PKZ0PVU7puj8LmUtkNOoaTcquGHUiQsAhBoO5SA6JtATB5+i3cZ9BvUa9hrs31bMEKfNY4+JgaecBqh/hJMG/WYOCDgWfhwkoPHqcdclQU/n4az3tBvBr8V/EYboU6Kd3DR0Cz/zPAXvNPpbThJRX7RLWURBU/pcH3f5n86xyPlTYyeCyOmXEQzSxCp9O/+3oZHPe2U8AHTzp7+4uer0YPofRZpRGT6O96GU0iI4iGj+P8Q8/th61NpvHbQO59SVeKG5Hy2NFv4B+JE1mg=</latexit>
slide-9
SLIDE 9
  • The maximum likelihood estimator (parameters in red):
  • Gradient ascent: initialize with random parameters and iteratively apply
  • Very important to start with a random initialization

Gradient Ascent

CSE 599I: Generative Models University of Washington Lecture 2

θ(i) = θ(i−1) + ηrθ

n

X

i=1

log pθ(xi).

<latexit sha1_base64="bT9JQh7MrDMata2+GMBmTlxaXIo=">ADIHicdVFdaxQxFE3Hrzp+tNVHX4JLYVfHZacICqVQsA8iFXctBslzuZ7GxoPoYkW2eN82f8EeJP8E181Adf9WeY2V2tXhOCf3nMnNycrBbeu1/u6F24eOnyleWr8bXrN26urK7d2rN6YijrUy20OcjAMsEV6zvuBDsoDQOZCbafHT9p6vsnzFiu1Ws3LdlAQqH4iFNwITVcfUXcmDk48m3eqfEWPj0+SEPiPibhSBRkAobzGiZ2Ioeb6X1kcJE6AKXi1K7GvJOd7ja6nV7s8DnSbogLbSI3eHa0geSazqRTDkqwNrDtFe6gQfjOBWsjsnEshLoMRTsMFAFktmBny1f4/WQyfFIm/Aph2fZvxUepJXgxklAO5VZg24sG2gENhHgWBUqSZbJpLJBxpLCQDnmtEpGQoNLGr3TWtgan72MGz0eK7KiWOKzu8ymgjsNG4eG+fcMOrENBCghod1MB2DAeqCJXFMdljY17DnYfyLkhlw2tzBEwhoarD/gVJGnbauQthYr4T7JY8zDjwhKtRzk+q+nBj4Nu179R+Pca4lZJNkjdtlr9l5B3ZbG2sx+ElFXtDtZSg8t/S2u/8GfK/3itJG1J2KPGRfQzDA4nf7r63myt9FNA3/5sLX9bOH5MrqD7qI2StEjtI2eol3URxR9RN/RD/Qzeh9ij5HX+at0dJCcxudiejbLxSNAvA=</latexit><latexit sha1_base64="bT9JQh7MrDMata2+GMBmTlxaXIo=">ADIHicdVFdaxQxFE3Hrzp+tNVHX4JLYVfHZacICqVQsA8iFXctBslzuZ7GxoPoYkW2eN82f8EeJP8E181Adf9WeY2V2tXhOCf3nMnNycrBbeu1/u6F24eOnyleWr8bXrN26urK7d2rN6YijrUy20OcjAMsEV6zvuBDsoDQOZCbafHT9p6vsnzFiu1Ws3LdlAQqH4iFNwITVcfUXcmDk48m3eqfEWPj0+SEPiPibhSBRkAobzGiZ2Ioeb6X1kcJE6AKXi1K7GvJOd7ja6nV7s8DnSbogLbSI3eHa0geSazqRTDkqwNrDtFe6gQfjOBWsjsnEshLoMRTsMFAFktmBny1f4/WQyfFIm/Aph2fZvxUepJXgxklAO5VZg24sG2gENhHgWBUqSZbJpLJBxpLCQDnmtEpGQoNLGr3TWtgan72MGz0eK7KiWOKzu8ymgjsNG4eG+fcMOrENBCghod1MB2DAeqCJXFMdljY17DnYfyLkhlw2tzBEwhoarD/gVJGnbauQthYr4T7JY8zDjwhKtRzk+q+nBj4Nu179R+Pca4lZJNkjdtlr9l5B3ZbG2sx+ElFXtDtZSg8t/S2u/8GfK/3itJG1J2KPGRfQzDA4nf7r63myt9FNA3/5sLX9bOH5MrqD7qI2StEjtI2eol3URxR9RN/RD/Qzeh9ij5HX+at0dJCcxudiejbLxSNAvA=</latexit><latexit sha1_base64="bT9JQh7MrDMata2+GMBmTlxaXIo=">ADIHicdVFdaxQxFE3Hrzp+tNVHX4JLYVfHZacICqVQsA8iFXctBslzuZ7GxoPoYkW2eN82f8EeJP8E181Adf9WeY2V2tXhOCf3nMnNycrBbeu1/u6F24eOnyleWr8bXrN26urK7d2rN6YijrUy20OcjAMsEV6zvuBDsoDQOZCbafHT9p6vsnzFiu1Ws3LdlAQqH4iFNwITVcfUXcmDk48m3eqfEWPj0+SEPiPibhSBRkAobzGiZ2Ioeb6X1kcJE6AKXi1K7GvJOd7ja6nV7s8DnSbogLbSI3eHa0geSazqRTDkqwNrDtFe6gQfjOBWsjsnEshLoMRTsMFAFktmBny1f4/WQyfFIm/Aph2fZvxUepJXgxklAO5VZg24sG2gENhHgWBUqSZbJpLJBxpLCQDnmtEpGQoNLGr3TWtgan72MGz0eK7KiWOKzu8ymgjsNG4eG+fcMOrENBCghod1MB2DAeqCJXFMdljY17DnYfyLkhlw2tzBEwhoarD/gVJGnbauQthYr4T7JY8zDjwhKtRzk+q+nBj4Nu179R+Pca4lZJNkjdtlr9l5B3ZbG2sx+ElFXtDtZSg8t/S2u/8GfK/3itJG1J2KPGRfQzDA4nf7r63myt9FNA3/5sLX9bOH5MrqD7qI2StEjtI2eol3URxR9RN/RD/Qzeh9ij5HX+at0dJCcxudiejbLxSNAvA=</latexit><latexit sha1_base64="bT9JQh7MrDMata2+GMBmTlxaXIo=">ADIHicdVFdaxQxFE3Hrzp+tNVHX4JLYVfHZacICqVQsA8iFXctBslzuZ7GxoPoYkW2eN82f8EeJP8E181Adf9WeY2V2tXhOCf3nMnNycrBbeu1/u6F24eOnyleWr8bXrN26urK7d2rN6YijrUy20OcjAMsEV6zvuBDsoDQOZCbafHT9p6vsnzFiu1Ws3LdlAQqH4iFNwITVcfUXcmDk48m3eqfEWPj0+SEPiPibhSBRkAobzGiZ2Ioeb6X1kcJE6AKXi1K7GvJOd7ja6nV7s8DnSbogLbSI3eHa0geSazqRTDkqwNrDtFe6gQfjOBWsjsnEshLoMRTsMFAFktmBny1f4/WQyfFIm/Aph2fZvxUepJXgxklAO5VZg24sG2gENhHgWBUqSZbJpLJBxpLCQDnmtEpGQoNLGr3TWtgan72MGz0eK7KiWOKzu8ymgjsNG4eG+fcMOrENBCghod1MB2DAeqCJXFMdljY17DnYfyLkhlw2tzBEwhoarD/gVJGnbauQthYr4T7JY8zDjwhKtRzk+q+nBj4Nu179R+Pca4lZJNkjdtlr9l5B3ZbG2sx+ElFXtDtZSg8t/S2u/8GfK/3itJG1J2KPGRfQzDA4nf7r63myt9FNA3/5sLX9bOH5MrqD7qI2StEjtI2eol3URxR9RN/RD/Qzeh9ij5HX+at0dJCcxudiejbLxSNAvA=</latexit>

sup

θ

1 n

n

X

i=1

log pθ(xi) = sup

θ

1 n

n

X

i=1

log

K

X

k=1

πkN(xi; µk, Σk).

<latexit sha1_base64="TrGTnKDeIBp3KVMaEj80TuQ5ybg=">ADoniclVLZbhMxFJ1pWErYWnjkxSJUatGoykRIKpKlcgDKgJSlS5SHQaPx5OYeBnZnpJi/Ft8CS/8DXeSsISKByxZ9/j6nGNfX+eV4NZ1u9/jldaVq9eur95o37x1+87dtfV7x1bXhrIjqoU2pzmxTHDFjhx3gp1WhGZC3aST140+yfnzFiu1Tt3UbGhJCPFS06Jg1S29g3buso8njl5w4qA3Zg5EhAuDaE+DV4F4MjM890vFcICz1CVTanbU4zvoV20f+6NCbz9aRZv0JL2opnE5BK4saUCP8mNMfsLFkDZRkKXIR5JAdms7W+t0t7uzgS6DdAE60WIMsvX4Ky40rSVTjgpi7VnardzQE+M4FSy0cW1ZReiEjNgZQEUks0M/OzygDcgUqNQGpnJolv1T4Ym0TS0JRHsh8ya6sWxCI7CJI5NYSfJc5lMLchYMjKkGnM6TUqhiUsavdNa2ICWL+PKZ0PVU7puj8LmUtkNOoaTcquGHUiQsAhBoO5SA6JtATB5+i3cZ9BvUa9hrs31bMEKfNY4+JgaecBqh/hJMG/WYOCDgWfhwkoPHqcdclQU/n4az3tBvBr8V/EYboU6Kd3DR0Cz/zPAXvNPpbThJRX7RLWURBU/pcH3f5n86xyPlTYyeCyOmXEQzSxCp9O/+3oZHPe2U8AHTzp7+4uer0YPofRZpRGT6O96GU0iI4iGj+P8Q8/th61NpvHbQO59SVeKG5Hy2NFv4B+JE1mg=</latexit><latexit sha1_base64="TrGTnKDeIBp3KVMaEj80TuQ5ybg=">ADoniclVLZbhMxFJ1pWErYWnjkxSJUatGoykRIKpKlcgDKgJSlS5SHQaPx5OYeBnZnpJi/Ft8CS/8DXeSsISKByxZ9/j6nGNfX+eV4NZ1u9/jldaVq9eur95o37x1+87dtfV7x1bXhrIjqoU2pzmxTHDFjhx3gp1WhGZC3aST140+yfnzFiu1Tt3UbGhJCPFS06Jg1S29g3buso8njl5w4qA3Zg5EhAuDaE+DV4F4MjM890vFcICz1CVTanbU4zvoV20f+6NCbz9aRZv0JL2opnE5BK4saUCP8mNMfsLFkDZRkKXIR5JAdms7W+t0t7uzgS6DdAE60WIMsvX4Ky40rSVTjgpi7VnardzQE+M4FSy0cW1ZReiEjNgZQEUks0M/OzygDcgUqNQGpnJolv1T4Ym0TS0JRHsh8ya6sWxCI7CJI5NYSfJc5lMLchYMjKkGnM6TUqhiUsavdNa2ICWL+PKZ0PVU7puj8LmUtkNOoaTcquGHUiQsAhBoO5SA6JtATB5+i3cZ9BvUa9hrs31bMEKfNY4+JgaecBqh/hJMG/WYOCDgWfhwkoPHqcdclQU/n4az3tBvBr8V/EYboU6Kd3DR0Cz/zPAXvNPpbThJRX7RLWURBU/pcH3f5n86xyPlTYyeCyOmXEQzSxCp9O/+3oZHPe2U8AHTzp7+4uer0YPofRZpRGT6O96GU0iI4iGj+P8Q8/th61NpvHbQO59SVeKG5Hy2NFv4B+JE1mg=</latexit><latexit sha1_base64="TrGTnKDeIBp3KVMaEj80TuQ5ybg=">ADoniclVLZbhMxFJ1pWErYWnjkxSJUatGoykRIKpKlcgDKgJSlS5SHQaPx5OYeBnZnpJi/Ft8CS/8DXeSsISKByxZ9/j6nGNfX+eV4NZ1u9/jldaVq9eur95o37x1+87dtfV7x1bXhrIjqoU2pzmxTHDFjhx3gp1WhGZC3aST140+yfnzFiu1Tt3UbGhJCPFS06Jg1S29g3buso8njl5w4qA3Zg5EhAuDaE+DV4F4MjM890vFcICz1CVTanbU4zvoV20f+6NCbz9aRZv0JL2opnE5BK4saUCP8mNMfsLFkDZRkKXIR5JAdms7W+t0t7uzgS6DdAE60WIMsvX4Ky40rSVTjgpi7VnardzQE+M4FSy0cW1ZReiEjNgZQEUks0M/OzygDcgUqNQGpnJolv1T4Ym0TS0JRHsh8ya6sWxCI7CJI5NYSfJc5lMLchYMjKkGnM6TUqhiUsavdNa2ICWL+PKZ0PVU7puj8LmUtkNOoaTcquGHUiQsAhBoO5SA6JtATB5+i3cZ9BvUa9hrs31bMEKfNY4+JgaecBqh/hJMG/WYOCDgWfhwkoPHqcdclQU/n4az3tBvBr8V/EYboU6Kd3DR0Cz/zPAXvNPpbThJRX7RLWURBU/pcH3f5n86xyPlTYyeCyOmXEQzSxCp9O/+3oZHPe2U8AHTzp7+4uer0YPofRZpRGT6O96GU0iI4iGj+P8Q8/th61NpvHbQO59SVeKG5Hy2NFv4B+JE1mg=</latexit><latexit sha1_base64="lm0MUkOzo5hm7fNgs9x04ULkiGs=">ACvnicdVLbhMxFHWGVxkKtGs2I6JIBY2qmWxA6gaJLNhUBImkeJRdcdzJ7Hqx8h2SlJ3foAt/8K/8Dd40vAoFVeyztG9Ptf34bIR3Los+9GL7t1/8PDR3uP4yX789Nnzg/2p1SvDcMK0GZWgkXBFU4cdwJnjUGQpcCz8uJ9Fz+7RGO5Vp/dpsFCwkLxmjNwTU+P+hnx9nWkrsk35E+2dn5Ye87rTRbSVSOCbB2nmeNKzwYx5nANqYriw2wC1jgPFAFEm3ht3W2ySB4qTWJhzlkq3b4UHaSW4ZRrQbmTZoVvKDjqBTQU4XIdIWpYyXdsgw3RhoFlytk5rocGlnd5pLWyb3C7G1W8Lz1WzcqjYTS31SiROJ91ckobZE5sAgFmeGgnYUswFyYXhzTEYZ+DZ6G9B8bNOC0e0pmIWEdRv6X9C0Y39ujiFkrEZhM5KHDNPuaorfrlu58PCH7X+VesHcZL0c3pCq+6a5VdIr+lJfziIwyQVfmFaSlDVL2nrR7+T/O8dT5U2svVUTNG4gGaLYdH5v2u9S6bD4zwTxnZIy/IS3JEcvKGvCMfyJhMCMV+Uq+RaeRja5uPkTU2/2MQ3LoufAi3gmA=</latexit><latexit sha1_base64="Vq6G3z5yj3l1yO6l7zoX6BkJhAY=">ADl3iclVLbhMxEPU2XEotOWVF4tQqUWrKpsXEFUlJPqAQECr0otUh8Xr9SYmvqxsb0kx/i2+hBf+htmkXELFA5asOR7POePxTFL4Xy/z1Z6ly7fuPm8q3u7ZU7d1fX1leOnWks40fMSGNPC+q4FJofeElP60tp6qQ/KSYPG/vT865dcLod/6i5kNFR1pUglEPrnztG3FNnQcyUwqWl5H4Mfc0YlJZykIWg4Qo/IgdrP4XmMizQjX+Txsc5qLbyL/1elFZmfJ+35FV7g1iKfAFVRP2ZUhjexTbOzEKIaCEkXIdipCh4t7bztV5/uz9b+CrILkEPXa79fD35SkrDGsW1Z5I6d5b1az8M1HrBJI9d0jheUzahI34GUFPF3TDMke8AZ4SV8bC1h7PvH8yAlWurSUF6y5U0Vo/Vq1pCS6V1PMp3KRFodKpAxpPR5bWY8GmaSUN9WnL98ZIF/HiY3z1ZBiErhvPNZu/pWok9ga37calsJx5eQGAMiugHMzGFHriYSi6XbLHoV7LX4P825pb6o19FAi18JXTCPWPSNqi35H7FBTLPRg4JUDjNBChq1KcT+PZYBg2Y9iKYaOLcS8jO6Rsw5z4zMkXstMbHThJzX/xIxSVJc/qTHs/RL5V5AtLEqBiKPufVg7cxCp7O/+3oVHA+2M8AHfbSM7qMHaBNl6DF6hl6gfXSEWPI0+ZCI5GPnYedl52A+E0vJ5XDcQwurc/gDjsc0gw=</latexit><latexit sha1_base64="Vq6G3z5yj3l1yO6l7zoX6BkJhAY=">ADl3iclVLbhMxEPU2XEotOWVF4tQqUWrKpsXEFUlJPqAQECr0otUh8Xr9SYmvqxsb0kx/i2+hBf+htmkXELFA5asOR7POePxTFL4Xy/z1Z6ly7fuPm8q3u7ZU7d1fX1leOnWks40fMSGNPC+q4FJofeElP60tp6qQ/KSYPG/vT865dcLod/6i5kNFR1pUglEPrnztG3FNnQcyUwqWl5H4Mfc0YlJZykIWg4Qo/IgdrP4XmMizQjX+Txsc5qLbyL/1elFZmfJ+35FV7g1iKfAFVRP2ZUhjexTbOzEKIaCEkXIdipCh4t7bztV5/uz9b+CrILkEPXa79fD35SkrDGsW1Z5I6d5b1az8M1HrBJI9d0jheUzahI34GUFPF3TDMke8AZ4SV8bC1h7PvH8yAlWurSUF6y5U0Vo/Vq1pCS6V1PMp3KRFodKpAxpPR5bWY8GmaSUN9WnL98ZIF/HiY3z1ZBiErhvPNZu/pWok9ga37calsJx5eQGAMiugHMzGFHriYSi6XbLHoV7LX4P825pb6o19FAi18JXTCPWPSNqi35H7FBTLPRg4JUDjNBChq1KcT+PZYBg2Y9iKYaOLcS8jO6Rsw5z4zMkXstMbHThJzX/xIxSVJc/qTHs/RL5V5AtLEqBiKPufVg7cxCp7O/+3oVHA+2M8AHfbSM7qMHaBNl6DF6hl6gfXSEWPI0+ZCI5GPnYedl52A+E0vJ5XDcQwurc/gDjsc0gw=</latexit><latexit sha1_base64="IEODs2waPsEchMze79Xav0QAo6Y=">ADoniclVLbhMxFJ1peJTwaAtLNhahUotGVSYbEFWlSmSBioBUpQ+pDoPH40lM/BjZnpJi/Ft8CRv+hjtJeISKBZase3x9zrGvr/NKcOu63e/xSuva9Rs3V2+1b9+5e29tfeP+idW1oeyYaqHNWU4sE1yxY8edYGeVYUTmgp3mkxfN/ukFM5Zr9c5dVmwoyUjxklPiIJWtf8O2rjKPZ07esCJgN2aOBIRLQ6hPg1cBODLzfC8N7xXCQo9Qlc1pW9OMb6M9L8ujcl8PWnWr9CStuLZBKSuDElwr8JzTG7SxRZAyVZSh3xkSQ3d7J1jvdne5soKsgXYBOtBiDbCP+igtNa8mUo4JYe52Kzf0xDhOBQtXFtWETohI3YOUBHJ7NDPDg9oEzIFKrWBqRyaZf9UeCJtU0sC0V7KvIluLJvQCGwiGNT2EnyXCZTCzKWjAypxpxOk1Jo4pJG7QWNqDly7jy2dBzVdWOKTq/S1kL5DRq2o0Kbh14hIAoYZDOYiOCfTEwadot3GfQb2GvQb7txUzxGnzxGNi4CmnAeof4aRBv5kDAo5FHz6c5OBx5jFXZcEvpuG8N/RbwW8Hv9lGqJPiXVw0NMs/M/wF73Z6m214ScU+US0lUcVPafD9Xyb/OsdjpY0MHosTZhxEM4vQ6fTvl4FJ72dFPBht7N/sOj5avQwehRtRWn0NqPXkaD6Di8fP4Q8zj63HrYPWYetoTl2JF5oH0dJo4R/3UTW</latexit><latexit sha1_base64="TrGTnKDeIBp3KVMaEj80TuQ5ybg=">ADoniclVLZbhMxFJ1pWErYWnjkxSJUatGoykRIKpKlcgDKgJSlS5SHQaPx5OYeBnZnpJi/Ft8CS/8DXeSsISKByxZ9/j6nGNfX+eV4NZ1u9/jldaVq9eur95o37x1+87dtfV7x1bXhrIjqoU2pzmxTHDFjhx3gp1WhGZC3aST140+yfnzFiu1Tt3UbGhJCPFS06Jg1S29g3buso8njl5w4qA3Zg5EhAuDaE+DV4F4MjM890vFcICz1CVTanbU4zvoV20f+6NCbz9aRZv0JL2opnE5BK4saUCP8mNMfsLFkDZRkKXIR5JAdms7W+t0t7uzgS6DdAE60WIMsvX4Ky40rSVTjgpi7VnardzQE+M4FSy0cW1ZReiEjNgZQEUks0M/OzygDcgUqNQGpnJolv1T4Ym0TS0JRHsh8ya6sWxCI7CJI5NYSfJc5lMLchYMjKkGnM6TUqhiUsavdNa2ICWL+PKZ0PVU7puj8LmUtkNOoaTcquGHUiQsAhBoO5SA6JtATB5+i3cZ9BvUa9hrs31bMEKfNY4+JgaecBqh/hJMG/WYOCDgWfhwkoPHqcdclQU/n4az3tBvBr8V/EYboU6Kd3DR0Cz/zPAXvNPpbThJRX7RLWURBU/pcH3f5n86xyPlTYyeCyOmXEQzSxCp9O/+3oZHPe2U8AHTzp7+4uer0YPofRZpRGT6O96GU0iI4iGj+P8Q8/th61NpvHbQO59SVeKG5Hy2NFv4B+JE1mg=</latexit><latexit sha1_base64="TrGTnKDeIBp3KVMaEj80TuQ5ybg=">ADoniclVLZbhMxFJ1pWErYWnjkxSJUatGoykRIKpKlcgDKgJSlS5SHQaPx5OYeBnZnpJi/Ft8CS/8DXeSsISKByxZ9/j6nGNfX+eV4NZ1u9/jldaVq9eur95o37x1+87dtfV7x1bXhrIjqoU2pzmxTHDFjhx3gp1WhGZC3aST140+yfnzFiu1Tt3UbGhJCPFS06Jg1S29g3buso8njl5w4qA3Zg5EhAuDaE+DV4F4MjM890vFcICz1CVTanbU4zvoV20f+6NCbz9aRZv0JL2opnE5BK4saUCP8mNMfsLFkDZRkKXIR5JAdms7W+t0t7uzgS6DdAE60WIMsvX4Ky40rSVTjgpi7VnardzQE+M4FSy0cW1ZReiEjNgZQEUks0M/OzygDcgUqNQGpnJolv1T4Ym0TS0JRHsh8ya6sWxCI7CJI5NYSfJc5lMLchYMjKkGnM6TUqhiUsavdNa2ICWL+PKZ0PVU7puj8LmUtkNOoaTcquGHUiQsAhBoO5SA6JtATB5+i3cZ9BvUa9hrs31bMEKfNY4+JgaecBqh/hJMG/WYOCDgWfhwkoPHqcdclQU/n4az3tBvBr8V/EYboU6Kd3DR0Cz/zPAXvNPpbThJRX7RLWURBU/pcH3f5n86xyPlTYyeCyOmXEQzSxCp9O/+3oZHPe2U8AHTzp7+4uer0YPofRZpRGT6O96GU0iI4iGj+P8Q8/th61NpvHbQO59SVeKG5Hy2NFv4B+JE1mg=</latexit><latexit sha1_base64="TrGTnKDeIBp3KVMaEj80TuQ5ybg=">ADoniclVLZbhMxFJ1pWErYWnjkxSJUatGoykRIKpKlcgDKgJSlS5SHQaPx5OYeBnZnpJi/Ft8CS/8DXeSsISKByxZ9/j6nGNfX+eV4NZ1u9/jldaVq9eur95o37x1+87dtfV7x1bXhrIjqoU2pzmxTHDFjhx3gp1WhGZC3aST140+yfnzFiu1Tt3UbGhJCPFS06Jg1S29g3buso8njl5w4qA3Zg5EhAuDaE+DV4F4MjM890vFcICz1CVTanbU4zvoV20f+6NCbz9aRZv0JL2opnE5BK4saUCP8mNMfsLFkDZRkKXIR5JAdms7W+t0t7uzgS6DdAE60WIMsvX4Ky40rSVTjgpi7VnardzQE+M4FSy0cW1ZReiEjNgZQEUks0M/OzygDcgUqNQGpnJolv1T4Ym0TS0JRHsh8ya6sWxCI7CJI5NYSfJc5lMLchYMjKkGnM6TUqhiUsavdNa2ICWL+PKZ0PVU7puj8LmUtkNOoaTcquGHUiQsAhBoO5SA6JtATB5+i3cZ9BvUa9hrs31bMEKfNY4+JgaecBqh/hJMG/WYOCDgWfhwkoPHqcdclQU/n4az3tBvBr8V/EYboU6Kd3DR0Cz/zPAXvNPpbThJRX7RLWURBU/pcH3f5n86xyPlTYyeCyOmXEQzSxCp9O/+3oZHPe2U8AHTzp7+4uer0YPofRZpRGT6O96GU0iI4iGj+P8Q8/th61NpvHbQO59SVeKG5Hy2NFv4B+JE1mg=</latexit><latexit sha1_base64="TrGTnKDeIBp3KVMaEj80TuQ5ybg=">ADoniclVLZbhMxFJ1pWErYWnjkxSJUatGoykRIKpKlcgDKgJSlS5SHQaPx5OYeBnZnpJi/Ft8CS/8DXeSsISKByxZ9/j6nGNfX+eV4NZ1u9/jldaVq9eur95o37x1+87dtfV7x1bXhrIjqoU2pzmxTHDFjhx3gp1WhGZC3aST140+yfnzFiu1Tt3UbGhJCPFS06Jg1S29g3buso8njl5w4qA3Zg5EhAuDaE+DV4F4MjM890vFcICz1CVTanbU4zvoV20f+6NCbz9aRZv0JL2opnE5BK4saUCP8mNMfsLFkDZRkKXIR5JAdms7W+t0t7uzgS6DdAE60WIMsvX4Ky40rSVTjgpi7VnardzQE+M4FSy0cW1ZReiEjNgZQEUks0M/OzygDcgUqNQGpnJolv1T4Ym0TS0JRHsh8ya6sWxCI7CJI5NYSfJc5lMLchYMjKkGnM6TUqhiUsavdNa2ICWL+PKZ0PVU7puj8LmUtkNOoaTcquGHUiQsAhBoO5SA6JtATB5+i3cZ9BvUa9hrs31bMEKfNY4+JgaecBqh/hJMG/WYOCDgWfhwkoPHqcdclQU/n4az3tBvBr8V/EYboU6Kd3DR0Cz/zPAXvNPpbThJRX7RLWURBU/pcH3f5n86xyPlTYyeCyOmXEQzSxCp9O/+3oZHPe2U8AHTzp7+4uer0YPofRZpRGT6O96GU0iI4iGj+P8Q8/th61NpvHbQO59SVeKG5Hy2NFv4B+JE1mg=</latexit><latexit sha1_base64="TrGTnKDeIBp3KVMaEj80TuQ5ybg=">ADoniclVLZbhMxFJ1pWErYWnjkxSJUatGoykRIKpKlcgDKgJSlS5SHQaPx5OYeBnZnpJi/Ft8CS/8DXeSsISKByxZ9/j6nGNfX+eV4NZ1u9/jldaVq9eur95o37x1+87dtfV7x1bXhrIjqoU2pzmxTHDFjhx3gp1WhGZC3aST140+yfnzFiu1Tt3UbGhJCPFS06Jg1S29g3buso8njl5w4qA3Zg5EhAuDaE+DV4F4MjM890vFcICz1CVTanbU4zvoV20f+6NCbz9aRZv0JL2opnE5BK4saUCP8mNMfsLFkDZRkKXIR5JAdms7W+t0t7uzgS6DdAE60WIMsvX4Ky40rSVTjgpi7VnardzQE+M4FSy0cW1ZReiEjNgZQEUks0M/OzygDcgUqNQGpnJolv1T4Ym0TS0JRHsh8ya6sWxCI7CJI5NYSfJc5lMLchYMjKkGnM6TUqhiUsavdNa2ICWL+PKZ0PVU7puj8LmUtkNOoaTcquGHUiQsAhBoO5SA6JtATB5+i3cZ9BvUa9hrs31bMEKfNY4+JgaecBqh/hJMG/WYOCDgWfhwkoPHqcdclQU/n4az3tBvBr8V/EYboU6Kd3DR0Cz/zPAXvNPpbThJRX7RLWURBU/pcH3f5n86xyPlTYyeCyOmXEQzSxCp9O/+3oZHPe2U8AHTzp7+4uer0YPofRZpRGT6O96GU0iI4iGj+P8Q8/th61NpvHbQO59SVeKG5Hy2NFv4B+JE1mg=</latexit><latexit sha1_base64="TrGTnKDeIBp3KVMaEj80TuQ5ybg=">ADoniclVLZbhMxFJ1pWErYWnjkxSJUatGoykRIKpKlcgDKgJSlS5SHQaPx5OYeBnZnpJi/Ft8CS/8DXeSsISKByxZ9/j6nGNfX+eV4NZ1u9/jldaVq9eur95o37x1+87dtfV7x1bXhrIjqoU2pzmxTHDFjhx3gp1WhGZC3aST140+yfnzFiu1Tt3UbGhJCPFS06Jg1S29g3buso8njl5w4qA3Zg5EhAuDaE+DV4F4MjM890vFcICz1CVTanbU4zvoV20f+6NCbz9aRZv0JL2opnE5BK4saUCP8mNMfsLFkDZRkKXIR5JAdms7W+t0t7uzgS6DdAE60WIMsvX4Ky40rSVTjgpi7VnardzQE+M4FSy0cW1ZReiEjNgZQEUks0M/OzygDcgUqNQGpnJolv1T4Ym0TS0JRHsh8ya6sWxCI7CJI5NYSfJc5lMLchYMjKkGnM6TUqhiUsavdNa2ICWL+PKZ0PVU7puj8LmUtkNOoaTcquGHUiQsAhBoO5SA6JtATB5+i3cZ9BvUa9hrs31bMEKfNY4+JgaecBqh/hJMG/WYOCDgWfhwkoPHqcdclQU/n4az3tBvBr8V/EYboU6Kd3DR0Cz/zPAXvNPpbThJRX7RLWURBU/pcH3f5n86xyPlTYyeCyOmXEQzSxCp9O/+3oZHPe2U8AHTzp7+4uer0YPofRZpRGT6O96GU0iI4iGj+P8Q8/th61NpvHbQO59SVeKG5Hy2NFv4B+JE1mg=</latexit>
slide-10
SLIDE 10
  • Gradient ascent:
  • Stochastic gradient ascent (SGD):
  • Gradient ascent update: O(n)
  • SGD update: O(1)

Stochastic Gradient Ascent

CSE 599I: Generative Models University of Washington Lecture 2

θ(i) = θ(i−1) + ηrθ log pθ(xi (mod n)).

<latexit sha1_base64="wWOY5uSo+OjKOR3MQhTUlCa49k=">ACQHicbVBNaxNBGJ6tVtvUj1iPXgZDIaEYdotgL0KhF/EUwSFbFzenbxJhs7OLDPvloRlf1ov/QnePHvxoIjXnpwkC9qPFwaej/dhZp40V9JRGH4Lth483H70eGe3sfk6bPnzRf7A2cK7AvjDL2LAWHSmrskySFZ7lFyFKFw/T8dOUPL9A6afRnWuY4zmCm5VQKIC8lzWFMcyT4UrZlp+Lv+T/6JvLCIY89jTWkCpKNx2NlZjyvWXuRlNKncElb2dmwnWnqjrdpNkKu+F6+F0Q1aDF6uklza/xIgiQ01CgXOjKMxpXIlKRWjbhwmIM4hxmOPNSQoRuX6wIqfuCVCZ8a648mvlb/T5SQObfMUr+ZAc3dbW8l3ueNCpoej0up84JQi81F0JxMnzVJp9Ii4LU0gMQVvq3cjEHC4J85w1fQnT7y3fB4KgbefzpbevkY13HDnvFXrM2i9g7dsI+sB7rM8Eu2Xf2k/0KroIfwe/gz2Z1K6gzL9mNCa7/Ak1WrnI=</latexit><latexit sha1_base64="wWOY5uSo+OjKOR3MQhTUlCa49k=">ACQHicbVBNaxNBGJ6tVtvUj1iPXgZDIaEYdotgL0KhF/EUwSFbFzenbxJhs7OLDPvloRlf1ov/QnePHvxoIjXnpwkC9qPFwaej/dhZp40V9JRGH4Lth483H70eGe3sfk6bPnzRf7A2cK7AvjDL2LAWHSmrskySFZ7lFyFKFw/T8dOUPL9A6afRnWuY4zmCm5VQKIC8lzWFMcyT4UrZlp+Lv+T/6JvLCIY89jTWkCpKNx2NlZjyvWXuRlNKncElb2dmwnWnqjrdpNkKu+F6+F0Q1aDF6uklza/xIgiQ01CgXOjKMxpXIlKRWjbhwmIM4hxmOPNSQoRuX6wIqfuCVCZ8a648mvlb/T5SQObfMUr+ZAc3dbW8l3ueNCpoej0up84JQi81F0JxMnzVJp9Ii4LU0gMQVvq3cjEHC4J85w1fQnT7y3fB4KgbefzpbevkY13HDnvFXrM2i9g7dsI+sB7rM8Eu2Xf2k/0KroIfwe/gz2Z1K6gzL9mNCa7/Ak1WrnI=</latexit><latexit sha1_base64="wWOY5uSo+OjKOR3MQhTUlCa49k=">ACQHicbVBNaxNBGJ6tVtvUj1iPXgZDIaEYdotgL0KhF/EUwSFbFzenbxJhs7OLDPvloRlf1ov/QnePHvxoIjXnpwkC9qPFwaej/dhZp40V9JRGH4Lth483H70eGe3sfk6bPnzRf7A2cK7AvjDL2LAWHSmrskySFZ7lFyFKFw/T8dOUPL9A6afRnWuY4zmCm5VQKIC8lzWFMcyT4UrZlp+Lv+T/6JvLCIY89jTWkCpKNx2NlZjyvWXuRlNKncElb2dmwnWnqjrdpNkKu+F6+F0Q1aDF6uklza/xIgiQ01CgXOjKMxpXIlKRWjbhwmIM4hxmOPNSQoRuX6wIqfuCVCZ8a648mvlb/T5SQObfMUr+ZAc3dbW8l3ueNCpoej0up84JQi81F0JxMnzVJp9Ii4LU0gMQVvq3cjEHC4J85w1fQnT7y3fB4KgbefzpbevkY13HDnvFXrM2i9g7dsI+sB7rM8Eu2Xf2k/0KroIfwe/gz2Z1K6gzL9mNCa7/Ak1WrnI=</latexit><latexit sha1_base64="wWOY5uSo+OjKOR3MQhTUlCa49k=">ACQHicbVBNaxNBGJ6tVtvUj1iPXgZDIaEYdotgL0KhF/EUwSFbFzenbxJhs7OLDPvloRlf1ov/QnePHvxoIjXnpwkC9qPFwaej/dhZp40V9JRGH4Lth483H70eGe3sfk6bPnzRf7A2cK7AvjDL2LAWHSmrskySFZ7lFyFKFw/T8dOUPL9A6afRnWuY4zmCm5VQKIC8lzWFMcyT4UrZlp+Lv+T/6JvLCIY89jTWkCpKNx2NlZjyvWXuRlNKncElb2dmwnWnqjrdpNkKu+F6+F0Q1aDF6uklza/xIgiQ01CgXOjKMxpXIlKRWjbhwmIM4hxmOPNSQoRuX6wIqfuCVCZ8a648mvlb/T5SQObfMUr+ZAc3dbW8l3ueNCpoej0up84JQi81F0JxMnzVJp9Ii4LU0gMQVvq3cjEHC4J85w1fQnT7y3fB4KgbefzpbevkY13HDnvFXrM2i9g7dsI+sB7rM8Eu2Xf2k/0KroIfwe/gz2Z1K6gzL9mNCa7/Ak1WrnI=</latexit>

θ(i) = θ(i−1) + ηrθ

n

X

i=1

log pθ(xi).

<latexit sha1_base64="bT9JQh7MrDMata2+GMBmTlxaXIo=">ADIHicdVFdaxQxFE3Hrzp+tNVHX4JLYVfHZacICqVQsA8iFXctBslzuZ7GxoPoYkW2eN82f8EeJP8E181Adf9WeY2V2tXhOCf3nMnNycrBbeu1/u6F24eOnyleWr8bXrN26urK7d2rN6YijrUy20OcjAMsEV6zvuBDsoDQOZCbafHT9p6vsnzFiu1Ws3LdlAQqH4iFNwITVcfUXcmDk48m3eqfEWPj0+SEPiPibhSBRkAobzGiZ2Ioeb6X1kcJE6AKXi1K7GvJOd7ja6nV7s8DnSbogLbSI3eHa0geSazqRTDkqwNrDtFe6gQfjOBWsjsnEshLoMRTsMFAFktmBny1f4/WQyfFIm/Aph2fZvxUepJXgxklAO5VZg24sG2gENhHgWBUqSZbJpLJBxpLCQDnmtEpGQoNLGr3TWtgan72MGz0eK7KiWOKzu8ymgjsNG4eG+fcMOrENBCghod1MB2DAeqCJXFMdljY17DnYfyLkhlw2tzBEwhoarD/gVJGnbauQthYr4T7JY8zDjwhKtRzk+q+nBj4Nu179R+Pca4lZJNkjdtlr9l5B3ZbG2sx+ElFXtDtZSg8t/S2u/8GfK/3itJG1J2KPGRfQzDA4nf7r63myt9FNA3/5sLX9bOH5MrqD7qI2StEjtI2eol3URxR9RN/RD/Qzeh9ij5HX+at0dJCcxudiejbLxSNAvA=</latexit><latexit sha1_base64="bT9JQh7MrDMata2+GMBmTlxaXIo=">ADIHicdVFdaxQxFE3Hrzp+tNVHX4JLYVfHZacICqVQsA8iFXctBslzuZ7GxoPoYkW2eN82f8EeJP8E181Adf9WeY2V2tXhOCf3nMnNycrBbeu1/u6F24eOnyleWr8bXrN26urK7d2rN6YijrUy20OcjAMsEV6zvuBDsoDQOZCbafHT9p6vsnzFiu1Ws3LdlAQqH4iFNwITVcfUXcmDk48m3eqfEWPj0+SEPiPibhSBRkAobzGiZ2Ioeb6X1kcJE6AKXi1K7GvJOd7ja6nV7s8DnSbogLbSI3eHa0geSazqRTDkqwNrDtFe6gQfjOBWsjsnEshLoMRTsMFAFktmBny1f4/WQyfFIm/Aph2fZvxUepJXgxklAO5VZg24sG2gENhHgWBUqSZbJpLJBxpLCQDnmtEpGQoNLGr3TWtgan72MGz0eK7KiWOKzu8ymgjsNG4eG+fcMOrENBCghod1MB2DAeqCJXFMdljY17DnYfyLkhlw2tzBEwhoarD/gVJGnbauQthYr4T7JY8zDjwhKtRzk+q+nBj4Nu179R+Pca4lZJNkjdtlr9l5B3ZbG2sx+ElFXtDtZSg8t/S2u/8GfK/3itJG1J2KPGRfQzDA4nf7r63myt9FNA3/5sLX9bOH5MrqD7qI2StEjtI2eol3URxR9RN/RD/Qzeh9ij5HX+at0dJCcxudiejbLxSNAvA=</latexit><latexit sha1_base64="bT9JQh7MrDMata2+GMBmTlxaXIo=">ADIHicdVFdaxQxFE3Hrzp+tNVHX4JLYVfHZacICqVQsA8iFXctBslzuZ7GxoPoYkW2eN82f8EeJP8E181Adf9WeY2V2tXhOCf3nMnNycrBbeu1/u6F24eOnyleWr8bXrN26urK7d2rN6YijrUy20OcjAMsEV6zvuBDsoDQOZCbafHT9p6vsnzFiu1Ws3LdlAQqH4iFNwITVcfUXcmDk48m3eqfEWPj0+SEPiPibhSBRkAobzGiZ2Ioeb6X1kcJE6AKXi1K7GvJOd7ja6nV7s8DnSbogLbSI3eHa0geSazqRTDkqwNrDtFe6gQfjOBWsjsnEshLoMRTsMFAFktmBny1f4/WQyfFIm/Aph2fZvxUepJXgxklAO5VZg24sG2gENhHgWBUqSZbJpLJBxpLCQDnmtEpGQoNLGr3TWtgan72MGz0eK7KiWOKzu8ymgjsNG4eG+fcMOrENBCghod1MB2DAeqCJXFMdljY17DnYfyLkhlw2tzBEwhoarD/gVJGnbauQthYr4T7JY8zDjwhKtRzk+q+nBj4Nu179R+Pca4lZJNkjdtlr9l5B3ZbG2sx+ElFXtDtZSg8t/S2u/8GfK/3itJG1J2KPGRfQzDA4nf7r63myt9FNA3/5sLX9bOH5MrqD7qI2StEjtI2eol3URxR9RN/RD/Qzeh9ij5HX+at0dJCcxudiejbLxSNAvA=</latexit><latexit sha1_base64="bT9JQh7MrDMata2+GMBmTlxaXIo=">ADIHicdVFdaxQxFE3Hrzp+tNVHX4JLYVfHZacICqVQsA8iFXctBslzuZ7GxoPoYkW2eN82f8EeJP8E181Adf9WeY2V2tXhOCf3nMnNycrBbeu1/u6F24eOnyleWr8bXrN26urK7d2rN6YijrUy20OcjAMsEV6zvuBDsoDQOZCbafHT9p6vsnzFiu1Ws3LdlAQqH4iFNwITVcfUXcmDk48m3eqfEWPj0+SEPiPibhSBRkAobzGiZ2Ioeb6X1kcJE6AKXi1K7GvJOd7ja6nV7s8DnSbogLbSI3eHa0geSazqRTDkqwNrDtFe6gQfjOBWsjsnEshLoMRTsMFAFktmBny1f4/WQyfFIm/Aph2fZvxUepJXgxklAO5VZg24sG2gENhHgWBUqSZbJpLJBxpLCQDnmtEpGQoNLGr3TWtgan72MGz0eK7KiWOKzu8ymgjsNG4eG+fcMOrENBCghod1MB2DAeqCJXFMdljY17DnYfyLkhlw2tzBEwhoarD/gVJGnbauQthYr4T7JY8zDjwhKtRzk+q+nBj4Nu179R+Pca4lZJNkjdtlr9l5B3ZbG2sx+ElFXtDtZSg8t/S2u/8GfK/3itJG1J2KPGRfQzDA4nf7r63myt9FNA3/5sLX9bOH5MrqD7qI2StEjtI2eol3URxR9RN/RD/Qzeh9ij5HX+at0dJCcxudiejbLxSNAvA=</latexit>

E

x∼p log pθ(x) ≈ log pθ(xi).

<latexit sha1_base64="R/7aLgpinF9UPLjLEvqSuqA4pWU=">ADGXicdVFdixMxFE3Hr3X86uqjL8FSaGVYOkVQ2JcFK4giVrDdQlOGTCYzDZtMQpKuU+P8En+FL7Jr76IL7pPzHTrR+7ixfCOdx7z01uTqo4M3Yw+N4KLly8dPnKztXw2vUbN2+1d29PjVxpQidEcqlnKTaUs5JOLOczpSmWKScHqZHj5v64THVhsnytV0ruhC4KFnOCLY+lbSfI4HtUiq3wTR1T+o6cRVEhgmoaoi4LKBKkF1Si3tVHyKslJbV2ULC+ntJuzPYG2wCnifxlnTANsbJbusjyiRZCVpawrEx83ig7MJhbRnhtA7RylCFyREu6NzTEgtqFm6zdQ27PpPBXGp/Sgs32X8VDgvTLBV5NGuRNmiXoFGYCKOLa18JUpTEVXGy2hUaKyWjFRziW2UaO3UnJTw9OPsfmjhWOlWlakpO35CsOrYTNL8OMaUosX3uCiWZ+HUiWGNivRdhiEbU76vpCz/+paIaW6nvO4R1IXBV+/0LFDXsb+cY+4nZyPsmJ8xc4iVecaOq3o+XLhe7fq164YQdmK0j7KmzbC3FL1D+51hN/Q/WdI3RAqBy+y3tHajP0P+d49DpdSidohPqbYe9Qa90/FZX8+T6XAv9vzVg87Bs63nO+AuAd6IAYPwQF4CsZgAgj4AL6BH+Bn8D74FHwOvpy0Bq2t5g4FcHX7sQAdo=</latexit><latexit sha1_base64="R/7aLgpinF9UPLjLEvqSuqA4pWU=">ADGXicdVFdixMxFE3Hr3X86uqjL8FSaGVYOkVQ2JcFK4giVrDdQlOGTCYzDZtMQpKuU+P8En+FL7Jr76IL7pPzHTrR+7ixfCOdx7z01uTqo4M3Yw+N4KLly8dPnKztXw2vUbN2+1d29PjVxpQidEcqlnKTaUs5JOLOczpSmWKScHqZHj5v64THVhsnytV0ruhC4KFnOCLY+lbSfI4HtUiq3wTR1T+o6cRVEhgmoaoi4LKBKkF1Si3tVHyKslJbV2ULC+ntJuzPYG2wCnifxlnTANsbJbusjyiRZCVpawrEx83ig7MJhbRnhtA7RylCFyREu6NzTEgtqFm6zdQ27PpPBXGp/Sgs32X8VDgvTLBV5NGuRNmiXoFGYCKOLa18JUpTEVXGy2hUaKyWjFRziW2UaO3UnJTw9OPsfmjhWOlWlakpO35CsOrYTNL8OMaUosX3uCiWZ+HUiWGNivRdhiEbU76vpCz/+paIaW6nvO4R1IXBV+/0LFDXsb+cY+4nZyPsmJ8xc4iVecaOq3o+XLhe7fq164YQdmK0j7KmzbC3FL1D+51hN/Q/WdI3RAqBy+y3tHajP0P+d49DpdSidohPqbYe9Qa90/FZX8+T6XAv9vzVg87Bs63nO+AuAd6IAYPwQF4CsZgAgj4AL6BH+Bn8D74FHwOvpy0Bq2t5g4FcHX7sQAdo=</latexit><latexit sha1_base64="R/7aLgpinF9UPLjLEvqSuqA4pWU=">ADGXicdVFdixMxFE3Hr3X86uqjL8FSaGVYOkVQ2JcFK4giVrDdQlOGTCYzDZtMQpKuU+P8En+FL7Jr76IL7pPzHTrR+7ixfCOdx7z01uTqo4M3Yw+N4KLly8dPnKztXw2vUbN2+1d29PjVxpQidEcqlnKTaUs5JOLOczpSmWKScHqZHj5v64THVhsnytV0ruhC4KFnOCLY+lbSfI4HtUiq3wTR1T+o6cRVEhgmoaoi4LKBKkF1Si3tVHyKslJbV2ULC+ntJuzPYG2wCnifxlnTANsbJbusjyiRZCVpawrEx83ig7MJhbRnhtA7RylCFyREu6NzTEgtqFm6zdQ27PpPBXGp/Sgs32X8VDgvTLBV5NGuRNmiXoFGYCKOLa18JUpTEVXGy2hUaKyWjFRziW2UaO3UnJTw9OPsfmjhWOlWlakpO35CsOrYTNL8OMaUosX3uCiWZ+HUiWGNivRdhiEbU76vpCz/+paIaW6nvO4R1IXBV+/0LFDXsb+cY+4nZyPsmJ8xc4iVecaOq3o+XLhe7fq164YQdmK0j7KmzbC3FL1D+51hN/Q/WdI3RAqBy+y3tHajP0P+d49DpdSidohPqbYe9Qa90/FZX8+T6XAv9vzVg87Bs63nO+AuAd6IAYPwQF4CsZgAgj4AL6BH+Bn8D74FHwOvpy0Bq2t5g4FcHX7sQAdo=</latexit><latexit sha1_base64="R/7aLgpinF9UPLjLEvqSuqA4pWU=">ADGXicdVFdixMxFE3Hr3X86uqjL8FSaGVYOkVQ2JcFK4giVrDdQlOGTCYzDZtMQpKuU+P8En+FL7Jr76IL7pPzHTrR+7ixfCOdx7z01uTqo4M3Yw+N4KLly8dPnKztXw2vUbN2+1d29PjVxpQidEcqlnKTaUs5JOLOczpSmWKScHqZHj5v64THVhsnytV0ruhC4KFnOCLY+lbSfI4HtUiq3wTR1T+o6cRVEhgmoaoi4LKBKkF1Si3tVHyKslJbV2ULC+ntJuzPYG2wCnifxlnTANsbJbusjyiRZCVpawrEx83ig7MJhbRnhtA7RylCFyREu6NzTEgtqFm6zdQ27PpPBXGp/Sgs32X8VDgvTLBV5NGuRNmiXoFGYCKOLa18JUpTEVXGy2hUaKyWjFRziW2UaO3UnJTw9OPsfmjhWOlWlakpO35CsOrYTNL8OMaUosX3uCiWZ+HUiWGNivRdhiEbU76vpCz/+paIaW6nvO4R1IXBV+/0LFDXsb+cY+4nZyPsmJ8xc4iVecaOq3o+XLhe7fq164YQdmK0j7KmzbC3FL1D+51hN/Q/WdI3RAqBy+y3tHajP0P+d49DpdSidohPqbYe9Qa90/FZX8+T6XAv9vzVg87Bs63nO+AuAd6IAYPwQF4CsZgAgj4AL6BH+Bn8D74FHwOvpy0Bq2t5g4FcHX7sQAdo=</latexit>

E

x∼p log pθ(x) ≈ 1

n

n

X

i=1

log pθ(xi).

<latexit sha1_base64="a95aEXi6A6sxpts9eEGxlWxWsU=">ADMXicdVFbaxQxFM6Ot7pe2uqjL8Gl0JVh2SmCQhEKVhBXMG2C5t1yGQys6G5kWTr1JhfJfgf/Al9E198FXfzGzXS1s8EL6Pk/Od5Jyv0JxZNxyedJLl69cvbZyvXvj5q3bq2vrd/atmhtC94jiyowLbClnku45jgda0OxKDg9KA6ftvcHR9RYpuQbd6zpVOBasoR7GIqXyNIYDdT2i+wKPyzEHLfQGSZgDpAxFUNdY7cjDq82fQhwlobFQsqg4nPgpcB2bnIPXuShbfyvCBn/UG+1hsOhouAF0m2JD2wjFG+3vmESkXmgkpHOLZ2kg21m3psHCOchi6aW6oxOcQ1nUQqsaB26hfbCHAjZkpYKROPdHCR/VfhsbDtsGlEeyKFt1MtNAKbMqxo028SYtCpI2NMprWBusZI01acYVd2uqdUtwGePYzrno89UzquaOSnP6lmnPoFGy3D0tmKH8OBJMDIvjQDLDcZMuetTtol0a5zX0ZWz/SlODnTIPMKmFrgJcf4apS37WznCsWO5G/0XLPYe8RkVbKjJky2pn4z+H7wG10IexnaRmVbZtl7ij6g7d7WRjduUtJ3RAmBZflbGvzunyb/e8cjqYwIHvF9alxEs8DodHbe14tkf2uQRf76YW/nxdLzFXAP3AebIAOPwA54DkZgDxDwGXwHP8DP5GNyknxJvp6WJp2l5i4E8m3X+10DBw=</latexit><latexit sha1_base64="a95aEXi6A6sxpts9eEGxlWxWsU=">ADMXicdVFbaxQxFM6Ot7pe2uqjL8Gl0JVh2SmCQhEKVhBXMG2C5t1yGQys6G5kWTr1JhfJfgf/Al9E198FXfzGzXS1s8EL6Pk/Od5Jyv0JxZNxyedJLl69cvbZyvXvj5q3bq2vrd/atmhtC94jiyowLbClnku45jgda0OxKDg9KA6ftvcHR9RYpuQbd6zpVOBasoR7GIqXyNIYDdT2i+wKPyzEHLfQGSZgDpAxFUNdY7cjDq82fQhwlobFQsqg4nPgpcB2bnIPXuShbfyvCBn/UG+1hsOhouAF0m2JD2wjFG+3vmESkXmgkpHOLZ2kg21m3psHCOchi6aW6oxOcQ1nUQqsaB26hfbCHAjZkpYKROPdHCR/VfhsbDtsGlEeyKFt1MtNAKbMqxo028SYtCpI2NMprWBusZI01acYVd2uqdUtwGePYzrno89UzquaOSnP6lmnPoFGy3D0tmKH8OBJMDIvjQDLDcZMuetTtol0a5zX0ZWz/SlODnTIPMKmFrgJcf4apS37WznCsWO5G/0XLPYe8RkVbKjJky2pn4z+H7wG10IexnaRmVbZtl7ij6g7d7WRjduUtJ3RAmBZflbGvzunyb/e8cjqYwIHvF9alxEs8DodHbe14tkf2uQRf76YW/nxdLzFXAP3AebIAOPwA54DkZgDxDwGXwHP8DP5GNyknxJvp6WJp2l5i4E8m3X+10DBw=</latexit><latexit sha1_base64="a95aEXi6A6sxpts9eEGxlWxWsU=">ADMXicdVFbaxQxFM6Ot7pe2uqjL8Gl0JVh2SmCQhEKVhBXMG2C5t1yGQys6G5kWTr1JhfJfgf/Al9E198FXfzGzXS1s8EL6Pk/Od5Jyv0JxZNxyedJLl69cvbZyvXvj5q3bq2vrd/atmhtC94jiyowLbClnku45jgda0OxKDg9KA6ftvcHR9RYpuQbd6zpVOBasoR7GIqXyNIYDdT2i+wKPyzEHLfQGSZgDpAxFUNdY7cjDq82fQhwlobFQsqg4nPgpcB2bnIPXuShbfyvCBn/UG+1hsOhouAF0m2JD2wjFG+3vmESkXmgkpHOLZ2kg21m3psHCOchi6aW6oxOcQ1nUQqsaB26hfbCHAjZkpYKROPdHCR/VfhsbDtsGlEeyKFt1MtNAKbMqxo028SYtCpI2NMprWBusZI01acYVd2uqdUtwGePYzrno89UzquaOSnP6lmnPoFGy3D0tmKH8OBJMDIvjQDLDcZMuetTtol0a5zX0ZWz/SlODnTIPMKmFrgJcf4apS37WznCsWO5G/0XLPYe8RkVbKjJky2pn4z+H7wG10IexnaRmVbZtl7ij6g7d7WRjduUtJ3RAmBZflbGvzunyb/e8cjqYwIHvF9alxEs8DodHbe14tkf2uQRf76YW/nxdLzFXAP3AebIAOPwA54DkZgDxDwGXwHP8DP5GNyknxJvp6WJp2l5i4E8m3X+10DBw=</latexit><latexit sha1_base64="lm0MUkOzo5hm7fNgs9x04ULkiGs=">ACvnicdVLbhMxFHWGVxkKtGs2I6JIBY2qmWxA6gaJLNhUBImkeJRdcdzJ7Hqx8h2SlJ3foAt/8K/8Dd40vAoFVeyztG9Ptf34bIR3Los+9GL7t1/8PDR3uP4yX789Nnzg/2p1SvDcMK0GZWgkXBFU4cdwJnjUGQpcCz8uJ9Fz+7RGO5Vp/dpsFCwkLxmjNwTU+P+hnx9nWkrsk35E+2dn5Ye87rTRbSVSOCbB2nmeNKzwYx5nANqYriw2wC1jgPFAFEm3ht3W2ySB4qTWJhzlkq3b4UHaSW4ZRrQbmTZoVvKDjqBTQU4XIdIWpYyXdsgw3RhoFlytk5rocGlnd5pLWyb3C7G1W8Lz1WzcqjYTS31SiROJ91ckobZE5sAgFmeGgnYUswFyYXhzTEYZ+DZ6G9B8bNOC0e0pmIWEdRv6X9C0Y39ujiFkrEZhM5KHDNPuaorfrlu58PCH7X+VesHcZL0c3pCq+6a5VdIr+lJfziIwyQVfmFaSlDVL2nrR7+T/O8dT5U2svVUTNG4gGaLYdH5v2u9S6bD4zwTxnZIy/IS3JEcvKGvCMfyJhMCMV+Uq+RaeRja5uPkTU2/2MQ3LoufAi3gmA=</latexit><latexit sha1_base64="Fub2ojQsMJlbFEOUyXAaz6WoBc=">ADJnicdVJbaxQxFM6Ot7pW3frqS3BZaGVYZvalQhEK/giVrAXaNYhk8nMhuZGkq2zxvwqwf/gT+ib+Ad81Tcz2/XSFg+E7+PkfCfnklJzZl2WnfWSa9dv3Ly1drt/Z/3uvfuDjfUDq+aG0H2iuDJHJbaUM0n3HXOcHmlDsSg5PSxPnf3h6fUWKbkW7fQdCpwI1nNCHbRVQwIEtjNlPZLEv/IoTCtxBZJqAOEHVQF0gN6MOb7ZbEGtjYoBtcHE58HLgOxcFJ49zcM7eVlQsK1xMRhm42xp8CrJV2QIVrZXbPQ+o0qRuaDSEY6tPc4z7aYeG8cIp6GP5pZqTE5wQ48jlVhQO/XLaQ4ip4K1srEIx1cev9VeCxs12wa0S5E2aGbiQ46gU05drSN2lZirS1UbTxmA9Y6RNa6wSzu9U4rbAC8W4+onU8+knjsqyXkt9ZxDp2A3fVgxQ4nji0gwMSy2A8kMx0m6uKN+H+3S2K+hr2L615oa7JR57BE2jcBtiP03KO3Y38g9HDNWu3H/gsUcRx4xWVfstA3Hk6nfDH4r+FEfwmGOdlDVhVn2gaKPaGc4GfXjJCV9T5QWFa/pcHv/knyv3c8ksqI4BE/oMZFNEuMm84v7/UqOZiM8jfZGANPASPwCbIwTZ4Bl6CPbAPCPgCvoMf4GfyKTlLvp7/iaS3+hwPwAVLv0Ca7EKdw=</latexit><latexit sha1_base64="Fub2ojQsMJlbFEOUyXAaz6WoBc=">ADJnicdVJbaxQxFM6Ot7pW3frqS3BZaGVYZvalQhEK/giVrAXaNYhk8nMhuZGkq2zxvwqwf/gT+ib+Ad81Tcz2/XSFg+E7+PkfCfnklJzZl2WnfWSa9dv3Ly1drt/Z/3uvfuDjfUDq+aG0H2iuDJHJbaUM0n3HXOcHmlDsSg5PSxPnf3h6fUWKbkW7fQdCpwI1nNCHbRVQwIEtjNlPZLEv/IoTCtxBZJqAOEHVQF0gN6MOb7ZbEGtjYoBtcHE58HLgOxcFJ49zcM7eVlQsK1xMRhm42xp8CrJV2QIVrZXbPQ+o0qRuaDSEY6tPc4z7aYeG8cIp6GP5pZqTE5wQ48jlVhQO/XLaQ4ip4K1srEIx1cev9VeCxs12wa0S5E2aGbiQ46gU05drSN2lZirS1UbTxmA9Y6RNa6wSzu9U4rbAC8W4+onU8+knjsqyXkt9ZxDp2A3fVgxQ4nji0gwMSy2A8kMx0m6uKN+H+3S2K+hr2L615oa7JR57BE2jcBtiP03KO3Y38g9HDNWu3H/gsUcRx4xWVfstA3Hk6nfDH4r+FEfwmGOdlDVhVn2gaKPaGc4GfXjJCV9T5QWFa/pcHv/knyv3c8ksqI4BE/oMZFNEuMm84v7/UqOZiM8jfZGANPASPwCbIwTZ4Bl6CPbAPCPgCvoMf4GfyKTlLvp7/iaS3+hwPwAVLv0Ca7EKdw=</latexit><latexit sha1_base64="Rb6jVQcGWRa2QxhG8aso70jUZU8=">ADMXicdVFbaxQxFE7HWx1vrT76ElwKrQxlZl8UilCwgjiCu620KxDJpPZDc2NJFtnjflVgv/Bn9A38dUHX/XNzHa9tMUD4fs4Od9JzvkqzZl1eX6ykly6fOXqtdXr6Y2bt27fWVu/O7JqZgdEsWVOaiwpZxJOnTMcXqgDcWi4nS/Onra3e8fU2OZkm/cXNOxwBPJGkawi6lyjSCB3VRpv8Cq8s9CKH0LkWUC6gARVxOoS+Sm1OHNdgsirLVRsaAxmPgieBmQnYnSsydFeCvPC0q2tV2u9fLtfBHwIimWpAeWMSjXVz6hWpGZoNIRjq09LHLtxh4bxwinIUzSzUmR3hCDyOVWFA79otBLgRMzVslIlHOrjI/qvwWNhu2CyinYuqQzcVHXQCm3HsaBtvsqoSWujGYTg/WUkTZruMIu6/ROKW4DPsZ1zweyb1zFJTv/SzDh0CnbhzUzlDg+jwQTw+I4kEx3KSLHqUp2qNxXkNfxvavNDXYKfPQI2wmArchzj9BWcf+Vg5w7FjvRf8Fiz0OPGKyqdlxGw7Y78Z/FbwGymEvQLtoLors+w9R/QTq+/kcZNSvqOKCGwrH9Lg9/70+R/73gklRHBIz6ixkU0C4xOF+d9vUhG/e0i8td5b/fF0vNVcB8AJugAI/ALngOBmAICPgMvoMf4GfyMTlJviRfT0uTlaXmHjgTybdf7DQMGA=</latexit><latexit sha1_base64="a95aEXi6A6sxpts9eEGxlWxWsU=">ADMXicdVFbaxQxFM6Ot7pe2uqjL8Gl0JVh2SmCQhEKVhBXMG2C5t1yGQys6G5kWTr1JhfJfgf/Al9E198FXfzGzXS1s8EL6Pk/Od5Jyv0JxZNxyedJLl69cvbZyvXvj5q3bq2vrd/atmhtC94jiyowLbClnku45jgda0OxKDg9KA6ftvcHR9RYpuQbd6zpVOBasoR7GIqXyNIYDdT2i+wKPyzEHLfQGSZgDpAxFUNdY7cjDq82fQhwlobFQsqg4nPgpcB2bnIPXuShbfyvCBn/UG+1hsOhouAF0m2JD2wjFG+3vmESkXmgkpHOLZ2kg21m3psHCOchi6aW6oxOcQ1nUQqsaB26hfbCHAjZkpYKROPdHCR/VfhsbDtsGlEeyKFt1MtNAKbMqxo028SYtCpI2NMprWBusZI01acYVd2uqdUtwGePYzrno89UzquaOSnP6lmnPoFGy3D0tmKH8OBJMDIvjQDLDcZMuetTtol0a5zX0ZWz/SlODnTIPMKmFrgJcf4apS37WznCsWO5G/0XLPYe8RkVbKjJky2pn4z+H7wG10IexnaRmVbZtl7ij6g7d7WRjduUtJ3RAmBZflbGvzunyb/e8cjqYwIHvF9alxEs8DodHbe14tkf2uQRf76YW/nxdLzFXAP3AebIAOPwA54DkZgDxDwGXwHP8DP5GNyknxJvp6WJp2l5i4E8m3X+10DBw=</latexit><latexit sha1_base64="a95aEXi6A6sxpts9eEGxlWxWsU=">ADMXicdVFbaxQxFM6Ot7pe2uqjL8Gl0JVh2SmCQhEKVhBXMG2C5t1yGQys6G5kWTr1JhfJfgf/Al9E198FXfzGzXS1s8EL6Pk/Od5Jyv0JxZNxyedJLl69cvbZyvXvj5q3bq2vrd/atmhtC94jiyowLbClnku45jgda0OxKDg9KA6ftvcHR9RYpuQbd6zpVOBasoR7GIqXyNIYDdT2i+wKPyzEHLfQGSZgDpAxFUNdY7cjDq82fQhwlobFQsqg4nPgpcB2bnIPXuShbfyvCBn/UG+1hsOhouAF0m2JD2wjFG+3vmESkXmgkpHOLZ2kg21m3psHCOchi6aW6oxOcQ1nUQqsaB26hfbCHAjZkpYKROPdHCR/VfhsbDtsGlEeyKFt1MtNAKbMqxo028SYtCpI2NMprWBusZI01acYVd2uqdUtwGePYzrno89UzquaOSnP6lmnPoFGy3D0tmKH8OBJMDIvjQDLDcZMuetTtol0a5zX0ZWz/SlODnTIPMKmFrgJcf4apS37WznCsWO5G/0XLPYe8RkVbKjJky2pn4z+H7wG10IexnaRmVbZtl7ij6g7d7WRjduUtJ3RAmBZflbGvzunyb/e8cjqYwIHvF9alxEs8DodHbe14tkf2uQRf76YW/nxdLzFXAP3AebIAOPwA54DkZgDxDwGXwHP8DP5GNyknxJvp6WJp2l5i4E8m3X+10DBw=</latexit><latexit sha1_base64="a95aEXi6A6sxpts9eEGxlWxWsU=">ADMXicdVFbaxQxFM6Ot7pe2uqjL8Gl0JVh2SmCQhEKVhBXMG2C5t1yGQys6G5kWTr1JhfJfgf/Al9E198FXfzGzXS1s8EL6Pk/Od5Jyv0JxZNxyedJLl69cvbZyvXvj5q3bq2vrd/atmhtC94jiyowLbClnku45jgda0OxKDg9KA6ftvcHR9RYpuQbd6zpVOBasoR7GIqXyNIYDdT2i+wKPyzEHLfQGSZgDpAxFUNdY7cjDq82fQhwlobFQsqg4nPgpcB2bnIPXuShbfyvCBn/UG+1hsOhouAF0m2JD2wjFG+3vmESkXmgkpHOLZ2kg21m3psHCOchi6aW6oxOcQ1nUQqsaB26hfbCHAjZkpYKROPdHCR/VfhsbDtsGlEeyKFt1MtNAKbMqxo028SYtCpI2NMprWBusZI01acYVd2uqdUtwGePYzrno89UzquaOSnP6lmnPoFGy3D0tmKH8OBJMDIvjQDLDcZMuetTtol0a5zX0ZWz/SlODnTIPMKmFrgJcf4apS37WznCsWO5G/0XLPYe8RkVbKjJky2pn4z+H7wG10IexnaRmVbZtl7ij6g7d7WRjduUtJ3RAmBZflbGvzunyb/e8cjqYwIHvF9alxEs8DodHbe14tkf2uQRf76YW/nxdLzFXAP3AebIAOPwA54DkZgDxDwGXwHP8DP5GNyknxJvp6WJp2l5i4E8m3X+10DBw=</latexit><latexit sha1_base64="a95aEXi6A6sxpts9eEGxlWxWsU=">ADMXicdVFbaxQxFM6Ot7pe2uqjL8Gl0JVh2SmCQhEKVhBXMG2C5t1yGQys6G5kWTr1JhfJfgf/Al9E198FXfzGzXS1s8EL6Pk/Od5Jyv0JxZNxyedJLl69cvbZyvXvj5q3bq2vrd/atmhtC94jiyowLbClnku45jgda0OxKDg9KA6ftvcHR9RYpuQbd6zpVOBasoR7GIqXyNIYDdT2i+wKPyzEHLfQGSZgDpAxFUNdY7cjDq82fQhwlobFQsqg4nPgpcB2bnIPXuShbfyvCBn/UG+1hsOhouAF0m2JD2wjFG+3vmESkXmgkpHOLZ2kg21m3psHCOchi6aW6oxOcQ1nUQqsaB26hfbCHAjZkpYKROPdHCR/VfhsbDtsGlEeyKFt1MtNAKbMqxo028SYtCpI2NMprWBusZI01acYVd2uqdUtwGePYzrno89UzquaOSnP6lmnPoFGy3D0tmKH8OBJMDIvjQDLDcZMuetTtol0a5zX0ZWz/SlODnTIPMKmFrgJcf4apS37WznCsWO5G/0XLPYe8RkVbKjJky2pn4z+H7wG10IexnaRmVbZtl7ij6g7d7WRjduUtJ3RAmBZflbGvzunyb/e8cjqYwIHvF9alxEs8DodHbe14tkf2uQRf76YW/nxdLzFXAP3AebIAOPwA54DkZgDxDwGXwHP8DP5GNyknxJvp6WJp2l5i4E8m3X+10DBw=</latexit><latexit sha1_base64="a95aEXi6A6sxpts9eEGxlWxWsU=">ADMXicdVFbaxQxFM6Ot7pe2uqjL8Gl0JVh2SmCQhEKVhBXMG2C5t1yGQys6G5kWTr1JhfJfgf/Al9E198FXfzGzXS1s8EL6Pk/Od5Jyv0JxZNxyedJLl69cvbZyvXvj5q3bq2vrd/atmhtC94jiyowLbClnku45jgda0OxKDg9KA6ftvcHR9RYpuQbd6zpVOBasoR7GIqXyNIYDdT2i+wKPyzEHLfQGSZgDpAxFUNdY7cjDq82fQhwlobFQsqg4nPgpcB2bnIPXuShbfyvCBn/UG+1hsOhouAF0m2JD2wjFG+3vmESkXmgkpHOLZ2kg21m3psHCOchi6aW6oxOcQ1nUQqsaB26hfbCHAjZkpYKROPdHCR/VfhsbDtsGlEeyKFt1MtNAKbMqxo028SYtCpI2NMprWBusZI01acYVd2uqdUtwGePYzrno89UzquaOSnP6lmnPoFGy3D0tmKH8OBJMDIvjQDLDcZMuetTtol0a5zX0ZWz/SlODnTIPMKmFrgJcf4apS37WznCsWO5G/0XLPYe8RkVbKjJky2pn4z+H7wG10IexnaRmVbZtl7ij6g7d7WRjduUtJ3RAmBZflbGvzunyb/e8cjqYwIHvF9alxEs8DodHbe14tkf2uQRf76YW/nxdLzFXAP3AebIAOPwA54DkZgDxDwGXwHP8DP5GNyknxJvp6WJp2l5i4E8m3X+10DBw=</latexit><latexit sha1_base64="a95aEXi6A6sxpts9eEGxlWxWsU=">ADMXicdVFbaxQxFM6Ot7pe2uqjL8Gl0JVh2SmCQhEKVhBXMG2C5t1yGQys6G5kWTr1JhfJfgf/Al9E198FXfzGzXS1s8EL6Pk/Od5Jyv0JxZNxyedJLl69cvbZyvXvj5q3bq2vrd/atmhtC94jiyowLbClnku45jgda0OxKDg9KA6ftvcHR9RYpuQbd6zpVOBasoR7GIqXyNIYDdT2i+wKPyzEHLfQGSZgDpAxFUNdY7cjDq82fQhwlobFQsqg4nPgpcB2bnIPXuShbfyvCBn/UG+1hsOhouAF0m2JD2wjFG+3vmESkXmgkpHOLZ2kg21m3psHCOchi6aW6oxOcQ1nUQqsaB26hfbCHAjZkpYKROPdHCR/VfhsbDtsGlEeyKFt1MtNAKbMqxo028SYtCpI2NMprWBusZI01acYVd2uqdUtwGePYzrno89UzquaOSnP6lmnPoFGy3D0tmKH8OBJMDIvjQDLDcZMuetTtol0a5zX0ZWz/SlODnTIPMKmFrgJcf4apS37WznCsWO5G/0XLPYe8RkVbKjJky2pn4z+H7wG10IexnaRmVbZtl7ij6g7d7WRjduUtJ3RAmBZflbGvzunyb/e8cjqYwIHvF9alxEs8DodHbe14tkf2uQRf76YW/nxdLzFXAP3AebIAOPwA54DkZgDxDwGXwHP8DP5GNyknxJvp6WJp2l5i4E8m3X+10DBw=</latexit>
slide-11
SLIDE 11

5-Minute Break

CSE 599I: Generative Models University of Washington Lecture 2

slide-12
SLIDE 12

Recap: Evaluating the Likelihood

CSE 599I: Generative Models University of Washington Lecture 2

  • To run SGD, we needed to evaluate the marginal probability :
  • For GMM’s, this is tractable (the integral is a simple analytical sum).
  • What would we do if the integral weren’t tractable?

pθ(x)

<latexit sha1_base64="4wvVJDUMns+Cpw3HhJZl69eqS2Y=">AB8nicbZBNS8NAEIYn9avWr6pHL4tFqJeSiKDHghfxVMF+QBvKZrtpl242YXciltCf4cWDIl79Nd78N27bHLT1hYWHd2bYmTdIpDout9OYW19Y3OruF3a2d3bPygfHrVMnGrGmyWse4E1HApFG+iQMk7ieY0CiRvB+ObWb39yLURsXrAScL9iA6VCAWjaK1u0u/hiCOtPp3yxW35s5FVsHLoQK5Gv3yV28QszTiCpmkxnQ9N0E/oxoFk3xa6qWGJ5SN6ZB3LSoaceNn85Wn5Mw6AxLG2j6FZO7+nshoZMwkCmxnRHFklmsz879aN8Xw2s+ESlLki0+ClNJMCaz+8lAaM5QTixQpoXdlbAR1ZShTalkQ/CWT16F1kXNs3x/Wanf5XEU4QROoQoeXEdbqEBTWAQwzO8wpuDzovz7nwsWgtOPnMf+R8/gDZfZD6</latexit><latexit sha1_base64="4wvVJDUMns+Cpw3HhJZl69eqS2Y=">AB8nicbZBNS8NAEIYn9avWr6pHL4tFqJeSiKDHghfxVMF+QBvKZrtpl242YXciltCf4cWDIl79Nd78N27bHLT1hYWHd2bYmTdIpDout9OYW19Y3OruF3a2d3bPygfHrVMnGrGmyWse4E1HApFG+iQMk7ieY0CiRvB+ObWb39yLURsXrAScL9iA6VCAWjaK1u0u/hiCOtPp3yxW35s5FVsHLoQK5Gv3yV28QszTiCpmkxnQ9N0E/oxoFk3xa6qWGJ5SN6ZB3LSoaceNn85Wn5Mw6AxLG2j6FZO7+nshoZMwkCmxnRHFklmsz879aN8Xw2s+ESlLki0+ClNJMCaz+8lAaM5QTixQpoXdlbAR1ZShTalkQ/CWT16F1kXNs3x/Wanf5XEU4QROoQoeXEdbqEBTWAQwzO8wpuDzovz7nwsWgtOPnMf+R8/gDZfZD6</latexit><latexit sha1_base64="4wvVJDUMns+Cpw3HhJZl69eqS2Y=">AB8nicbZBNS8NAEIYn9avWr6pHL4tFqJeSiKDHghfxVMF+QBvKZrtpl242YXciltCf4cWDIl79Nd78N27bHLT1hYWHd2bYmTdIpDout9OYW19Y3OruF3a2d3bPygfHrVMnGrGmyWse4E1HApFG+iQMk7ieY0CiRvB+ObWb39yLURsXrAScL9iA6VCAWjaK1u0u/hiCOtPp3yxW35s5FVsHLoQK5Gv3yV28QszTiCpmkxnQ9N0E/oxoFk3xa6qWGJ5SN6ZB3LSoaceNn85Wn5Mw6AxLG2j6FZO7+nshoZMwkCmxnRHFklmsz879aN8Xw2s+ESlLki0+ClNJMCaz+8lAaM5QTixQpoXdlbAR1ZShTalkQ/CWT16F1kXNs3x/Wanf5XEU4QROoQoeXEdbqEBTWAQwzO8wpuDzovz7nwsWgtOPnMf+R8/gDZfZD6</latexit><latexit sha1_base64="4wvVJDUMns+Cpw3HhJZl69eqS2Y=">AB8nicbZBNS8NAEIYn9avWr6pHL4tFqJeSiKDHghfxVMF+QBvKZrtpl242YXciltCf4cWDIl79Nd78N27bHLT1hYWHd2bYmTdIpDout9OYW19Y3OruF3a2d3bPygfHrVMnGrGmyWse4E1HApFG+iQMk7ieY0CiRvB+ObWb39yLURsXrAScL9iA6VCAWjaK1u0u/hiCOtPp3yxW35s5FVsHLoQK5Gv3yV28QszTiCpmkxnQ9N0E/oxoFk3xa6qWGJ5SN6ZB3LSoaceNn85Wn5Mw6AxLG2j6FZO7+nshoZMwkCmxnRHFklmsz879aN8Xw2s+ESlLki0+ClNJMCaz+8lAaM5QTixQpoXdlbAR1ZShTalkQ/CWT16F1kXNs3x/Wanf5XEU4QROoQoeXEdbqEBTWAQwzO8wpuDzovz7nwsWgtOPnMf+R8/gDZfZD6</latexit>

pθ(x) = Z

Z

pθ(x, z) dz = Z

Z

pθ(x|z)pθ(z) dz.

<latexit sha1_base64="jR6vKQaJ9+bknO+CNGk/kvhSlSU=">ACS3icfVBLSwMxGMxW6O+qh69BItiQcquCHoRCl7Ek4JVsbs2TRtQ7PZJflWbNf+Py9evPknvHhQxIPZtviqOBCYzMxH8k0QC67Bth+t3MRkfmp6ZrYwN7+wuFRcXjnXUaIoq9FIROoyIJoJLlkNOAh2GStGwkCwi6BzmPkX10xpHskz6MbMC0lL8ianBIzkF4PYd6HNgGzdlPHmAXa5BN8NCbQpEelVH3/5272yu93o4X9Dt73y52WYr/jFkl2xB8DjxBmREhrhxC8+uI2IJiGTQAXRu7YMXgpUcCpYP2Cm2gWE9ohLVY3VJKQaS8dNHG0Zp4GakzJGAB+r3iZSEWnfDwCSzBfRvLxP/8uoJNPe9lMs4ASbp8KFmIjBEOCsWN7hiFETXEIVN3/FtE0UoWDqL5gSnN8rj5PznYpj+OluqXo8qmMGraF1tIUctIeq6AidoBqi6A49oRf0at1bz9ab9T6M5qzRzCr6gVz+A+GisQ=</latexit><latexit sha1_base64="jR6vKQaJ9+bknO+CNGk/kvhSlSU=">ACS3icfVBLSwMxGMxW6O+qh69BItiQcquCHoRCl7Ek4JVsbs2TRtQ7PZJflWbNf+Py9evPknvHhQxIPZtviqOBCYzMxH8k0QC67Bth+t3MRkfmp6ZrYwN7+wuFRcXjnXUaIoq9FIROoyIJoJLlkNOAh2GStGwkCwi6BzmPkX10xpHskz6MbMC0lL8ianBIzkF4PYd6HNgGzdlPHmAXa5BN8NCbQpEelVH3/5272yu93o4X9Dt73y52WYr/jFkl2xB8DjxBmREhrhxC8+uI2IJiGTQAXRu7YMXgpUcCpYP2Cm2gWE9ohLVY3VJKQaS8dNHG0Zp4GakzJGAB+r3iZSEWnfDwCSzBfRvLxP/8uoJNPe9lMs4ASbp8KFmIjBEOCsWN7hiFETXEIVN3/FtE0UoWDqL5gSnN8rj5PznYpj+OluqXo8qmMGraF1tIUctIeq6AidoBqi6A49oRf0at1bz9ab9T6M5qzRzCr6gVz+A+GisQ=</latexit><latexit sha1_base64="jR6vKQaJ9+bknO+CNGk/kvhSlSU=">ACS3icfVBLSwMxGMxW6O+qh69BItiQcquCHoRCl7Ek4JVsbs2TRtQ7PZJflWbNf+Py9evPknvHhQxIPZtviqOBCYzMxH8k0QC67Bth+t3MRkfmp6ZrYwN7+wuFRcXjnXUaIoq9FIROoyIJoJLlkNOAh2GStGwkCwi6BzmPkX10xpHskz6MbMC0lL8ianBIzkF4PYd6HNgGzdlPHmAXa5BN8NCbQpEelVH3/5272yu93o4X9Dt73y52WYr/jFkl2xB8DjxBmREhrhxC8+uI2IJiGTQAXRu7YMXgpUcCpYP2Cm2gWE9ohLVY3VJKQaS8dNHG0Zp4GakzJGAB+r3iZSEWnfDwCSzBfRvLxP/8uoJNPe9lMs4ASbp8KFmIjBEOCsWN7hiFETXEIVN3/FtE0UoWDqL5gSnN8rj5PznYpj+OluqXo8qmMGraF1tIUctIeq6AidoBqi6A49oRf0at1bz9ab9T6M5qzRzCr6gVz+A+GisQ=</latexit><latexit sha1_base64="jR6vKQaJ9+bknO+CNGk/kvhSlSU=">ACS3icfVBLSwMxGMxW6O+qh69BItiQcquCHoRCl7Ek4JVsbs2TRtQ7PZJflWbNf+Py9evPknvHhQxIPZtviqOBCYzMxH8k0QC67Bth+t3MRkfmp6ZrYwN7+wuFRcXjnXUaIoq9FIROoyIJoJLlkNOAh2GStGwkCwi6BzmPkX10xpHskz6MbMC0lL8ianBIzkF4PYd6HNgGzdlPHmAXa5BN8NCbQpEelVH3/5272yu93o4X9Dt73y52WYr/jFkl2xB8DjxBmREhrhxC8+uI2IJiGTQAXRu7YMXgpUcCpYP2Cm2gWE9ohLVY3VJKQaS8dNHG0Zp4GakzJGAB+r3iZSEWnfDwCSzBfRvLxP/8uoJNPe9lMs4ASbp8KFmIjBEOCsWN7hiFETXEIVN3/FtE0UoWDqL5gSnN8rj5PznYpj+OluqXo8qmMGraF1tIUctIeq6AidoBqi6A49oRf0at1bz9ab9T6M5qzRzCr6gVz+A+GisQ=</latexit>
slide-13
SLIDE 13

The Evidence Lower-Bound

CSE 599I: Generative Models University of Washington Lecture 2

  • Approximate the marginal with importance sampling:
  • The distribution is called a proposal distribution
  • Bound is tight when (when the proposal is the posterior)

q(z|x)

<latexit sha1_base64="R8QmCbgIMTfQExIUTVUJLgPCKk=">ACqXicdVFdb9MwFHXD1whfGzyYhFV6kQ0NRUSHuZRB8QT51Eu4omhznpjXzR7Cdkc7LT0DiFf4Z/wanK4gxcSXrHN17j319bl5xZuxw+LMX3Lp95+69nfvhg4ePHj/Z3Xs6M6rWFKZUcaXnOTHAmYSpZbDvNJARM7hJD9729VPzkEbpuQHu64gE2QpWckosT41+zy4uGz2T3ej4cFwE/gmSbYkQtuYnO71vqaForUAaSknxiySYWUzR7RlEMbprWBitAzsoSFp5IMJnbjNvivs8UuFTaH2nxJvu3whFhBLGr2KNZi7xDuxIdAITc2Kh8ZU4z0XcGC+DeKlJtWK0iUuiI07vVWKmxZfH8aWbzLHZFVbkPRqlrLm2Crc2YMLpoFavaEUM38dzBdEU2o9SaGYZiOwX9Yw4T4xmLsfRfMl+YuZbIs2HnTLkaZG7Ruv3X9EOMoSQ/Tomsz7ALSy/QwGvVDb5CEL1QJQWTxW9q68Z9L/veOS6XSonUpn4G2HvUG/QKTf9d1k8xGB4nx6+io/fbVe6g5+gFGqAEvUZH6B2aoCmi6BP6hr6jH8HL4DiYBx+vWoPeVvMXYuA/gJt/dRU</latexit><latexit sha1_base64="R8QmCbgIMTfQExIUTVUJLgPCKk=">ACqXicdVFdb9MwFHXD1whfGzyYhFV6kQ0NRUSHuZRB8QT51Eu4omhznpjXzR7Cdkc7LT0DiFf4Z/wanK4gxcSXrHN17j319bl5xZuxw+LMX3Lp95+69nfvhg4ePHj/Z3Xs6M6rWFKZUcaXnOTHAmYSpZbDvNJARM7hJD9729VPzkEbpuQHu64gE2QpWckosT41+zy4uGz2T3ej4cFwE/gmSbYkQtuYnO71vqaForUAaSknxiySYWUzR7RlEMbprWBitAzsoSFp5IMJnbjNvivs8UuFTaH2nxJvu3whFhBLGr2KNZi7xDuxIdAITc2Kh8ZU4z0XcGC+DeKlJtWK0iUuiI07vVWKmxZfH8aWbzLHZFVbkPRqlrLm2Crc2YMLpoFavaEUM38dzBdEU2o9SaGYZiOwX9Yw4T4xmLsfRfMl+YuZbIs2HnTLkaZG7Ruv3X9EOMoSQ/Tomsz7ALSy/QwGvVDb5CEL1QJQWTxW9q68Z9L/veOS6XSonUpn4G2HvUG/QKTf9d1k8xGB4nx6+io/fbVe6g5+gFGqAEvUZH6B2aoCmi6BP6hr6jH8HL4DiYBx+vWoPeVvMXYuA/gJt/dRU</latexit><latexit sha1_base64="R8QmCbgIMTfQExIUTVUJLgPCKk=">ACqXicdVFdb9MwFHXD1whfGzyYhFV6kQ0NRUSHuZRB8QT51Eu4omhznpjXzR7Cdkc7LT0DiFf4Z/wanK4gxcSXrHN17j319bl5xZuxw+LMX3Lp95+69nfvhg4ePHj/Z3Xs6M6rWFKZUcaXnOTHAmYSpZbDvNJARM7hJD9729VPzkEbpuQHu64gE2QpWckosT41+zy4uGz2T3ej4cFwE/gmSbYkQtuYnO71vqaForUAaSknxiySYWUzR7RlEMbprWBitAzsoSFp5IMJnbjNvivs8UuFTaH2nxJvu3whFhBLGr2KNZi7xDuxIdAITc2Kh8ZU4z0XcGC+DeKlJtWK0iUuiI07vVWKmxZfH8aWbzLHZFVbkPRqlrLm2Crc2YMLpoFavaEUM38dzBdEU2o9SaGYZiOwX9Yw4T4xmLsfRfMl+YuZbIs2HnTLkaZG7Ruv3X9EOMoSQ/Tomsz7ALSy/QwGvVDb5CEL1QJQWTxW9q68Z9L/veOS6XSonUpn4G2HvUG/QKTf9d1k8xGB4nx6+io/fbVe6g5+gFGqAEvUZH6B2aoCmi6BP6hr6jH8HL4DiYBx+vWoPeVvMXYuA/gJt/dRU</latexit><latexit sha1_base64="R8QmCbgIMTfQExIUTVUJLgPCKk=">ACqXicdVFdb9MwFHXD1whfGzyYhFV6kQ0NRUSHuZRB8QT51Eu4omhznpjXzR7Cdkc7LT0DiFf4Z/wanK4gxcSXrHN17j319bl5xZuxw+LMX3Lp95+69nfvhg4ePHj/Z3Xs6M6rWFKZUcaXnOTHAmYSpZbDvNJARM7hJD9729VPzkEbpuQHu64gE2QpWckosT41+zy4uGz2T3ej4cFwE/gmSbYkQtuYnO71vqaForUAaSknxiySYWUzR7RlEMbprWBitAzsoSFp5IMJnbjNvivs8UuFTaH2nxJvu3whFhBLGr2KNZi7xDuxIdAITc2Kh8ZU4z0XcGC+DeKlJtWK0iUuiI07vVWKmxZfH8aWbzLHZFVbkPRqlrLm2Crc2YMLpoFavaEUM38dzBdEU2o9SaGYZiOwX9Yw4T4xmLsfRfMl+YuZbIs2HnTLkaZG7Ruv3X9EOMoSQ/Tomsz7ALSy/QwGvVDb5CEL1QJQWTxW9q68Z9L/veOS6XSonUpn4G2HvUG/QKTf9d1k8xGB4nx6+io/fbVe6g5+gFGqAEvUZH6B2aoCmi6BP6hr6jH8HL4DiYBx+vWoPeVvMXYuA/gJt/dRU</latexit>

log pθ(x) = log E

z∼q(·|x)

pθ(x, z) q(z|x)

  • =

E

z∼q(·|x)

 log pθ(x, z) q(z|x)

  • + D(q(z|x) k pθ(z|x))
  • E

z∼q(·|x)

 log pθ(x, z) q(z|x)

  • .
<latexit sha1_base64="1TLbHbYBw5qNrbwjv36dJRTRSqw=">AD63ictVJbaxNBFN5tvNT1uqjL4OhkuJSkiAoFKFgCuJTBXuBTAizs2eToTM725mTma6P8En38RX3/w1von+Gc3qbUWER8MHyHc75zn6SQwmK7/TVcaly5eu368o3o5q3bd+6urN7bs3piOxyLbU5SJgFKXLYRYESDgoDTCUS9pPDF5V/xiMFTp/gycFDBQb5SITnKE3DVfDbSr1iBRDimNA1pquE/LoOamNVDEc68LVmCRuyHbkaoFYoctShPNZ5O10sqIcM+zQzj7jxPFsv3VFrVjOMGI1xQGlUpf6nrHUf09NHhMq8iwVx2eOc/6c52vTERz9n/Ibw5Vme6NdC7msdBZKM1jIjl/8O5pqPlGQI5fM2n6nXeDAMYOCSygjOrFQMH7IRtD3as4U2IGrD16SNW9JSaNfzmS2vprhGPKVvPFHu2JSirEsaqgCrCxZAhT74mTRMVT68MgHhlWjAWfxpnUDOMqHrWtiQXm8Hs2cCJvJg5HzeSzaRBDWpPhJhQGO8sQrjBvhxyF8zPwK0X/DKIpoD/zABnaYJ6Y9/3OV8K4DtzjgtOx3B65VOr/ltYiQZodu0rSiWTEDeko3m921yC8oh7dcK8Xy9Cy0dL2fSf5Ux9FcG1U6KvfAoEdToz9g5/dzXVb2uhsdr79+0tx6tTjlcvAgeBi0gk7wNgKXgY7wW7Aw8/hl/Bb+L2hGu8bHxof59SlcBFzP7gjU8/AJwTzM=</latexit><latexit sha1_base64="1TLbHbYBw5qNrbwjv36dJRTRSqw=">AD63ictVJbaxNBFN5tvNT1uqjL4OhkuJSkiAoFKFgCuJTBXuBTAizs2eToTM725mTma6P8En38RX3/w1von+Gc3qbUWER8MHyHc75zn6SQwmK7/TVcaly5eu368o3o5q3bd+6urN7bs3piOxyLbU5SJgFKXLYRYESDgoDTCUS9pPDF5V/xiMFTp/gycFDBQb5SITnKE3DVfDbSr1iBRDimNA1pquE/LoOamNVDEc68LVmCRuyHbkaoFYoctShPNZ5O10sqIcM+zQzj7jxPFsv3VFrVjOMGI1xQGlUpf6nrHUf09NHhMq8iwVx2eOc/6c52vTERz9n/Ibw5Vme6NdC7msdBZKM1jIjl/8O5pqPlGQI5fM2n6nXeDAMYOCSygjOrFQMH7IRtD3as4U2IGrD16SNW9JSaNfzmS2vprhGPKVvPFHu2JSirEsaqgCrCxZAhT74mTRMVT68MgHhlWjAWfxpnUDOMqHrWtiQXm8Hs2cCJvJg5HzeSzaRBDWpPhJhQGO8sQrjBvhxyF8zPwK0X/DKIpoD/zABnaYJ6Y9/3OV8K4DtzjgtOx3B65VOr/ltYiQZodu0rSiWTEDeko3m921yC8oh7dcK8Xy9Cy0dL2fSf5Ux9FcG1U6KvfAoEdToz9g5/dzXVb2uhsdr79+0tx6tTjlcvAgeBi0gk7wNgKXgY7wW7Aw8/hl/Bb+L2hGu8bHxof59SlcBFzP7gjU8/AJwTzM=</latexit><latexit sha1_base64="1TLbHbYBw5qNrbwjv36dJRTRSqw=">AD63ictVJbaxNBFN5tvNT1uqjL4OhkuJSkiAoFKFgCuJTBXuBTAizs2eToTM725mTma6P8En38RX3/w1von+Gc3qbUWER8MHyHc75zn6SQwmK7/TVcaly5eu368o3o5q3bd+6urN7bs3piOxyLbU5SJgFKXLYRYESDgoDTCUS9pPDF5V/xiMFTp/gycFDBQb5SITnKE3DVfDbSr1iBRDimNA1pquE/LoOamNVDEc68LVmCRuyHbkaoFYoctShPNZ5O10sqIcM+zQzj7jxPFsv3VFrVjOMGI1xQGlUpf6nrHUf09NHhMq8iwVx2eOc/6c52vTERz9n/Ibw5Vme6NdC7msdBZKM1jIjl/8O5pqPlGQI5fM2n6nXeDAMYOCSygjOrFQMH7IRtD3as4U2IGrD16SNW9JSaNfzmS2vprhGPKVvPFHu2JSirEsaqgCrCxZAhT74mTRMVT68MgHhlWjAWfxpnUDOMqHrWtiQXm8Hs2cCJvJg5HzeSzaRBDWpPhJhQGO8sQrjBvhxyF8zPwK0X/DKIpoD/zABnaYJ6Y9/3OV8K4DtzjgtOx3B65VOr/ltYiQZodu0rSiWTEDeko3m921yC8oh7dcK8Xy9Cy0dL2fSf5Ux9FcG1U6KvfAoEdToz9g5/dzXVb2uhsdr79+0tx6tTjlcvAgeBi0gk7wNgKXgY7wW7Aw8/hl/Bb+L2hGu8bHxof59SlcBFzP7gjU8/AJwTzM=</latexit><latexit sha1_base64="1TLbHbYBw5qNrbwjv36dJRTRSqw=">AD63ictVJbaxNBFN5tvNT1uqjL4OhkuJSkiAoFKFgCuJTBXuBTAizs2eToTM725mTma6P8En38RX3/w1von+Gc3qbUWER8MHyHc75zn6SQwmK7/TVcaly5eu368o3o5q3bd+6urN7bs3piOxyLbU5SJgFKXLYRYESDgoDTCUS9pPDF5V/xiMFTp/gycFDBQb5SITnKE3DVfDbSr1iBRDimNA1pquE/LoOamNVDEc68LVmCRuyHbkaoFYoctShPNZ5O10sqIcM+zQzj7jxPFsv3VFrVjOMGI1xQGlUpf6nrHUf09NHhMq8iwVx2eOc/6c52vTERz9n/Ibw5Vme6NdC7msdBZKM1jIjl/8O5pqPlGQI5fM2n6nXeDAMYOCSygjOrFQMH7IRtD3as4U2IGrD16SNW9JSaNfzmS2vprhGPKVvPFHu2JSirEsaqgCrCxZAhT74mTRMVT68MgHhlWjAWfxpnUDOMqHrWtiQXm8Hs2cCJvJg5HzeSzaRBDWpPhJhQGO8sQrjBvhxyF8zPwK0X/DKIpoD/zABnaYJ6Y9/3OV8K4DtzjgtOx3B65VOr/ltYiQZodu0rSiWTEDeko3m921yC8oh7dcK8Xy9Cy0dL2fSf5Ux9FcG1U6KvfAoEdToz9g5/dzXVb2uhsdr79+0tx6tTjlcvAgeBi0gk7wNgKXgY7wW7Aw8/hl/Bb+L2hGu8bHxof59SlcBFzP7gjU8/AJwTzM=</latexit>

q(z|x) = p(z|x)

<latexit sha1_base64="G3Q4BqZesyNr1DXJYFyqNrg4cY=">AC3XicdVFbixMxFE7H2zpetquPvgRLoSvD0imCwiIsuA8iBVst9CUksmcacMmkzHJrNPNzqNv4qsPvupf8L/4b8x062VdPBC+j3POd3IuSG4sf3+j1Zw5eq16ze2boa3bt+5u93euTc2qtQMRkwJpScJNSB4DiPLrYBJoYHKRMBRcvy8iR+dgDZc5W/tqoCZpIucZ5xR613z9va73ulZtYuf4WJN5u1Of6+/NnyZxBvSQRsbznda30mqWCkht0xQY6Zxv7AzR7XlTEAdktJAQdkxXcDU05xKMDO37rzGXe9Jca0f7nFa+/fCkelkdQuI49mJZMG7VI20AhMJKiFykeiJFRZbwMoWmxZKzKsqEojZq9FYpYWp8sRmbPZ05nhelhZyd95KVAluFm03hlGtgVqw8oUxzPw5mS6ops36fYUgOwc+r4ZUv/7oATa3SjxyheiFpVfv5FyRq2J/MIfUV0N/K8l9jYkjPM9SflLV08HM9Wq3W7tuiHEnJvskbdIMPwVyRvY7g27oN5nDe6akpHn6S1q7w9F/vePI7nSsnZEjEFbj3qN/tLxv3e9TMaDvdjzN487By83N9CD9BD1EMxeoIO0As0RCPEUIm+oK/oWzAPgQfg0/nqUFro7mPLljw+SczbOhr</latexit><latexit sha1_base64="G3Q4BqZesyNr1DXJYFyqNrg4cY=">AC3XicdVFbixMxFE7H2zpetquPvgRLoSvD0imCwiIsuA8iBVst9CUksmcacMmkzHJrNPNzqNv4qsPvupf8L/4b8x062VdPBC+j3POd3IuSG4sf3+j1Zw5eq16ze2boa3bt+5u93euTc2qtQMRkwJpScJNSB4DiPLrYBJoYHKRMBRcvy8iR+dgDZc5W/tqoCZpIucZ5xR613z9va73ulZtYuf4WJN5u1Of6+/NnyZxBvSQRsbznda30mqWCkht0xQY6Zxv7AzR7XlTEAdktJAQdkxXcDU05xKMDO37rzGXe9Jca0f7nFa+/fCkelkdQuI49mJZMG7VI20AhMJKiFykeiJFRZbwMoWmxZKzKsqEojZq9FYpYWp8sRmbPZ05nhelhZyd95KVAluFm03hlGtgVqw8oUxzPw5mS6ops36fYUgOwc+r4ZUv/7oATa3SjxyheiFpVfv5FyRq2J/MIfUV0N/K8l9jYkjPM9SflLV08HM9Wq3W7tuiHEnJvskbdIMPwVyRvY7g27oN5nDe6akpHn6S1q7w9F/vePI7nSsnZEjEFbj3qN/tLxv3e9TMaDvdjzN487By83N9CD9BD1EMxeoIO0As0RCPEUIm+oK/oWzAPgQfg0/nqUFro7mPLljw+SczbOhr</latexit><latexit sha1_base64="G3Q4BqZesyNr1DXJYFyqNrg4cY=">AC3XicdVFbixMxFE7H2zpetquPvgRLoSvD0imCwiIsuA8iBVst9CUksmcacMmkzHJrNPNzqNv4qsPvupf8L/4b8x062VdPBC+j3POd3IuSG4sf3+j1Zw5eq16ze2boa3bt+5u93euTc2qtQMRkwJpScJNSB4DiPLrYBJoYHKRMBRcvy8iR+dgDZc5W/tqoCZpIucZ5xR613z9va73ulZtYuf4WJN5u1Of6+/NnyZxBvSQRsbznda30mqWCkht0xQY6Zxv7AzR7XlTEAdktJAQdkxXcDU05xKMDO37rzGXe9Jca0f7nFa+/fCkelkdQuI49mJZMG7VI20AhMJKiFykeiJFRZbwMoWmxZKzKsqEojZq9FYpYWp8sRmbPZ05nhelhZyd95KVAluFm03hlGtgVqw8oUxzPw5mS6ops36fYUgOwc+r4ZUv/7oATa3SjxyheiFpVfv5FyRq2J/MIfUV0N/K8l9jYkjPM9SflLV08HM9Wq3W7tuiHEnJvskbdIMPwVyRvY7g27oN5nDe6akpHn6S1q7w9F/vePI7nSsnZEjEFbj3qN/tLxv3e9TMaDvdjzN487By83N9CD9BD1EMxeoIO0As0RCPEUIm+oK/oWzAPgQfg0/nqUFro7mPLljw+SczbOhr</latexit><latexit sha1_base64="G3Q4BqZesyNr1DXJYFyqNrg4cY=">AC3XicdVFbixMxFE7H2zpetquPvgRLoSvD0imCwiIsuA8iBVst9CUksmcacMmkzHJrNPNzqNv4qsPvupf8L/4b8x062VdPBC+j3POd3IuSG4sf3+j1Zw5eq16ze2boa3bt+5u93euTc2qtQMRkwJpScJNSB4DiPLrYBJoYHKRMBRcvy8iR+dgDZc5W/tqoCZpIucZ5xR613z9va73ulZtYuf4WJN5u1Of6+/NnyZxBvSQRsbznda30mqWCkht0xQY6Zxv7AzR7XlTEAdktJAQdkxXcDU05xKMDO37rzGXe9Jca0f7nFa+/fCkelkdQuI49mJZMG7VI20AhMJKiFykeiJFRZbwMoWmxZKzKsqEojZq9FYpYWp8sRmbPZ05nhelhZyd95KVAluFm03hlGtgVqw8oUxzPw5mS6ops36fYUgOwc+r4ZUv/7oATa3SjxyheiFpVfv5FyRq2J/MIfUV0N/K8l9jYkjPM9SflLV08HM9Wq3W7tuiHEnJvskbdIMPwVyRvY7g27oN5nDe6akpHn6S1q7w9F/vePI7nSsnZEjEFbj3qN/tLxv3e9TMaDvdjzN487By83N9CD9BD1EMxeoIO0As0RCPEUIm+oK/oWzAPgQfg0/nqUFro7mPLljw+SczbOhr</latexit>
slide-14
SLIDE 14

The Evidence Lower-Bound

CSE 599I: Generative Models University of Washington Lecture 2

  • We can also derive the Evidence Lower-Bound by Jensen’s inequality:
  • Machine Learning community calls this lower bound the ELBO
  • Another way to look at the ELBO:

log pθ(x) = log E

z∼q(·|x)

pθ(x, z) q(z|x)

E

z∼q(·|x)

 log pθ(x, z) q(z|x)

  • .
<latexit sha1_base64="qBhzTcrJ7RNnIgtodHruN+kUKqA=">ADlHiclVLtahQxFJ3p+lHXr1bBP/4JLpWtDGW3CApVKHQLIqhV7AdsliWTuTMbmkymyd06u+m8lo/gO/g2ZqartRYRLwzncHPmXtzExdSWOz1vodLrWvXb9xcvtW+fefuvfsrqw8OrJ4aDvtcS2OYmZBihz2UaCEo8IAU7GEw/h4pz4/PAVjhc4/46yAkWJZLlLBGfrUeOUblTojxZjiBJB1y3VCnr4mTZIqhNduAbj2O1W1djNCbVCkZMu5YnGs3K9ohJSHNLUMO4ufKL5euVOuvOmwohsgiNCMzj5T9emj39b4xXOr2NXhPkKukvSCdYxN54NfxKE82nCnLklk7PcKHDlmUHAJVZtOLRSMH7Mhp7mTIEduebCK7LmMwlJtfFfjqTJ/q5wTNl6vsijnam4RpyoGmqBjSRDKP1JFMcqKq2XQZQZVkwEL6NUaoZRrUetpa3I5WYwfTlyIi+mCDk/7yWdSoKa1AsmiTDAUc48YdwIPw7hE+avEP0zaLfpAPy8Bt5+w8FGIbaPHOUmUyxsvLzZzSq2UXlHvOycA/MSW8x5GjIk8TcVpWw82R61bOr2OtTUinT7doUpdZMQd6Rrc6m2tf5M5fOFaKZYnP6WVG/wy+dt/HM21UZWj8gAMejQN+k3/9zrVXKwudH3/OPzvbxc6Xg8fBk6Ab9IMXwXbwJtgL9gMeRuGncBjS1qPWq9ZOa/e8dClcaB4Gl6L1/gdXCyJ</latexit><latexit sha1_base64="qBhzTcrJ7RNnIgtodHruN+kUKqA=">ADlHiclVLtahQxFJ3p+lHXr1bBP/4JLpWtDGW3CApVKHQLIqhV7AdsliWTuTMbmkymyd06u+m8lo/gO/g2ZqartRYRLwzncHPmXtzExdSWOz1vodLrWvXb9xcvtW+fefuvfsrqw8OrJ4aDvtcS2OYmZBihz2UaCEo8IAU7GEw/h4pz4/PAVjhc4/46yAkWJZLlLBGfrUeOUblTojxZjiBJB1y3VCnr4mTZIqhNduAbj2O1W1djNCbVCkZMu5YnGs3K9ohJSHNLUMO4ufKL5euVOuvOmwohsgiNCMzj5T9emj39b4xXOr2NXhPkKukvSCdYxN54NfxKE82nCnLklk7PcKHDlmUHAJVZtOLRSMH7Mhp7mTIEduebCK7LmMwlJtfFfjqTJ/q5wTNl6vsijnam4RpyoGmqBjSRDKP1JFMcqKq2XQZQZVkwEL6NUaoZRrUetpa3I5WYwfTlyIi+mCDk/7yWdSoKa1AsmiTDAUc48YdwIPw7hE+avEP0zaLfpAPy8Bt5+w8FGIbaPHOUmUyxsvLzZzSq2UXlHvOycA/MSW8x5GjIk8TcVpWw82R61bOr2OtTUinT7doUpdZMQd6Rrc6m2tf5M5fOFaKZYnP6WVG/wy+dt/HM21UZWj8gAMejQN+k3/9zrVXKwudH3/OPzvbxc6Xg8fBk6Ab9IMXwXbwJtgL9gMeRuGncBjS1qPWq9ZOa/e8dClcaB4Gl6L1/gdXCyJ</latexit><latexit sha1_base64="qBhzTcrJ7RNnIgtodHruN+kUKqA=">ADlHiclVLtahQxFJ3p+lHXr1bBP/4JLpWtDGW3CApVKHQLIqhV7AdsliWTuTMbmkymyd06u+m8lo/gO/g2ZqartRYRLwzncHPmXtzExdSWOz1vodLrWvXb9xcvtW+fefuvfsrqw8OrJ4aDvtcS2OYmZBihz2UaCEo8IAU7GEw/h4pz4/PAVjhc4/46yAkWJZLlLBGfrUeOUblTojxZjiBJB1y3VCnr4mTZIqhNduAbj2O1W1djNCbVCkZMu5YnGs3K9ohJSHNLUMO4ufKL5euVOuvOmwohsgiNCMzj5T9emj39b4xXOr2NXhPkKukvSCdYxN54NfxKE82nCnLklk7PcKHDlmUHAJVZtOLRSMH7Mhp7mTIEduebCK7LmMwlJtfFfjqTJ/q5wTNl6vsijnam4RpyoGmqBjSRDKP1JFMcqKq2XQZQZVkwEL6NUaoZRrUetpa3I5WYwfTlyIi+mCDk/7yWdSoKa1AsmiTDAUc48YdwIPw7hE+avEP0zaLfpAPy8Bt5+w8FGIbaPHOUmUyxsvLzZzSq2UXlHvOycA/MSW8x5GjIk8TcVpWw82R61bOr2OtTUinT7doUpdZMQd6Rrc6m2tf5M5fOFaKZYnP6WVG/wy+dt/HM21UZWj8gAMejQN+k3/9zrVXKwudH3/OPzvbxc6Xg8fBk6Ab9IMXwXbwJtgL9gMeRuGncBjS1qPWq9ZOa/e8dClcaB4Gl6L1/gdXCyJ</latexit><latexit sha1_base64="qBhzTcrJ7RNnIgtodHruN+kUKqA=">ADlHiclVLtahQxFJ3p+lHXr1bBP/4JLpWtDGW3CApVKHQLIqhV7AdsliWTuTMbmkymyd06u+m8lo/gO/g2ZqartRYRLwzncHPmXtzExdSWOz1vodLrWvXb9xcvtW+fefuvfsrqw8OrJ4aDvtcS2OYmZBihz2UaCEo8IAU7GEw/h4pz4/PAVjhc4/46yAkWJZLlLBGfrUeOUblTojxZjiBJB1y3VCnr4mTZIqhNduAbj2O1W1djNCbVCkZMu5YnGs3K9ohJSHNLUMO4ufKL5euVOuvOmwohsgiNCMzj5T9emj39b4xXOr2NXhPkKukvSCdYxN54NfxKE82nCnLklk7PcKHDlmUHAJVZtOLRSMH7Mhp7mTIEduebCK7LmMwlJtfFfjqTJ/q5wTNl6vsijnam4RpyoGmqBjSRDKP1JFMcqKq2XQZQZVkwEL6NUaoZRrUetpa3I5WYwfTlyIi+mCDk/7yWdSoKa1AsmiTDAUc48YdwIPw7hE+avEP0zaLfpAPy8Bt5+w8FGIbaPHOUmUyxsvLzZzSq2UXlHvOycA/MSW8x5GjIk8TcVpWw82R61bOr2OtTUinT7doUpdZMQd6Rrc6m2tf5M5fOFaKZYnP6WVG/wy+dt/HM21UZWj8gAMejQN+k3/9zrVXKwudH3/OPzvbxc6Xg8fBk6Ab9IMXwXbwJtgL9gMeRuGncBjS1qPWq9ZOa/e8dClcaB4Gl6L1/gdXCyJ</latexit>

E

z∼q(·|x)

 log pθ(x, z) q(z|x)

  • =

E

z∼q(·|x)

⇥ log pθ(x|z) ⇤ D(q(z|x) k p(z)).

<latexit sha1_base64="qDKSMaXMrzIGxeukaOIupQfQtGc=">ADgnicjVLtahQxFJ1xq9b1q9V/+ie4FLZlLDtVUKhCwQpaECvYD9gMSyZzZzY0mUyTu3W6byV+C6+jZnt1lqLYGA4h5t7TnJzJq2ksDgY/AxvdBZu3rq9eKd7979Bw+Xlh/tWz0xHPa4ltocpsyCFCXsoUAJh5UBplIJB+nRu3b/4ASMFbr8iqcVJIoVpcgFZ+hLo6XvVDEc68rNME3d+6YZuSmhVihy3Kc803hWrzZUQo5DKnVBaG4Yd9WI4hiQ9etoutq4/501mZEMcaEvCX/65uK4tz20vBsutqWE/KcUFHmTi5sHdV3x+2PlrqDdYHs0Wuk3hOesF87Y6Wwx803yioEQumbXDeFBh4phBwSU0XTqxUDF+xAoYeloyBTZxs+dtyIqvZCTXxn8lkln1T4VjyrZTRh7tqUpbxLFqoRXYSDKE2u9Eaqi2noZRIVh1VjwOsqlZhi1etRa2oZcvQzmrxMnymqCUPLzu+QTSVCTNk6SCQMc5aknjBvhxyF8zHxA6EPvduk2+HkNfPL2nyswDLVZc5SZQrG68fMXNGrZecu847Ztv+hlPAeh24eQd0MNxLXb5xPYaVLSC+mzRr26yYAj2jm72Nla5/yRK+ca0UK7MLaeO2f5v86xHS21U46jcB4MezQx90vHfuV4n+xvrsedfXva2duaZLwZPg2dBP4iDV8FW8CHYDfYCHj4Jt8KP4U5nobPWiTsvzltvhHPN4+DK6rz5Be10JWU=</latexit><latexit sha1_base64="qDKSMaXMrzIGxeukaOIupQfQtGc=">ADgnicjVLtahQxFJ1xq9b1q9V/+ie4FLZlLDtVUKhCwQpaECvYD9gMSyZzZzY0mUyTu3W6byV+C6+jZnt1lqLYGA4h5t7TnJzJq2ksDgY/AxvdBZu3rq9eKd7979Bw+Xlh/tWz0xHPa4ltocpsyCFCXsoUAJh5UBplIJB+nRu3b/4ASMFbr8iqcVJIoVpcgFZ+hLo6XvVDEc68rNME3d+6YZuSmhVihy3Kc803hWrzZUQo5DKnVBaG4Yd9WI4hiQ9etoutq4/501mZEMcaEvCX/65uK4tz20vBsutqWE/KcUFHmTi5sHdV3x+2PlrqDdYHs0Wuk3hOesF87Y6Wwx803yioEQumbXDeFBh4phBwSU0XTqxUDF+xAoYeloyBTZxs+dtyIqvZCTXxn8lkln1T4VjyrZTRh7tqUpbxLFqoRXYSDKE2u9Eaqi2noZRIVh1VjwOsqlZhi1etRa2oZcvQzmrxMnymqCUPLzu+QTSVCTNk6SCQMc5aknjBvhxyF8zHxA6EPvduk2+HkNfPL2nyswDLVZc5SZQrG68fMXNGrZecu847Ztv+hlPAeh24eQd0MNxLXb5xPYaVLSC+mzRr26yYAj2jm72Nla5/yRK+ca0UK7MLaeO2f5v86xHS21U46jcB4MezQx90vHfuV4n+xvrsedfXva2duaZLwZPg2dBP4iDV8FW8CHYDfYCHj4Jt8KP4U5nobPWiTsvzltvhHPN4+DK6rz5Be10JWU=</latexit><latexit sha1_base64="qDKSMaXMrzIGxeukaOIupQfQtGc=">ADgnicjVLtahQxFJ1xq9b1q9V/+ie4FLZlLDtVUKhCwQpaECvYD9gMSyZzZzY0mUyTu3W6byV+C6+jZnt1lqLYGA4h5t7TnJzJq2ksDgY/AxvdBZu3rq9eKd7979Bw+Xlh/tWz0xHPa4ltocpsyCFCXsoUAJh5UBplIJB+nRu3b/4ASMFbr8iqcVJIoVpcgFZ+hLo6XvVDEc68rNME3d+6YZuSmhVihy3Kc803hWrzZUQo5DKnVBaG4Yd9WI4hiQ9etoutq4/501mZEMcaEvCX/65uK4tz20vBsutqWE/KcUFHmTi5sHdV3x+2PlrqDdYHs0Wuk3hOesF87Y6Wwx803yioEQumbXDeFBh4phBwSU0XTqxUDF+xAoYeloyBTZxs+dtyIqvZCTXxn8lkln1T4VjyrZTRh7tqUpbxLFqoRXYSDKE2u9Eaqi2noZRIVh1VjwOsqlZhi1etRa2oZcvQzmrxMnymqCUPLzu+QTSVCTNk6SCQMc5aknjBvhxyF8zHxA6EPvduk2+HkNfPL2nyswDLVZc5SZQrG68fMXNGrZecu847Ztv+hlPAeh24eQd0MNxLXb5xPYaVLSC+mzRr26yYAj2jm72Nla5/yRK+ca0UK7MLaeO2f5v86xHS21U46jcB4MezQx90vHfuV4n+xvrsedfXva2duaZLwZPg2dBP4iDV8FW8CHYDfYCHj4Jt8KP4U5nobPWiTsvzltvhHPN4+DK6rz5Be10JWU=</latexit><latexit sha1_base64="qDKSMaXMrzIGxeukaOIupQfQtGc=">ADgnicjVLtahQxFJ1xq9b1q9V/+ie4FLZlLDtVUKhCwQpaECvYD9gMSyZzZzY0mUyTu3W6byV+C6+jZnt1lqLYGA4h5t7TnJzJq2ksDgY/AxvdBZu3rq9eKd7979Bw+Xlh/tWz0xHPa4ltocpsyCFCXsoUAJh5UBplIJB+nRu3b/4ASMFbr8iqcVJIoVpcgFZ+hLo6XvVDEc68rNME3d+6YZuSmhVihy3Kc803hWrzZUQo5DKnVBaG4Yd9WI4hiQ9etoutq4/501mZEMcaEvCX/65uK4tz20vBsutqWE/KcUFHmTi5sHdV3x+2PlrqDdYHs0Wuk3hOesF87Y6Wwx803yioEQumbXDeFBh4phBwSU0XTqxUDF+xAoYeloyBTZxs+dtyIqvZCTXxn8lkln1T4VjyrZTRh7tqUpbxLFqoRXYSDKE2u9Eaqi2noZRIVh1VjwOsqlZhi1etRa2oZcvQzmrxMnymqCUPLzu+QTSVCTNk6SCQMc5aknjBvhxyF8zHxA6EPvduk2+HkNfPL2nyswDLVZc5SZQrG68fMXNGrZecu847Ztv+hlPAeh24eQd0MNxLXb5xPYaVLSC+mzRr26yYAj2jm72Nla5/yRK+ca0UK7MLaeO2f5v86xHS21U46jcB4MezQx90vHfuV4n+xvrsedfXva2duaZLwZPg2dBP4iDV8FW8CHYDfYCHj4Jt8KP4U5nobPWiTsvzltvhHPN4+DK6rz5Be10JWU=</latexit>
slide-15
SLIDE 15

Expectation Maximization

CSE 599I: Generative Models University of Washington Lecture 2

  • Jointly optimize the ELBO over and :
  • Alternating Optimization (Expectation Maximization):

ˆ θmle = arg max

θ

max

q

E

x∼p z∼q(·|x)

 log pθ(x, z) q(z|x)

  • .
<latexit sha1_base64="MKFYofbzWKlahkZph+29oPXfNQ=">ADZ3icdVLrahNBFJ40Xmq8tFUQwT+jIZDKkiZFUChCwQoiBXsBTIhzM7O7g6d2dnMnNRNpvMgvojv4iP4FDqbxEstHli+j3Obc863cSmFhX7/W2Ote36jZvrt1q379y9t7G5df/Y6qlh/Ihpqc1pTC2XouBHIEDy09JwqmLJT+Kz13X85JwbK3TxCWYlHymaFSIVjEJwjTe/kJwCgZwDHRPgFTglucevMKEmU7QaL2OY1HxSA+S6dAuMY/fG+7EjdhpboOzMVZhYoXBJyHzJl3CEg0X1b3RPIUhkTqDJPUObKVfNuFc23vZt053UeMSLYdQb7b7vf7C8FUyWJE2WtnheKvxlSaTRUvgElq7XDQL2HkqAHBwlItMrW8DGPSjA8DLajiduQWN/S4EzwJTrUJXwF4f27wlFl6WjgHam4hohVzXUBTaSNJwvRKI4VlFlQxmPMkPLXLAqSqWmENX1oLW0Hl8eBtKXIyeKcgq8YMtZ0qnEoHGtGU6E4QzkLBDKjAjrYJbTcEIyrZa5ICHfQ1/H9p/KLmhoM0ztxLQh/0zEtXsT+YhDR2Tg/DXKBF6nDoijQR5Uf7o5c17sgR6eFcXtA9khSp1kx5+SC7LV3O61wyYJ/ZlopWiS/Sr07+N3kf+84UmijvCPymBsIaBYlB78q+tVcrzbGwT+8Xl7/91K83X0GD1FXTRAL9A+eosO0RFi6Eej0+g1dta+NzeaD5uPlqlrjVXNA3TJmk9+Al9BHSI=</latexit><latexit sha1_base64="MKFYofbzWKlahkZph+29oPXfNQ=">ADZ3icdVLrahNBFJ40Xmq8tFUQwT+jIZDKkiZFUChCwQoiBXsBTIhzM7O7g6d2dnMnNRNpvMgvojv4iP4FDqbxEstHli+j3Obc863cSmFhX7/W2Ote36jZvrt1q379y9t7G5df/Y6qlh/Ihpqc1pTC2XouBHIEDy09JwqmLJT+Kz13X85JwbK3TxCWYlHymaFSIVjEJwjTe/kJwCgZwDHRPgFTglucevMKEmU7QaL2OY1HxSA+S6dAuMY/fG+7EjdhpboOzMVZhYoXBJyHzJl3CEg0X1b3RPIUhkTqDJPUObKVfNuFc23vZt053UeMSLYdQb7b7vf7C8FUyWJE2WtnheKvxlSaTRUvgElq7XDQL2HkqAHBwlItMrW8DGPSjA8DLajiduQWN/S4EzwJTrUJXwF4f27wlFl6WjgHam4hohVzXUBTaSNJwvRKI4VlFlQxmPMkPLXLAqSqWmENX1oLW0Hl8eBtKXIyeKcgq8YMtZ0qnEoHGtGU6E4QzkLBDKjAjrYJbTcEIyrZa5ICHfQ1/H9p/KLmhoM0ztxLQh/0zEtXsT+YhDR2Tg/DXKBF6nDoijQR5Uf7o5c17sgR6eFcXtA9khSp1kx5+SC7LV3O61wyYJ/ZlopWiS/Sr07+N3kf+84UmijvCPymBsIaBYlB78q+tVcrzbGwT+8Xl7/91K83X0GD1FXTRAL9A+eosO0RFi6Eej0+g1dta+NzeaD5uPlqlrjVXNA3TJmk9+Al9BHSI=</latexit><latexit sha1_base64="MKFYofbzWKlahkZph+29oPXfNQ=">ADZ3icdVLrahNBFJ40Xmq8tFUQwT+jIZDKkiZFUChCwQoiBXsBTIhzM7O7g6d2dnMnNRNpvMgvojv4iP4FDqbxEstHli+j3Obc863cSmFhX7/W2Ote36jZvrt1q379y9t7G5df/Y6qlh/Ihpqc1pTC2XouBHIEDy09JwqmLJT+Kz13X85JwbK3TxCWYlHymaFSIVjEJwjTe/kJwCgZwDHRPgFTglucevMKEmU7QaL2OY1HxSA+S6dAuMY/fG+7EjdhpboOzMVZhYoXBJyHzJl3CEg0X1b3RPIUhkTqDJPUObKVfNuFc23vZt053UeMSLYdQb7b7vf7C8FUyWJE2WtnheKvxlSaTRUvgElq7XDQL2HkqAHBwlItMrW8DGPSjA8DLajiduQWN/S4EzwJTrUJXwF4f27wlFl6WjgHam4hohVzXUBTaSNJwvRKI4VlFlQxmPMkPLXLAqSqWmENX1oLW0Hl8eBtKXIyeKcgq8YMtZ0qnEoHGtGU6E4QzkLBDKjAjrYJbTcEIyrZa5ICHfQ1/H9p/KLmhoM0ztxLQh/0zEtXsT+YhDR2Tg/DXKBF6nDoijQR5Uf7o5c17sgR6eFcXtA9khSp1kx5+SC7LV3O61wyYJ/ZlopWiS/Sr07+N3kf+84UmijvCPymBsIaBYlB78q+tVcrzbGwT+8Xl7/91K83X0GD1FXTRAL9A+eosO0RFi6Eej0+g1dta+NzeaD5uPlqlrjVXNA3TJmk9+Al9BHSI=</latexit><latexit sha1_base64="MKFYofbzWKlahkZph+29oPXfNQ=">ADZ3icdVLrahNBFJ40Xmq8tFUQwT+jIZDKkiZFUChCwQoiBXsBTIhzM7O7g6d2dnMnNRNpvMgvojv4iP4FDqbxEstHli+j3Obc863cSmFhX7/W2Ote36jZvrt1q379y9t7G5df/Y6qlh/Ihpqc1pTC2XouBHIEDy09JwqmLJT+Kz13X85JwbK3TxCWYlHymaFSIVjEJwjTe/kJwCgZwDHRPgFTglucevMKEmU7QaL2OY1HxSA+S6dAuMY/fG+7EjdhpboOzMVZhYoXBJyHzJl3CEg0X1b3RPIUhkTqDJPUObKVfNuFc23vZt053UeMSLYdQb7b7vf7C8FUyWJE2WtnheKvxlSaTRUvgElq7XDQL2HkqAHBwlItMrW8DGPSjA8DLajiduQWN/S4EzwJTrUJXwF4f27wlFl6WjgHam4hohVzXUBTaSNJwvRKI4VlFlQxmPMkPLXLAqSqWmENX1oLW0Hl8eBtKXIyeKcgq8YMtZ0qnEoHGtGU6E4QzkLBDKjAjrYJbTcEIyrZa5ICHfQ1/H9p/KLmhoM0ztxLQh/0zEtXsT+YhDR2Tg/DXKBF6nDoijQR5Uf7o5c17sgR6eFcXtA9khSp1kx5+SC7LV3O61wyYJ/ZlopWiS/Sr07+N3kf+84UmijvCPymBsIaBYlB78q+tVcrzbGwT+8Xl7/91K83X0GD1FXTRAL9A+eosO0RFi6Eej0+g1dta+NzeaD5uPlqlrjVXNA3TJmk9+Al9BHSI=</latexit>

θ

<latexit sha1_base64="UghdNP64m8y7LXZCevN3ALQ2OVY=">AC0nicdVFLixNBEO6Mr3V87erRy2AIrDIsmSAo7GXBHEQI2ygXRYenpqknb7MXRX1mTbOYhXD171h/hf/Df2ZONjXSxovo+q+qrkVdSOx2f7SiK1evXb+xdTO+dfvO3XvbO/dHziwshyE30thxzhxIoWGIAiWMKwtM5RKO8pMXTfzoFKwTRh/iqoKpYjMtSsEZBteI4hyQHW+3u3vdtSWXSbYhbKxwfFO6zstDF8o0Mglc26SdSucemZRcAl1TBcOKsZP2AwmgWqmwE39ut06QRPkZTGhqcxWXv/VnimnGI4TwO6lcobxLlqoBG4VDKEZYikea7SpQsySGeWVXPBl2kpDcO0aMx0tXJxWawfD71QlcLBM3PeykXMkGTNOtJCmGBo1wFwrgVYZyEz5lHMS45j2Icxr4XUo/6YCy9DYJ54yO1NsWYf5ZzRt2J/MAQsVi34kBKhxthToctCnC7rSW/qd2v/uPadOEnaGd2nRZPmxBnQD3S/3evEYZMa3nOjFNPFL2nt+7+L/O8fT7WxqvZUjsBiQLvGcOns37teJqPeXhb426ftg1ebm2+Rh+QR2SUZeUYOyEsyIEPCyTvyhXwl36LD6Cz6GH06T41aG80DcsGiz8BXO3ldg=</latexit><latexit sha1_base64="UghdNP64m8y7LXZCevN3ALQ2OVY=">AC0nicdVFLixNBEO6Mr3V87erRy2AIrDIsmSAo7GXBHEQI2ygXRYenpqknb7MXRX1mTbOYhXD171h/hf/Df2ZONjXSxovo+q+qrkVdSOx2f7SiK1evXb+xdTO+dfvO3XvbO/dHziwshyE30thxzhxIoWGIAiWMKwtM5RKO8pMXTfzoFKwTRh/iqoKpYjMtSsEZBteI4hyQHW+3u3vdtSWXSbYhbKxwfFO6zstDF8o0Mglc26SdSucemZRcAl1TBcOKsZP2AwmgWqmwE39ut06QRPkZTGhqcxWXv/VnimnGI4TwO6lcobxLlqoBG4VDKEZYikea7SpQsySGeWVXPBl2kpDcO0aMx0tXJxWawfD71QlcLBM3PeykXMkGTNOtJCmGBo1wFwrgVYZyEz5lHMS45j2Icxr4XUo/6YCy9DYJ54yO1NsWYf5ZzRt2J/MAQsVi34kBKhxthToctCnC7rSW/qd2v/uPadOEnaGd2nRZPmxBnQD3S/3evEYZMa3nOjFNPFL2nt+7+L/O8fT7WxqvZUjsBiQLvGcOns37teJqPeXhb426ftg1ebm2+Rh+QR2SUZeUYOyEsyIEPCyTvyhXwl36LD6Cz6GH06T41aG80DcsGiz8BXO3ldg=</latexit><latexit sha1_base64="UghdNP64m8y7LXZCevN3ALQ2OVY=">AC0nicdVFLixNBEO6Mr3V87erRy2AIrDIsmSAo7GXBHEQI2ygXRYenpqknb7MXRX1mTbOYhXD171h/hf/Df2ZONjXSxovo+q+qrkVdSOx2f7SiK1evXb+xdTO+dfvO3XvbO/dHziwshyE30thxzhxIoWGIAiWMKwtM5RKO8pMXTfzoFKwTRh/iqoKpYjMtSsEZBteI4hyQHW+3u3vdtSWXSbYhbKxwfFO6zstDF8o0Mglc26SdSucemZRcAl1TBcOKsZP2AwmgWqmwE39ut06QRPkZTGhqcxWXv/VnimnGI4TwO6lcobxLlqoBG4VDKEZYikea7SpQsySGeWVXPBl2kpDcO0aMx0tXJxWawfD71QlcLBM3PeykXMkGTNOtJCmGBo1wFwrgVYZyEz5lHMS45j2Icxr4XUo/6YCy9DYJ54yO1NsWYf5ZzRt2J/MAQsVi34kBKhxthToctCnC7rSW/qd2v/uPadOEnaGd2nRZPmxBnQD3S/3evEYZMa3nOjFNPFL2nt+7+L/O8fT7WxqvZUjsBiQLvGcOns37teJqPeXhb426ftg1ebm2+Rh+QR2SUZeUYOyEsyIEPCyTvyhXwl36LD6Cz6GH06T41aG80DcsGiz8BXO3ldg=</latexit><latexit sha1_base64="UghdNP64m8y7LXZCevN3ALQ2OVY=">AC0nicdVFLixNBEO6Mr3V87erRy2AIrDIsmSAo7GXBHEQI2ygXRYenpqknb7MXRX1mTbOYhXD171h/hf/Df2ZONjXSxovo+q+qrkVdSOx2f7SiK1evXb+xdTO+dfvO3XvbO/dHziwshyE30thxzhxIoWGIAiWMKwtM5RKO8pMXTfzoFKwTRh/iqoKpYjMtSsEZBteI4hyQHW+3u3vdtSWXSbYhbKxwfFO6zstDF8o0Mglc26SdSucemZRcAl1TBcOKsZP2AwmgWqmwE39ut06QRPkZTGhqcxWXv/VnimnGI4TwO6lcobxLlqoBG4VDKEZYikea7SpQsySGeWVXPBl2kpDcO0aMx0tXJxWawfD71QlcLBM3PeykXMkGTNOtJCmGBo1wFwrgVYZyEz5lHMS45j2Icxr4XUo/6YCy9DYJ54yO1NsWYf5ZzRt2J/MAQsVi34kBKhxthToctCnC7rSW/qd2v/uPadOEnaGd2nRZPmxBnQD3S/3evEYZMa3nOjFNPFL2nt+7+L/O8fT7WxqvZUjsBiQLvGcOns37teJqPeXhb426ftg1ebm2+Rh+QR2SUZeUYOyEsyIEPCyTvyhXwl36LD6Cz6GH06T41aG80DcsGiz8BXO3ldg=</latexit>

q

<latexit sha1_base64="GJ8CPraw2VieuoGnQL+0tM23KQ8=">ACzXicdVFbaxNBFJ6st7reWn30ZTAEqiwlGwSFvhTMgwjSFEwayIQyO3s2GTqXdWa2bpyurz74qn/F/+K/cTaNl1o8MHwf5zvzLlkpeDW9fs/OtG16zdu3tq6Hd+5e+/+g+2dhxOrK8NgzLTQZpRC4IrGDvuBExLA1RmAo6z01dt/PgMjOVavXOrEuaSLhQvOKMuI7en2x3+3v9teGrJN2QLtrY6GSn853kmlUSlGOCWjtL+6Wbe2ocZwKamFQWSspO6QJmgSoqwc79utMG94Inx4U24SmH196/FZ5K6lbJgHtSmYtuqVsoRXYRFAHdYgkWSaT2gYZJAtDyVndVITV3S6p3Wwjb4cjOueDn3XJWVA8UueikqgZ3G7WZwzg0wJ1aBUGZ4GAezJTWUubC/OCZDCPMaeBvKH5ZgqNPmSfULCStmzD/giQt+5M5oqFiPgy3kTzUmHrCVZHzs7qZDeZ+t/FPG9+LMe6mZJ/kbZrlH4Gck/3uoBeHTSr4wLSUVOW/pI0f/i7yv38UdrIxhMxAeMCmjWGS6f/3vUqmQz20sCPncP3mxuvoUeoydoF6XoBTpAr9EIjRFDgL6gr+hbdBhV0Xn06SI16mw0j9Aliz7/BMUX40M=</latexit><latexit sha1_base64="GJ8CPraw2VieuoGnQL+0tM23KQ8=">ACzXicdVFbaxNBFJ6st7reWn30ZTAEqiwlGwSFvhTMgwjSFEwayIQyO3s2GTqXdWa2bpyurz74qn/F/+K/cTaNl1o8MHwf5zvzLlkpeDW9fs/OtG16zdu3tq6Hd+5e+/+g+2dhxOrK8NgzLTQZpRC4IrGDvuBExLA1RmAo6z01dt/PgMjOVavXOrEuaSLhQvOKMuI7en2x3+3v9teGrJN2QLtrY6GSn853kmlUSlGOCWjtL+6Wbe2ocZwKamFQWSspO6QJmgSoqwc79utMG94Inx4U24SmH196/FZ5K6lbJgHtSmYtuqVsoRXYRFAHdYgkWSaT2gYZJAtDyVndVITV3S6p3Wwjb4cjOueDn3XJWVA8UueikqgZ3G7WZwzg0wJ1aBUGZ4GAezJTWUubC/OCZDCPMaeBvKH5ZgqNPmSfULCStmzD/giQt+5M5oqFiPgy3kTzUmHrCVZHzs7qZDeZ+t/FPG9+LMe6mZJ/kbZrlH4Gck/3uoBeHTSr4wLSUVOW/pI0f/i7yv38UdrIxhMxAeMCmjWGS6f/3vUqmQz20sCPncP3mxuvoUeoydoF6XoBTpAr9EIjRFDgL6gr+hbdBhV0Xn06SI16mw0j9Aliz7/BMUX40M=</latexit><latexit sha1_base64="GJ8CPraw2VieuoGnQL+0tM23KQ8=">ACzXicdVFbaxNBFJ6st7reWn30ZTAEqiwlGwSFvhTMgwjSFEwayIQyO3s2GTqXdWa2bpyurz74qn/F/+K/cTaNl1o8MHwf5zvzLlkpeDW9fs/OtG16zdu3tq6Hd+5e+/+g+2dhxOrK8NgzLTQZpRC4IrGDvuBExLA1RmAo6z01dt/PgMjOVavXOrEuaSLhQvOKMuI7en2x3+3v9teGrJN2QLtrY6GSn853kmlUSlGOCWjtL+6Wbe2ocZwKamFQWSspO6QJmgSoqwc79utMG94Inx4U24SmH196/FZ5K6lbJgHtSmYtuqVsoRXYRFAHdYgkWSaT2gYZJAtDyVndVITV3S6p3Wwjb4cjOueDn3XJWVA8UueikqgZ3G7WZwzg0wJ1aBUGZ4GAezJTWUubC/OCZDCPMaeBvKH5ZgqNPmSfULCStmzD/giQt+5M5oqFiPgy3kTzUmHrCVZHzs7qZDeZ+t/FPG9+LMe6mZJ/kbZrlH4Gck/3uoBeHTSr4wLSUVOW/pI0f/i7yv38UdrIxhMxAeMCmjWGS6f/3vUqmQz20sCPncP3mxuvoUeoydoF6XoBTpAr9EIjRFDgL6gr+hbdBhV0Xn06SI16mw0j9Aliz7/BMUX40M=</latexit><latexit sha1_base64="GJ8CPraw2VieuoGnQL+0tM23KQ8=">ACzXicdVFbaxNBFJ6st7reWn30ZTAEqiwlGwSFvhTMgwjSFEwayIQyO3s2GTqXdWa2bpyurz74qn/F/+K/cTaNl1o8MHwf5zvzLlkpeDW9fs/OtG16zdu3tq6Hd+5e+/+g+2dhxOrK8NgzLTQZpRC4IrGDvuBExLA1RmAo6z01dt/PgMjOVavXOrEuaSLhQvOKMuI7en2x3+3v9teGrJN2QLtrY6GSn853kmlUSlGOCWjtL+6Wbe2ocZwKamFQWSspO6QJmgSoqwc79utMG94Inx4U24SmH196/FZ5K6lbJgHtSmYtuqVsoRXYRFAHdYgkWSaT2gYZJAtDyVndVITV3S6p3Wwjb4cjOueDn3XJWVA8UueikqgZ3G7WZwzg0wJ1aBUGZ4GAezJTWUubC/OCZDCPMaeBvKH5ZgqNPmSfULCStmzD/giQt+5M5oqFiPgy3kTzUmHrCVZHzs7qZDeZ+t/FPG9+LMe6mZJ/kbZrlH4Gck/3uoBeHTSr4wLSUVOW/pI0f/i7yv38UdrIxhMxAeMCmjWGS6f/3vUqmQz20sCPncP3mxuvoUeoydoF6XoBTpAr9EIjRFDgL6gr+hbdBhV0Xn06SI16mw0j9Aliz7/BMUX40M=</latexit>
  • 1. Fix and optimize the proposal distribution (E-step).
  • 2. Fix the proposal distribution and optimize (M-step).

θ

<latexit sha1_base64="UghdNP64m8y7LXZCevN3ALQ2OVY=">AC0nicdVFLixNBEO6Mr3V87erRy2AIrDIsmSAo7GXBHEQI2ygXRYenpqknb7MXRX1mTbOYhXD171h/hf/Df2ZONjXSxovo+q+qrkVdSOx2f7SiK1evXb+xdTO+dfvO3XvbO/dHziwshyE30thxzhxIoWGIAiWMKwtM5RKO8pMXTfzoFKwTRh/iqoKpYjMtSsEZBteI4hyQHW+3u3vdtSWXSbYhbKxwfFO6zstDF8o0Mglc26SdSucemZRcAl1TBcOKsZP2AwmgWqmwE39ut06QRPkZTGhqcxWXv/VnimnGI4TwO6lcobxLlqoBG4VDKEZYikea7SpQsySGeWVXPBl2kpDcO0aMx0tXJxWawfD71QlcLBM3PeykXMkGTNOtJCmGBo1wFwrgVYZyEz5lHMS45j2Icxr4XUo/6YCy9DYJ54yO1NsWYf5ZzRt2J/MAQsVi34kBKhxthToctCnC7rSW/qd2v/uPadOEnaGd2nRZPmxBnQD3S/3evEYZMa3nOjFNPFL2nt+7+L/O8fT7WxqvZUjsBiQLvGcOns37teJqPeXhb426ftg1ebm2+Rh+QR2SUZeUYOyEsyIEPCyTvyhXwl36LD6Cz6GH06T41aG80DcsGiz8BXO3ldg=</latexit><latexit sha1_base64="UghdNP64m8y7LXZCevN3ALQ2OVY=">AC0nicdVFLixNBEO6Mr3V87erRy2AIrDIsmSAo7GXBHEQI2ygXRYenpqknb7MXRX1mTbOYhXD171h/hf/Df2ZONjXSxovo+q+qrkVdSOx2f7SiK1evXb+xdTO+dfvO3XvbO/dHziwshyE30thxzhxIoWGIAiWMKwtM5RKO8pMXTfzoFKwTRh/iqoKpYjMtSsEZBteI4hyQHW+3u3vdtSWXSbYhbKxwfFO6zstDF8o0Mglc26SdSucemZRcAl1TBcOKsZP2AwmgWqmwE39ut06QRPkZTGhqcxWXv/VnimnGI4TwO6lcobxLlqoBG4VDKEZYikea7SpQsySGeWVXPBl2kpDcO0aMx0tXJxWawfD71QlcLBM3PeykXMkGTNOtJCmGBo1wFwrgVYZyEz5lHMS45j2Icxr4XUo/6YCy9DYJ54yO1NsWYf5ZzRt2J/MAQsVi34kBKhxthToctCnC7rSW/qd2v/uPadOEnaGd2nRZPmxBnQD3S/3evEYZMa3nOjFNPFL2nt+7+L/O8fT7WxqvZUjsBiQLvGcOns37teJqPeXhb426ftg1ebm2+Rh+QR2SUZeUYOyEsyIEPCyTvyhXwl36LD6Cz6GH06T41aG80DcsGiz8BXO3ldg=</latexit><latexit sha1_base64="UghdNP64m8y7LXZCevN3ALQ2OVY=">AC0nicdVFLixNBEO6Mr3V87erRy2AIrDIsmSAo7GXBHEQI2ygXRYenpqknb7MXRX1mTbOYhXD171h/hf/Df2ZONjXSxovo+q+qrkVdSOx2f7SiK1evXb+xdTO+dfvO3XvbO/dHziwshyE30thxzhxIoWGIAiWMKwtM5RKO8pMXTfzoFKwTRh/iqoKpYjMtSsEZBteI4hyQHW+3u3vdtSWXSbYhbKxwfFO6zstDF8o0Mglc26SdSucemZRcAl1TBcOKsZP2AwmgWqmwE39ut06QRPkZTGhqcxWXv/VnimnGI4TwO6lcobxLlqoBG4VDKEZYikea7SpQsySGeWVXPBl2kpDcO0aMx0tXJxWawfD71QlcLBM3PeykXMkGTNOtJCmGBo1wFwrgVYZyEz5lHMS45j2Icxr4XUo/6YCy9DYJ54yO1NsWYf5ZzRt2J/MAQsVi34kBKhxthToctCnC7rSW/qd2v/uPadOEnaGd2nRZPmxBnQD3S/3evEYZMa3nOjFNPFL2nt+7+L/O8fT7WxqvZUjsBiQLvGcOns37teJqPeXhb426ftg1ebm2+Rh+QR2SUZeUYOyEsyIEPCyTvyhXwl36LD6Cz6GH06T41aG80DcsGiz8BXO3ldg=</latexit><latexit sha1_base64="UghdNP64m8y7LXZCevN3ALQ2OVY=">AC0nicdVFLixNBEO6Mr3V87erRy2AIrDIsmSAo7GXBHEQI2ygXRYenpqknb7MXRX1mTbOYhXD171h/hf/Df2ZONjXSxovo+q+qrkVdSOx2f7SiK1evXb+xdTO+dfvO3XvbO/dHziwshyE30thxzhxIoWGIAiWMKwtM5RKO8pMXTfzoFKwTRh/iqoKpYjMtSsEZBteI4hyQHW+3u3vdtSWXSbYhbKxwfFO6zstDF8o0Mglc26SdSucemZRcAl1TBcOKsZP2AwmgWqmwE39ut06QRPkZTGhqcxWXv/VnimnGI4TwO6lcobxLlqoBG4VDKEZYikea7SpQsySGeWVXPBl2kpDcO0aMx0tXJxWawfD71QlcLBM3PeykXMkGTNOtJCmGBo1wFwrgVYZyEz5lHMS45j2Icxr4XUo/6YCy9DYJ54yO1NsWYf5ZzRt2J/MAQsVi34kBKhxthToctCnC7rSW/qd2v/uPadOEnaGd2nRZPmxBnQD3S/3evEYZMa3nOjFNPFL2nt+7+L/O8fT7WxqvZUjsBiQLvGcOns37teJqPeXhb426ftg1ebm2+Rh+QR2SUZeUYOyEsyIEPCyTvyhXwl36LD6Cz6GH06T41aG80DcsGiz8BXO3ldg=</latexit>

q

<latexit sha1_base64="GJ8CPraw2VieuoGnQL+0tM23KQ8=">ACzXicdVFbaxNBFJ6st7reWn30ZTAEqiwlGwSFvhTMgwjSFEwayIQyO3s2GTqXdWa2bpyurz74qn/F/+K/cTaNl1o8MHwf5zvzLlkpeDW9fs/OtG16zdu3tq6Hd+5e+/+g+2dhxOrK8NgzLTQZpRC4IrGDvuBExLA1RmAo6z01dt/PgMjOVavXOrEuaSLhQvOKMuI7en2x3+3v9teGrJN2QLtrY6GSn853kmlUSlGOCWjtL+6Wbe2ocZwKamFQWSspO6QJmgSoqwc79utMG94Inx4U24SmH196/FZ5K6lbJgHtSmYtuqVsoRXYRFAHdYgkWSaT2gYZJAtDyVndVITV3S6p3Wwjb4cjOueDn3XJWVA8UueikqgZ3G7WZwzg0wJ1aBUGZ4GAezJTWUubC/OCZDCPMaeBvKH5ZgqNPmSfULCStmzD/giQt+5M5oqFiPgy3kTzUmHrCVZHzs7qZDeZ+t/FPG9+LMe6mZJ/kbZrlH4Gck/3uoBeHTSr4wLSUVOW/pI0f/i7yv38UdrIxhMxAeMCmjWGS6f/3vUqmQz20sCPncP3mxuvoUeoydoF6XoBTpAr9EIjRFDgL6gr+hbdBhV0Xn06SI16mw0j9Aliz7/BMUX40M=</latexit><latexit sha1_base64="GJ8CPraw2VieuoGnQL+0tM23KQ8=">ACzXicdVFbaxNBFJ6st7reWn30ZTAEqiwlGwSFvhTMgwjSFEwayIQyO3s2GTqXdWa2bpyurz74qn/F/+K/cTaNl1o8MHwf5zvzLlkpeDW9fs/OtG16zdu3tq6Hd+5e+/+g+2dhxOrK8NgzLTQZpRC4IrGDvuBExLA1RmAo6z01dt/PgMjOVavXOrEuaSLhQvOKMuI7en2x3+3v9teGrJN2QLtrY6GSn853kmlUSlGOCWjtL+6Wbe2ocZwKamFQWSspO6QJmgSoqwc79utMG94Inx4U24SmH196/FZ5K6lbJgHtSmYtuqVsoRXYRFAHdYgkWSaT2gYZJAtDyVndVITV3S6p3Wwjb4cjOueDn3XJWVA8UueikqgZ3G7WZwzg0wJ1aBUGZ4GAezJTWUubC/OCZDCPMaeBvKH5ZgqNPmSfULCStmzD/giQt+5M5oqFiPgy3kTzUmHrCVZHzs7qZDeZ+t/FPG9+LMe6mZJ/kbZrlH4Gck/3uoBeHTSr4wLSUVOW/pI0f/i7yv38UdrIxhMxAeMCmjWGS6f/3vUqmQz20sCPncP3mxuvoUeoydoF6XoBTpAr9EIjRFDgL6gr+hbdBhV0Xn06SI16mw0j9Aliz7/BMUX40M=</latexit><latexit sha1_base64="GJ8CPraw2VieuoGnQL+0tM23KQ8=">ACzXicdVFbaxNBFJ6st7reWn30ZTAEqiwlGwSFvhTMgwjSFEwayIQyO3s2GTqXdWa2bpyurz74qn/F/+K/cTaNl1o8MHwf5zvzLlkpeDW9fs/OtG16zdu3tq6Hd+5e+/+g+2dhxOrK8NgzLTQZpRC4IrGDvuBExLA1RmAo6z01dt/PgMjOVavXOrEuaSLhQvOKMuI7en2x3+3v9teGrJN2QLtrY6GSn853kmlUSlGOCWjtL+6Wbe2ocZwKamFQWSspO6QJmgSoqwc79utMG94Inx4U24SmH196/FZ5K6lbJgHtSmYtuqVsoRXYRFAHdYgkWSaT2gYZJAtDyVndVITV3S6p3Wwjb4cjOueDn3XJWVA8UueikqgZ3G7WZwzg0wJ1aBUGZ4GAezJTWUubC/OCZDCPMaeBvKH5ZgqNPmSfULCStmzD/giQt+5M5oqFiPgy3kTzUmHrCVZHzs7qZDeZ+t/FPG9+LMe6mZJ/kbZrlH4Gck/3uoBeHTSr4wLSUVOW/pI0f/i7yv38UdrIxhMxAeMCmjWGS6f/3vUqmQz20sCPncP3mxuvoUeoydoF6XoBTpAr9EIjRFDgL6gr+hbdBhV0Xn06SI16mw0j9Aliz7/BMUX40M=</latexit><latexit sha1_base64="GJ8CPraw2VieuoGnQL+0tM23KQ8=">ACzXicdVFbaxNBFJ6st7reWn30ZTAEqiwlGwSFvhTMgwjSFEwayIQyO3s2GTqXdWa2bpyurz74qn/F/+K/cTaNl1o8MHwf5zvzLlkpeDW9fs/OtG16zdu3tq6Hd+5e+/+g+2dhxOrK8NgzLTQZpRC4IrGDvuBExLA1RmAo6z01dt/PgMjOVavXOrEuaSLhQvOKMuI7en2x3+3v9teGrJN2QLtrY6GSn853kmlUSlGOCWjtL+6Wbe2ocZwKamFQWSspO6QJmgSoqwc79utMG94Inx4U24SmH196/FZ5K6lbJgHtSmYtuqVsoRXYRFAHdYgkWSaT2gYZJAtDyVndVITV3S6p3Wwjb4cjOueDn3XJWVA8UueikqgZ3G7WZwzg0wJ1aBUGZ4GAezJTWUubC/OCZDCPMaeBvKH5ZgqNPmSfULCStmzD/giQt+5M5oqFiPgy3kTzUmHrCVZHzs7qZDeZ+t/FPG9+LMe6mZJ/kbZrlH4Gck/3uoBeHTSr4wLSUVOW/pI0f/i7yv38UdrIxhMxAeMCmjWGS6f/3vUqmQz20sCPncP3mxuvoUeoydoF6XoBTpAr9EIjRFDgL6gr+hbdBhV0Xn06SI16mw0j9Aliz7/BMUX40M=</latexit>

q

<latexit sha1_base64="GJ8CPraw2VieuoGnQL+0tM23KQ8=">ACzXicdVFbaxNBFJ6st7reWn30ZTAEqiwlGwSFvhTMgwjSFEwayIQyO3s2GTqXdWa2bpyurz74qn/F/+K/cTaNl1o8MHwf5zvzLlkpeDW9fs/OtG16zdu3tq6Hd+5e+/+g+2dhxOrK8NgzLTQZpRC4IrGDvuBExLA1RmAo6z01dt/PgMjOVavXOrEuaSLhQvOKMuI7en2x3+3v9teGrJN2QLtrY6GSn853kmlUSlGOCWjtL+6Wbe2ocZwKamFQWSspO6QJmgSoqwc79utMG94Inx4U24SmH196/FZ5K6lbJgHtSmYtuqVsoRXYRFAHdYgkWSaT2gYZJAtDyVndVITV3S6p3Wwjb4cjOueDn3XJWVA8UueikqgZ3G7WZwzg0wJ1aBUGZ4GAezJTWUubC/OCZDCPMaeBvKH5ZgqNPmSfULCStmzD/giQt+5M5oqFiPgy3kTzUmHrCVZHzs7qZDeZ+t/FPG9+LMe6mZJ/kbZrlH4Gck/3uoBeHTSr4wLSUVOW/pI0f/i7yv38UdrIxhMxAeMCmjWGS6f/3vUqmQz20sCPncP3mxuvoUeoydoF6XoBTpAr9EIjRFDgL6gr+hbdBhV0Xn06SI16mw0j9Aliz7/BMUX40M=</latexit><latexit sha1_base64="GJ8CPraw2VieuoGnQL+0tM23KQ8=">ACzXicdVFbaxNBFJ6st7reWn30ZTAEqiwlGwSFvhTMgwjSFEwayIQyO3s2GTqXdWa2bpyurz74qn/F/+K/cTaNl1o8MHwf5zvzLlkpeDW9fs/OtG16zdu3tq6Hd+5e+/+g+2dhxOrK8NgzLTQZpRC4IrGDvuBExLA1RmAo6z01dt/PgMjOVavXOrEuaSLhQvOKMuI7en2x3+3v9teGrJN2QLtrY6GSn853kmlUSlGOCWjtL+6Wbe2ocZwKamFQWSspO6QJmgSoqwc79utMG94Inx4U24SmH196/FZ5K6lbJgHtSmYtuqVsoRXYRFAHdYgkWSaT2gYZJAtDyVndVITV3S6p3Wwjb4cjOueDn3XJWVA8UueikqgZ3G7WZwzg0wJ1aBUGZ4GAezJTWUubC/OCZDCPMaeBvKH5ZgqNPmSfULCStmzD/giQt+5M5oqFiPgy3kTzUmHrCVZHzs7qZDeZ+t/FPG9+LMe6mZJ/kbZrlH4Gck/3uoBeHTSr4wLSUVOW/pI0f/i7yv38UdrIxhMxAeMCmjWGS6f/3vUqmQz20sCPncP3mxuvoUeoydoF6XoBTpAr9EIjRFDgL6gr+hbdBhV0Xn06SI16mw0j9Aliz7/BMUX40M=</latexit><latexit sha1_base64="GJ8CPraw2VieuoGnQL+0tM23KQ8=">ACzXicdVFbaxNBFJ6st7reWn30ZTAEqiwlGwSFvhTMgwjSFEwayIQyO3s2GTqXdWa2bpyurz74qn/F/+K/cTaNl1o8MHwf5zvzLlkpeDW9fs/OtG16zdu3tq6Hd+5e+/+g+2dhxOrK8NgzLTQZpRC4IrGDvuBExLA1RmAo6z01dt/PgMjOVavXOrEuaSLhQvOKMuI7en2x3+3v9teGrJN2QLtrY6GSn853kmlUSlGOCWjtL+6Wbe2ocZwKamFQWSspO6QJmgSoqwc79utMG94Inx4U24SmH196/FZ5K6lbJgHtSmYtuqVsoRXYRFAHdYgkWSaT2gYZJAtDyVndVITV3S6p3Wwjb4cjOueDn3XJWVA8UueikqgZ3G7WZwzg0wJ1aBUGZ4GAezJTWUubC/OCZDCPMaeBvKH5ZgqNPmSfULCStmzD/giQt+5M5oqFiPgy3kTzUmHrCVZHzs7qZDeZ+t/FPG9+LMe6mZJ/kbZrlH4Gck/3uoBeHTSr4wLSUVOW/pI0f/i7yv38UdrIxhMxAeMCmjWGS6f/3vUqmQz20sCPncP3mxuvoUeoydoF6XoBTpAr9EIjRFDgL6gr+hbdBhV0Xn06SI16mw0j9Aliz7/BMUX40M=</latexit><latexit sha1_base64="GJ8CPraw2VieuoGnQL+0tM23KQ8=">ACzXicdVFbaxNBFJ6st7reWn30ZTAEqiwlGwSFvhTMgwjSFEwayIQyO3s2GTqXdWa2bpyurz74qn/F/+K/cTaNl1o8MHwf5zvzLlkpeDW9fs/OtG16zdu3tq6Hd+5e+/+g+2dhxOrK8NgzLTQZpRC4IrGDvuBExLA1RmAo6z01dt/PgMjOVavXOrEuaSLhQvOKMuI7en2x3+3v9teGrJN2QLtrY6GSn853kmlUSlGOCWjtL+6Wbe2ocZwKamFQWSspO6QJmgSoqwc79utMG94Inx4U24SmH196/FZ5K6lbJgHtSmYtuqVsoRXYRFAHdYgkWSaT2gYZJAtDyVndVITV3S6p3Wwjb4cjOueDn3XJWVA8UueikqgZ3G7WZwzg0wJ1aBUGZ4GAezJTWUubC/OCZDCPMaeBvKH5ZgqNPmSfULCStmzD/giQt+5M5oqFiPgy3kTzUmHrCVZHzs7qZDeZ+t/FPG9+LMe6mZJ/kbZrlH4Gck/3uoBeHTSr4wLSUVOW/pI0f/i7yv38UdrIxhMxAeMCmjWGS6f/3vUqmQz20sCPncP3mxuvoUeoydoF6XoBTpAr9EIjRFDgL6gr+hbdBhV0Xn06SI16mw0j9Aliz7/BMUX40M=</latexit>

θ

<latexit sha1_base64="UghdNP64m8y7LXZCevN3ALQ2OVY=">AC0nicdVFLixNBEO6Mr3V87erRy2AIrDIsmSAo7GXBHEQI2ygXRYenpqknb7MXRX1mTbOYhXD171h/hf/Df2ZONjXSxovo+q+qrkVdSOx2f7SiK1evXb+xdTO+dfvO3XvbO/dHziwshyE30thxzhxIoWGIAiWMKwtM5RKO8pMXTfzoFKwTRh/iqoKpYjMtSsEZBteI4hyQHW+3u3vdtSWXSbYhbKxwfFO6zstDF8o0Mglc26SdSucemZRcAl1TBcOKsZP2AwmgWqmwE39ut06QRPkZTGhqcxWXv/VnimnGI4TwO6lcobxLlqoBG4VDKEZYikea7SpQsySGeWVXPBl2kpDcO0aMx0tXJxWawfD71QlcLBM3PeykXMkGTNOtJCmGBo1wFwrgVYZyEz5lHMS45j2Icxr4XUo/6YCy9DYJ54yO1NsWYf5ZzRt2J/MAQsVi34kBKhxthToctCnC7rSW/qd2v/uPadOEnaGd2nRZPmxBnQD3S/3evEYZMa3nOjFNPFL2nt+7+L/O8fT7WxqvZUjsBiQLvGcOns37teJqPeXhb426ftg1ebm2+Rh+QR2SUZeUYOyEsyIEPCyTvyhXwl36LD6Cz6GH06T41aG80DcsGiz8BXO3ldg=</latexit><latexit sha1_base64="UghdNP64m8y7LXZCevN3ALQ2OVY=">AC0nicdVFLixNBEO6Mr3V87erRy2AIrDIsmSAo7GXBHEQI2ygXRYenpqknb7MXRX1mTbOYhXD171h/hf/Df2ZONjXSxovo+q+qrkVdSOx2f7SiK1evXb+xdTO+dfvO3XvbO/dHziwshyE30thxzhxIoWGIAiWMKwtM5RKO8pMXTfzoFKwTRh/iqoKpYjMtSsEZBteI4hyQHW+3u3vdtSWXSbYhbKxwfFO6zstDF8o0Mglc26SdSucemZRcAl1TBcOKsZP2AwmgWqmwE39ut06QRPkZTGhqcxWXv/VnimnGI4TwO6lcobxLlqoBG4VDKEZYikea7SpQsySGeWVXPBl2kpDcO0aMx0tXJxWawfD71QlcLBM3PeykXMkGTNOtJCmGBo1wFwrgVYZyEz5lHMS45j2Icxr4XUo/6YCy9DYJ54yO1NsWYf5ZzRt2J/MAQsVi34kBKhxthToctCnC7rSW/qd2v/uPadOEnaGd2nRZPmxBnQD3S/3evEYZMa3nOjFNPFL2nt+7+L/O8fT7WxqvZUjsBiQLvGcOns37teJqPeXhb426ftg1ebm2+Rh+QR2SUZeUYOyEsyIEPCyTvyhXwl36LD6Cz6GH06T41aG80DcsGiz8BXO3ldg=</latexit><latexit sha1_base64="UghdNP64m8y7LXZCevN3ALQ2OVY=">AC0nicdVFLixNBEO6Mr3V87erRy2AIrDIsmSAo7GXBHEQI2ygXRYenpqknb7MXRX1mTbOYhXD171h/hf/Df2ZONjXSxovo+q+qrkVdSOx2f7SiK1evXb+xdTO+dfvO3XvbO/dHziwshyE30thxzhxIoWGIAiWMKwtM5RKO8pMXTfzoFKwTRh/iqoKpYjMtSsEZBteI4hyQHW+3u3vdtSWXSbYhbKxwfFO6zstDF8o0Mglc26SdSucemZRcAl1TBcOKsZP2AwmgWqmwE39ut06QRPkZTGhqcxWXv/VnimnGI4TwO6lcobxLlqoBG4VDKEZYikea7SpQsySGeWVXPBl2kpDcO0aMx0tXJxWawfD71QlcLBM3PeykXMkGTNOtJCmGBo1wFwrgVYZyEz5lHMS45j2Icxr4XUo/6YCy9DYJ54yO1NsWYf5ZzRt2J/MAQsVi34kBKhxthToctCnC7rSW/qd2v/uPadOEnaGd2nRZPmxBnQD3S/3evEYZMa3nOjFNPFL2nt+7+L/O8fT7WxqvZUjsBiQLvGcOns37teJqPeXhb426ftg1ebm2+Rh+QR2SUZeUYOyEsyIEPCyTvyhXwl36LD6Cz6GH06T41aG80DcsGiz8BXO3ldg=</latexit><latexit sha1_base64="UghdNP64m8y7LXZCevN3ALQ2OVY=">AC0nicdVFLixNBEO6Mr3V87erRy2AIrDIsmSAo7GXBHEQI2ygXRYenpqknb7MXRX1mTbOYhXD171h/hf/Df2ZONjXSxovo+q+qrkVdSOx2f7SiK1evXb+xdTO+dfvO3XvbO/dHziwshyE30thxzhxIoWGIAiWMKwtM5RKO8pMXTfzoFKwTRh/iqoKpYjMtSsEZBteI4hyQHW+3u3vdtSWXSbYhbKxwfFO6zstDF8o0Mglc26SdSucemZRcAl1TBcOKsZP2AwmgWqmwE39ut06QRPkZTGhqcxWXv/VnimnGI4TwO6lcobxLlqoBG4VDKEZYikea7SpQsySGeWVXPBl2kpDcO0aMx0tXJxWawfD71QlcLBM3PeykXMkGTNOtJCmGBo1wFwrgVYZyEz5lHMS45j2Icxr4XUo/6YCy9DYJ54yO1NsWYf5ZzRt2J/MAQsVi34kBKhxthToctCnC7rSW/qd2v/uPadOEnaGd2nRZPmxBnQD3S/3evEYZMa3nOjFNPFL2nt+7+L/O8fT7WxqvZUjsBiQLvGcOns37teJqPeXhb426ftg1ebm2+Rh+QR2SUZeUYOyEsyIEPCyTvyhXwl36LD6Cz6GH06T41aG80DcsGiz8BXO3ldg=</latexit>
slide-16
SLIDE 16

EM for GMMs

CSE 599I: Generative Models University of Washington Lecture 2

  • Sometimes the maximizer of the E-step has an analytic solution.
  • For GMM’s, the E-step is:
  • GMM’s also have an analytical M-step:

pθ(z|x) = pθ(x|z)pθ(z) pθ(x) = πzN(x; µz, Σz) PK

k=1 πkN(x; µk, Σk)

.

<latexit sha1_base64="VNILz5qlmvS6wvOKPmut7Ytgab8=">ADanicdVJtb9MwE5aXkZ568YntC8WpVKLoqpkECqJk2iHxAIKIJ1leoSOY7TWrXjyHZGWjf/hN/Bf+E/8CNwuq7bGJwU3aO7ey539zhMGVW62/3lVq3bt+5u3evdv/Bw0eP6/sHIyUyickJFkzIcYgUYTQhJ5pqRsapJIiHjJyGizdl/vSMSEVF8lUvUzLlaJbQmGKkbSio/0gDqOdEo9ZqnbfBEYCxRNjsovl61b4saRdXMu1iVw5TGqwA5EjPMWLmY9HK+5BnwcqDX+iMo6BkQpXxwCyO/OLbe1AyFv9gLC4Yi3bRCeqNbqe7MXAT+FvQcLY2DPbdnzASOMk0ZghpSZ+N9VTg6SmJGiBjNFUoQXaEYmFiaIEzU1mzMWoGkjEYiFtF+iwSZ6lWEQV+XAnvVqycPS6zkvXUlQHkOa5DbjhSH3cmVpxJtJlM4pzr2YCaS9kq+FYKoA14fR8eupoUmaZLg81nijAEtQCkbiKgkWLOlBQhLatcBeI7s7bUVt1aDA2L3leSDbf8pJRJpIV8YiKQ9ZV7Y/WfQK9Fl5RDZjtHAPhxObY+xgTSJI3qWF5Pe1LQKYxVr1gBo+LAPo7JM0RWBa9hv9Jo1e8mEfMeCc5REF9TCDHZN/vcfAxMhuX0MbESktl5uvFXa/1vXm2DU6/gWf37ZOH631XzPOXSeOS3Hd145x85bZ+icONituC3Xd3uV39WD6tPq4Xlpxd1ynjXrPr8DzLgGuE=</latexit><latexit sha1_base64="VNILz5qlmvS6wvOKPmut7Ytgab8=">ADanicdVJtb9MwE5aXkZ568YntC8WpVKLoqpkECqJk2iHxAIKIJ1leoSOY7TWrXjyHZGWjf/hN/Bf+E/8CNwuq7bGJwU3aO7ey539zhMGVW62/3lVq3bt+5u3evdv/Bw0eP6/sHIyUyickJFkzIcYgUYTQhJ5pqRsapJIiHjJyGizdl/vSMSEVF8lUvUzLlaJbQmGKkbSio/0gDqOdEo9ZqnbfBEYCxRNjsovl61b4saRdXMu1iVw5TGqwA5EjPMWLmY9HK+5BnwcqDX+iMo6BkQpXxwCyO/OLbe1AyFv9gLC4Yi3bRCeqNbqe7MXAT+FvQcLY2DPbdnzASOMk0ZghpSZ+N9VTg6SmJGiBjNFUoQXaEYmFiaIEzU1mzMWoGkjEYiFtF+iwSZ6lWEQV+XAnvVqycPS6zkvXUlQHkOa5DbjhSH3cmVpxJtJlM4pzr2YCaS9kq+FYKoA14fR8eupoUmaZLg81nijAEtQCkbiKgkWLOlBQhLatcBeI7s7bUVt1aDA2L3leSDbf8pJRJpIV8YiKQ9ZV7Y/WfQK9Fl5RDZjtHAPhxObY+xgTSJI3qWF5Pe1LQKYxVr1gBo+LAPo7JM0RWBa9hv9Jo1e8mEfMeCc5REF9TCDHZN/vcfAxMhuX0MbESktl5uvFXa/1vXm2DU6/gWf37ZOH631XzPOXSeOS3Hd145x85bZ+icONituC3Xd3uV39WD6tPq4Xlpxd1ynjXrPr8DzLgGuE=</latexit><latexit sha1_base64="VNILz5qlmvS6wvOKPmut7Ytgab8=">ADanicdVJtb9MwE5aXkZ568YntC8WpVKLoqpkECqJk2iHxAIKIJ1leoSOY7TWrXjyHZGWjf/hN/Bf+E/8CNwuq7bGJwU3aO7ey539zhMGVW62/3lVq3bt+5u3evdv/Bw0eP6/sHIyUyickJFkzIcYgUYTQhJ5pqRsapJIiHjJyGizdl/vSMSEVF8lUvUzLlaJbQmGKkbSio/0gDqOdEo9ZqnbfBEYCxRNjsovl61b4saRdXMu1iVw5TGqwA5EjPMWLmY9HK+5BnwcqDX+iMo6BkQpXxwCyO/OLbe1AyFv9gLC4Yi3bRCeqNbqe7MXAT+FvQcLY2DPbdnzASOMk0ZghpSZ+N9VTg6SmJGiBjNFUoQXaEYmFiaIEzU1mzMWoGkjEYiFtF+iwSZ6lWEQV+XAnvVqycPS6zkvXUlQHkOa5DbjhSH3cmVpxJtJlM4pzr2YCaS9kq+FYKoA14fR8eupoUmaZLg81nijAEtQCkbiKgkWLOlBQhLatcBeI7s7bUVt1aDA2L3leSDbf8pJRJpIV8YiKQ9ZV7Y/WfQK9Fl5RDZjtHAPhxObY+xgTSJI3qWF5Pe1LQKYxVr1gBo+LAPo7JM0RWBa9hv9Jo1e8mEfMeCc5REF9TCDHZN/vcfAxMhuX0MbESktl5uvFXa/1vXm2DU6/gWf37ZOH631XzPOXSeOS3Hd145x85bZ+icONituC3Xd3uV39WD6tPq4Xlpxd1ynjXrPr8DzLgGuE=</latexit><latexit sha1_base64="VNILz5qlmvS6wvOKPmut7Ytgab8=">ADanicdVJtb9MwE5aXkZ568YntC8WpVKLoqpkECqJk2iHxAIKIJ1leoSOY7TWrXjyHZGWjf/hN/Bf+E/8CNwuq7bGJwU3aO7ey539zhMGVW62/3lVq3bt+5u3evdv/Bw0eP6/sHIyUyickJFkzIcYgUYTQhJ5pqRsapJIiHjJyGizdl/vSMSEVF8lUvUzLlaJbQmGKkbSio/0gDqOdEo9ZqnbfBEYCxRNjsovl61b4saRdXMu1iVw5TGqwA5EjPMWLmY9HK+5BnwcqDX+iMo6BkQpXxwCyO/OLbe1AyFv9gLC4Yi3bRCeqNbqe7MXAT+FvQcLY2DPbdnzASOMk0ZghpSZ+N9VTg6SmJGiBjNFUoQXaEYmFiaIEzU1mzMWoGkjEYiFtF+iwSZ6lWEQV+XAnvVqycPS6zkvXUlQHkOa5DbjhSH3cmVpxJtJlM4pzr2YCaS9kq+FYKoA14fR8eupoUmaZLg81nijAEtQCkbiKgkWLOlBQhLatcBeI7s7bUVt1aDA2L3leSDbf8pJRJpIV8YiKQ9ZV7Y/WfQK9Fl5RDZjtHAPhxObY+xgTSJI3qWF5Pe1LQKYxVr1gBo+LAPo7JM0RWBa9hv9Jo1e8mEfMeCc5REF9TCDHZN/vcfAxMhuX0MbESktl5uvFXa/1vXm2DU6/gWf37ZOH631XzPOXSeOS3Hd145x85bZ+icONituC3Xd3uV39WD6tPq4Xlpxd1ynjXrPr8DzLgGuE=</latexit>

π0

k = 1

n

n

X

i=1

pθ(zk|xi), µ0

k =

1 nπ0

k n

X

i=1

pθ(zk|xi)xi, Σ0

k =

1 nπ0

k n

X

i=1

pθ(zk|xi)(xi − µ0

k)⌦2.

<latexit sha1_base64="HS6hKg9FsA8OLECSPZSsKqh8Z7I=">ADrXicnVJtaxQxEN7t+VLPt1Y/+iV4lF5lPW8PRaEUClYQazotYXmumSz2btweVmSbL02zd/yv/jJv+Ls9XxpiwgGknmYzDyZmSd5Jbh1/f63eKl17fqNm8u32rfv3L13f2X1wZ7VtaFsSLXQ5iAnlgmu2NBxJ9hBZRiRuWD7+fR1c79/zIzlWn12JxUbSTJWvOSUOHBlK9xbPpOtpCuDSE+jR4FbCtZeb5VhqOFKoy7CbMke5pNj2bZXwjQXhiK0KZfyFlwLK+nA+U69n0HywLxF94mNJ/ourCwd6is5L2TjyWDsumUWD0MtWOv1ef7QVZAuQCdarN1sNf6KC01ryZSjglh7mPYrN/LEOE4FC21cWwZFT8mYHQJUBF4a+bkSAa2Bp0ClNrCVQ3PvnxmeSCuJmyRg7YnMG+smsjFNgk0EcWwGN0mey2Q2n04yNqSacDpLSqGJS5p8p7WwAV0sxpWvRp6rqnZM0fNaylogp1GjPCq4YdSJEwCEGg7tIDohMGgH/6PdxjsM+jXsPdB/qJghTpsnHhMDqswC9D/GSYN+R+4SYCx24O9JDhwHnNVFvx4Fg4HI98NfiP4tTZCnRv4qIJs/yU4TO82RmstWGSin2hWkqip+pwe/8IvnbOx4rbWTwWOwx48CauQWl08u6XgV7g14K+OPzva7hebL0aPocdSN0uhltB29jXajYUTjN/E0dnHdetYatnDr6Dx0KV7kPIwurNb4B6MuND8=</latexit><latexit sha1_base64="HS6hKg9FsA8OLECSPZSsKqh8Z7I=">ADrXicnVJtaxQxEN7t+VLPt1Y/+iV4lF5lPW8PRaEUClYQazotYXmumSz2btweVmSbL02zd/yv/jJv+Ls9XxpiwgGknmYzDyZmSd5Jbh1/f63eKl17fqNm8u32rfv3L13f2X1wZ7VtaFsSLXQ5iAnlgmu2NBxJ9hBZRiRuWD7+fR1c79/zIzlWn12JxUbSTJWvOSUOHBlK9xbPpOtpCuDSE+jR4FbCtZeb5VhqOFKoy7CbMke5pNj2bZXwjQXhiK0KZfyFlwLK+nA+U69n0HywLxF94mNJ/ourCwd6is5L2TjyWDsumUWD0MtWOv1ef7QVZAuQCdarN1sNf6KC01ryZSjglh7mPYrN/LEOE4FC21cWwZFT8mYHQJUBF4a+bkSAa2Bp0ClNrCVQ3PvnxmeSCuJmyRg7YnMG+smsjFNgk0EcWwGN0mey2Q2n04yNqSacDpLSqGJS5p8p7WwAV0sxpWvRp6rqnZM0fNaylogp1GjPCq4YdSJEwCEGg7tIDohMGgH/6PdxjsM+jXsPdB/qJghTpsnHhMDqswC9D/GSYN+R+4SYCx24O9JDhwHnNVFvx4Fg4HI98NfiP4tTZCnRv4qIJs/yU4TO82RmstWGSin2hWkqip+pwe/8IvnbOx4rbWTwWOwx48CauQWl08u6XgV7g14K+OPzva7hebL0aPocdSN0uhltB29jXajYUTjN/E0dnHdetYatnDr6Dx0KV7kPIwurNb4B6MuND8=</latexit><latexit sha1_base64="HS6hKg9FsA8OLECSPZSsKqh8Z7I=">ADrXicnVJtaxQxEN7t+VLPt1Y/+iV4lF5lPW8PRaEUClYQazotYXmumSz2btweVmSbL02zd/yv/jJv+Ls9XxpiwgGknmYzDyZmSd5Jbh1/f63eKl17fqNm8u32rfv3L13f2X1wZ7VtaFsSLXQ5iAnlgmu2NBxJ9hBZRiRuWD7+fR1c79/zIzlWn12JxUbSTJWvOSUOHBlK9xbPpOtpCuDSE+jR4FbCtZeb5VhqOFKoy7CbMke5pNj2bZXwjQXhiK0KZfyFlwLK+nA+U69n0HywLxF94mNJ/ourCwd6is5L2TjyWDsumUWD0MtWOv1ef7QVZAuQCdarN1sNf6KC01ryZSjglh7mPYrN/LEOE4FC21cWwZFT8mYHQJUBF4a+bkSAa2Bp0ClNrCVQ3PvnxmeSCuJmyRg7YnMG+smsjFNgk0EcWwGN0mey2Q2n04yNqSacDpLSqGJS5p8p7WwAV0sxpWvRp6rqnZM0fNaylogp1GjPCq4YdSJEwCEGg7tIDohMGgH/6PdxjsM+jXsPdB/qJghTpsnHhMDqswC9D/GSYN+R+4SYCx24O9JDhwHnNVFvx4Fg4HI98NfiP4tTZCnRv4qIJs/yU4TO82RmstWGSin2hWkqip+pwe/8IvnbOx4rbWTwWOwx48CauQWl08u6XgV7g14K+OPzva7hebL0aPocdSN0uhltB29jXajYUTjN/E0dnHdetYatnDr6Dx0KV7kPIwurNb4B6MuND8=</latexit><latexit sha1_base64="HS6hKg9FsA8OLECSPZSsKqh8Z7I=">ADrXicnVJtaxQxEN7t+VLPt1Y/+iV4lF5lPW8PRaEUClYQazotYXmumSz2btweVmSbL02zd/yv/jJv+Ls9XxpiwgGknmYzDyZmSd5Jbh1/f63eKl17fqNm8u32rfv3L13f2X1wZ7VtaFsSLXQ5iAnlgmu2NBxJ9hBZRiRuWD7+fR1c79/zIzlWn12JxUbSTJWvOSUOHBlK9xbPpOtpCuDSE+jR4FbCtZeb5VhqOFKoy7CbMke5pNj2bZXwjQXhiK0KZfyFlwLK+nA+U69n0HywLxF94mNJ/ourCwd6is5L2TjyWDsumUWD0MtWOv1ef7QVZAuQCdarN1sNf6KC01ryZSjglh7mPYrN/LEOE4FC21cWwZFT8mYHQJUBF4a+bkSAa2Bp0ClNrCVQ3PvnxmeSCuJmyRg7YnMG+smsjFNgk0EcWwGN0mey2Q2n04yNqSacDpLSqGJS5p8p7WwAV0sxpWvRp6rqnZM0fNaylogp1GjPCq4YdSJEwCEGg7tIDohMGgH/6PdxjsM+jXsPdB/qJghTpsnHhMDqswC9D/GSYN+R+4SYCx24O9JDhwHnNVFvx4Fg4HI98NfiP4tTZCnRv4qIJs/yU4TO82RmstWGSin2hWkqip+pwe/8IvnbOx4rbWTwWOwx48CauQWl08u6XgV7g14K+OPzva7hebL0aPocdSN0uhltB29jXajYUTjN/E0dnHdetYatnDr6Dx0KV7kPIwurNb4B6MuND8=</latexit>
slide-17
SLIDE 17

Preview of Upcoming Topics

CSE 599I: Generative Models University of Washington Lecture 2

  • On Wednesday we’ll start talking about sequence models
  • Next week: deep dive into neural parameterization of sequence models
  • Homework 1 will be posted by the end of the week (due October 26th)
  • The language modeling on HW1 will rely on next week’s lectures