algorithms for nlp
play

Algorithms for NLP CS 11-711 Fall 2020 Lecture 9: CRFs, neural - PowerPoint PPT Presentation

Jul 23, 2023 •327 likes •2.04k views

Algorithms for NLP CS 11-711 Fall 2020 Lecture 9: CRFs, neural sequence labeling Emma Strubell Announcements Project 2 released today after class : sequence labeling. Due: October 16. You will implement part-of-speech taggers for

  1. <latexit sha1_base64="LEiuqxhwrDWuN4pQW8a63QJUygI=">AEvXiclVLdbtMwFE5GgVH+NrjkxlBNa2GbmoENxMTcMHNRJHYD5pL5TgnrTX/RLaztbLyIDwNt/AIvA1O0k3LOpCwFPvk+DvnfMfnizPOjO3f4dLN1o3b91evtO+e+/+g4crq48OjMo1hX2quNJHMTHAmYR9yHo0wDETGHw/jkfXl/eAraMCW/2FkGQ0HGkqWMEutdo9Vwu72GB4Z1cSzcWbGBynNW9NAOwiYXmDPBrBk5sRMV39zei6goAdhOwJICYZoW4WkxUWG2UiUmxObkf8TvfbaoFtnxYIlqIZVBVJNqMwzbrXUegVrkGhcq/7okwiLIidUMLd1/O6veIvidbLTO3/7DG7hF7oZ7TS6W/1q4UWjWhudIL5GoxWlzROFM0FSEs5MeY46md26Ii2jHLw7HIDGaEnZAzH3pREgBm6aroFWvOeBKVK+09aVHkvRzgijJmJ2CPLVzFX70rndXfHuU3fDB2TW5B0rpQmnNkFSqlghKmgVo+8wahmnmuiE6In5n1gvKyuVRmAvwUbLMRKobOpFX1BqVYFM3gad1oG2uQcEaVEQmzx1OiWB8lkBKcm5LMaTn9nXvtZGcszMn+4iJQeLlWZjJgnkFpcbk23PyYWV3sbfwA/IA17nvWnDSxSnsmRI8FmRZ+YGP8FJfmv5BMXiC92WzLVQR8M+W7qAykK2otc2UAx2Ot8qxBeCG+IuoTkNSPocZDM6xGeJVGVzW5aBxsb0Uvt7Y/v+rsvpvrdTl4EjwLukEUvA52g4/BINgPaPg9/BH+DH+13ragxVuyhi6F85jHQWO1zv4AU5+hA=</latexit> <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> <latexit sha1_base64="Hl4SzGDfq/v1/mRkJxB+iyWEgs4=">AH9nicnVTPb9s2FayZG69X0137IVbEFha3cDqCmyXAkW3wy7tPMBpOoSOQFGUTYeUBIpybBD8V3obdt3fstv+kx1HUrJrOU4CTICl5/e+7+P3HgnGBaOlHAz+2dv/5ODw086Dh93Pv/iy68eHT1+V+aVwOQM5ywX72NUEkYzciapZOR9IQjiMSPn8dVPtn4+J6KkeTaSy4KMOZpkNKUYSZOKjg6uidwiqSOgovM/ASQCQmHC0iJW1Cg6HvAgA5TcC1DYPuyU0YTAXCaug7hMO6QtDQA72qBXoX/XZi96SpgY2i4ReFyBcAmncSKfoy1JeZBdIGR3xbrRBrUQVfRbq4H/M4qP29kBudwZ2Lj2PpD8L7KpOxhHfzo2eKfs0QCY103GkZtrPI2m9IlZM0ZpVnzFaip3UG2bI73R4qj2SzesXVtrM32p6NOZ7tdOr+xf6zdo5rvG0v6FyvrjgMki8KHckokAhAnuQSpfTvkdbKNgMZ5VSWposepOZkIDnFiKnftd4p2+vfI2r9jmq359sWhyX1z/tgtL30qAfM2mBzcdBm9KzwR357I9RbDQtTur9n0IjAmKtrU7XfpV7Jrf1wp/rmagtoB6BbmZgM6leKywjbl+KG/k+4EF36NeibgI1atcYthxsT8Sle3p7LA0t0LcI9axS9Oh4cDpwD7gZhE1w7DXPMDraFzDJcVJjFDZXkRDgo5VkhIihnRXViVpED4Ck3IhQkzxEk5Vu5a1ODEZBKQ5sL8MglcdpOhEC/LJY8N0jZSbtdsclftopLpj2NFs6KSJMP1QmnFgMyBvWNBQgXBki1NgLCgxivAU2TGLM1NbDZ6Y5kpYXMi241gPlZl6lZvWYq5bpMXdaNdKEhGrnHOcqS7xRMEadsmZAUVUza/UtX8a59ZM5LcpmdGtJRiTMBZ3QDFGUgntq502n6mE7t2FPxOzQYK8Ma5/LYhAMhfGSX25aLNhE/iNvdf0XUiarZEmbLelnAHTjJ1LXpBM6fr4sbwkMJ6IvCpahm/wnVEjgFKzDTWetGk1wpzScPtM3gzePT8Nvz9/tuL41evm/P6wHvifev5Xuj94L3yfvG3pmHD/4+Pdw73C/s+h86PzR+bOG7u81nK+91tP56z9G0Lzs</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! exp( Ψ ( w , y )) M +1 P ( y | w ) = X Ψ ( w , y ) = ψ ( w , y m , y m − 1 , m ) P exp( Ψ ( w , y 0 )) y 0 ∈ Y ( w ) m =1 y’ is an entire sequence decompose into local scores M +1 X y ) = θ · f ( w , y m , y m − 1 , m ) m =1 4

  2. <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) m =1 5

  3. <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) m =1 Janet will back the bill <s> w w m-2 w m-1 w m w m+1 5

  4. <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) m =1 VB? Janet will back the bill <s> w w m-2 w m-1 w m w m+1 5

  5. <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) m =1 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 VB? Janet will back the bill <s> w w m-2 w m-1 w m w m+1 5

  6. <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) m =1 f ( w , VB, MD, 3) 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 y m-1 MD VB? y NNP Janet will back the bill <s> w w m-2 w m-1 w m w m+1 5

  7. <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) m =1 f ( w , VB, MD, 3) = f ( w 1 , w 2 , w 3 , w 4 , VB, MD) 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 w m = back w m+1 = ache w m-1 = will w m-1 = my y m-1 = MD y m-1 MD VB? y NNP Janet will back the bill <s> w w m-2 w m-1 w m w m+1 5

  8. <latexit sha1_base64="wMAXQnpawC1ge/9hOKMhUb3NFJc=">AExXiclVJb9MwFE5GgRFuGzyYpimtWxUzUACIU2aAESmigSu6C5VI5z0lqznch2ulVWxG/h1/AKj/wbnKSblnUgYanO1+PvnPOdS5Rxpk2v9tfuNK6eu364o3g5q3bd+4uLd/b02muKOzSlKfqICIaOJOwa5jhcJApICLisB8dvS7f9yegNEvlZzPNYCDISLKEUWKcabjsvwxWcV+zNo6EPS42UPmdFh20hbDOBeZMKOHVmyFxVe7sx4WJQGbMRhSIEzj1FQuSXEWYToU5WXFk9D9E51gtd+uo2LBYlTqgSJItRiOMnal0noFLYhoTKvuaRMIiyIGVPC7ZfTvJ3iL4HWykj/XWR2j1X0Ae0jt6iYLi0uv2qoPmQTgDK97s9IfLCwrHKc0FSEM50fow7GVmYIkyjHIoApxryAg9IiM4dFASAXpgqykXaNVZYpSkyv2kQZX1vIclQupiByz7I6+FYaL3s7zE3yYmCZzHIDktaJkpwjk6JyZVDMFDpw4QqpjTiuiYuNkZt1ius+fSjIFPwDQLoWJgdVJlb0iKRNF0PqkLDbACc0FYLI+LHFCRGMT2NISM5NuRTJKb6sXxvxhGV61rqzkBwMThUbMUk4h8Tg8mqa3WdscHUH+A24ASnYcao/ZqCISZVTQtRIkJPCDWyEH+IS/ovJ5BnTwWZthLgin7kmYgbVHvNE814Gik0jxrCJ7zr4S6ACRxY6j50HSrGW5Lw4s7OQ/2Nrvh0+7mp2cr269m+7roPfAeW0v9J572957r+/tetT/7v/wf/q/Wu9aomVak5q64M987nuN0/r2BwZQops=</latexit> <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) m =1 K + F f ( w , VB, MD, 3) = f ( w 1 , w 2 , w 3 , w 4 , VB, MD) 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 w m = back w m+1 = ache w m-1 = will w m-1 = my y m-1 = MD y m-1 MD VB? y NNP Janet will back the bill <s> w w m-2 w m-1 w m w m+1 5

  9. <latexit sha1_base64="wMAXQnpawC1ge/9hOKMhUb3NFJc=">AExXiclVJb9MwFE5GgRFuGzyYpimtWxUzUACIU2aAESmigSu6C5VI5z0lqznch2ulVWxG/h1/AKj/wbnKSblnUgYanO1+PvnPOdS5Rxpk2v9tfuNK6eu364o3g5q3bd+4uLd/b02muKOzSlKfqICIaOJOwa5jhcJApICLisB8dvS7f9yegNEvlZzPNYCDISLKEUWKcabjsvwxWcV+zNo6EPS42UPmdFh20hbDOBeZMKOHVmyFxVe7sx4WJQGbMRhSIEzj1FQuSXEWYToU5WXFk9D9E51gtd+uo2LBYlTqgSJItRiOMnal0noFLYhoTKvuaRMIiyIGVPC7ZfTvJ3iL4HWykj/XWR2j1X0Ae0jt6iYLi0uv2qoPmQTgDK97s9IfLCwrHKc0FSEM50fow7GVmYIkyjHIoApxryAg9IiM4dFASAXpgqykXaNVZYpSkyv2kQZX1vIclQupiByz7I6+FYaL3s7zE3yYmCZzHIDktaJkpwjk6JyZVDMFDpw4QqpjTiuiYuNkZt1ius+fSjIFPwDQLoWJgdVJlb0iKRNF0PqkLDbACc0FYLI+LHFCRGMT2NISM5NuRTJKb6sXxvxhGV61rqzkBwMThUbMUk4h8Tg8mqa3WdscHUH+A24ASnYcao/ZqCISZVTQtRIkJPCDWyEH+IS/ovJ5BnTwWZthLgin7kmYgbVHvNE814Gik0jxrCJ7zr4S6ACRxY6j50HSrGW5Lw4s7OQ/2Nrvh0+7mp2cr269m+7roPfAeW0v9J572957r+/tetT/7v/wf/q/Wu9aomVak5q64M987nuN0/r2BwZQops=</latexit> <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) # labels m =1 K + F f ( w , VB, MD, 3) = f ( w 1 , w 2 , w 3 , w 4 , VB, MD) 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 w m = back w m+1 = ache w m-1 = will w m-1 = my y m-1 = MD y m-1 MD VB? y NNP Janet will back the bill <s> w w m-2 w m-1 w m w m+1 5

  10. <latexit sha1_base64="wMAXQnpawC1ge/9hOKMhUb3NFJc=">AExXiclVJb9MwFE5GgRFuGzyYpimtWxUzUACIU2aAESmigSu6C5VI5z0lqznch2ulVWxG/h1/AKj/wbnKSblnUgYanO1+PvnPOdS5Rxpk2v9tfuNK6eu364o3g5q3bd+4uLd/b02muKOzSlKfqICIaOJOwa5jhcJApICLisB8dvS7f9yegNEvlZzPNYCDISLKEUWKcabjsvwxWcV+zNo6EPS42UPmdFh20hbDOBeZMKOHVmyFxVe7sx4WJQGbMRhSIEzj1FQuSXEWYToU5WXFk9D9E51gtd+uo2LBYlTqgSJItRiOMnal0noFLYhoTKvuaRMIiyIGVPC7ZfTvJ3iL4HWykj/XWR2j1X0Ae0jt6iYLi0uv2qoPmQTgDK97s9IfLCwrHKc0FSEM50fow7GVmYIkyjHIoApxryAg9IiM4dFASAXpgqykXaNVZYpSkyv2kQZX1vIclQupiByz7I6+FYaL3s7zE3yYmCZzHIDktaJkpwjk6JyZVDMFDpw4QqpjTiuiYuNkZt1ius+fSjIFPwDQLoWJgdVJlb0iKRNF0PqkLDbACc0FYLI+LHFCRGMT2NISM5NuRTJKb6sXxvxhGV61rqzkBwMThUbMUk4h8Tg8mqa3WdscHUH+A24ASnYcao/ZqCISZVTQtRIkJPCDWyEH+IS/ovJ5BnTwWZthLgin7kmYgbVHvNE814Gik0jxrCJ7zr4S6ACRxY6j50HSrGW5Lw4s7OQ/2Nrvh0+7mp2cr269m+7roPfAeW0v9J572957r+/tetT/7v/wf/q/Wu9aomVak5q64M987nuN0/r2BwZQops=</latexit> <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) # features # labels m =1 K + F f ( w , VB, MD, 3) = f ( w 1 , w 2 , w 3 , w 4 , VB, MD) 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 w m = back w m+1 = ache w m-1 = will w m-1 = my y m-1 = MD y m-1 MD VB? y NNP Janet will back the bill <s> w w m-2 w m-1 w m w m+1 5

  11. <latexit sha1_base64="PlZBZr6ihz0X20hclBQ9g8VkZw=">AEzHiclVJb9MwFE5GgRFuGzyYpimtWyrmoEL5PGRQhpGhSJXdBcKsc5a3ZTmQ72yor/wWfg2vwL/BSbpWQcSlup8Pf7Od+5RBln2vR6v/25a63rN27O3wpu37l7/7C4oM9neaKwi5NeaoOIqKBMwm7hkOB5kCIiIO+9HRm/J9/xiUZqn8bCYZDAQZSZYwSowzDRf9V8Ey7mvWxpGwJ8UaKr+ToM2Eda5wJwJZvTQis2w+Gp3VsOiJGAzBkMKhGmcmsolKc4jTIaivKxYD90/0QmW+06KhYsRjWtSpAoQi2G06x9lYROYRsSKvOKS8okwoKYMSXcfjnL2yn+EmiljPTfRWYX2DMFtbfXw057G62idx0UDBeWet1edAsCKdgyZue/nBxTuE4pbkAaSgnWh+GvcwMLFGUQ5FgHMNGaFHZASHDkoiQA9sNewCLTtLjJUuZ80qLJe9LBEaD0RkWOWTdKX30rjVW+HuUleDiyTW5A0jpRknNkUlRuDoqZAmr4xAFCFXNaER0TN0Lj9s1+EKaMfBjM1CqBhYnVTZG5IiUTSdT+tCA6xAwglNhSAyfmpxQgTjkxgSknNT7kZyhq/q1p8zDI9bd15SA4Gp4qNmCScQ2JweTXN7jM2uLoD/BbcgBTsONUfM1DEpMopIWokyGnhBjbCj3EJ/8Vk8pzpYLMsWwlwxZR9STOQtqhXm6cacDRSaZ41BM/4V0JdAJK4MdR8aLrVDLel4eWdnAV7G93wWXfj0/OlrdfTfZ3HnlPvLYXei+8Le+91/d2Pep/93/4P/1frQ8t07KtoqbO+VOfh17jtL79ATypCw=</latexit> <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> <latexit sha1_base64="wMAXQnpawC1ge/9hOKMhUb3NFJc=">AExXiclVJb9MwFE5GgRFuGzyYpimtWxUzUACIU2aAESmigSu6C5VI5z0lqznch2ulVWxG/h1/AKj/wbnKSblnUgYanO1+PvnPOdS5Rxpk2v9tfuNK6eu364o3g5q3bd+4uLd/b02muKOzSlKfqICIaOJOwa5jhcJApICLisB8dvS7f9yegNEvlZzPNYCDISLKEUWKcabjsvwxWcV+zNo6EPS42UPmdFh20hbDOBeZMKOHVmyFxVe7sx4WJQGbMRhSIEzj1FQuSXEWYToU5WXFk9D9E51gtd+uo2LBYlTqgSJItRiOMnal0noFLYhoTKvuaRMIiyIGVPC7ZfTvJ3iL4HWykj/XWR2j1X0Ae0jt6iYLi0uv2qoPmQTgDK97s9IfLCwrHKc0FSEM50fow7GVmYIkyjHIoApxryAg9IiM4dFASAXpgqykXaNVZYpSkyv2kQZX1vIclQupiByz7I6+FYaL3s7zE3yYmCZzHIDktaJkpwjk6JyZVDMFDpw4QqpjTiuiYuNkZt1ius+fSjIFPwDQLoWJgdVJlb0iKRNF0PqkLDbACc0FYLI+LHFCRGMT2NISM5NuRTJKb6sXxvxhGV61rqzkBwMThUbMUk4h8Tg8mqa3WdscHUH+A24ASnYcao/ZqCISZVTQtRIkJPCDWyEH+IS/ovJ5BnTwWZthLgin7kmYgbVHvNE814Gik0jxrCJ7zr4S6ACRxY6j50HSrGW5Lw4s7OQ/2Nrvh0+7mp2cr269m+7roPfAeW0v9J572957r+/tetT/7v/wf/q/Wu9aomVak5q64M987nuN0/r2BwZQops=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) # features # labels m =1 ( K − 1)( K + F ) K + F f ( w , VB, MD, 3) = f ( w 1 , w 2 , w 3 , w 4 , VB, MD) 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 w m = back w m+1 = ache w m-1 = will w m-1 = my y m-1 = MD y m-1 MD VB? y NNP Janet will back the bill <s> w w m-2 w m-1 w m w m+1 5

  12. <latexit sha1_base64="Hl4SzGDfq/v1/mRkJxB+iyWEgs4=">AH9nicnVTPb9s2FayZG69X0137IVbEFha3cDqCmyXAkW3wy7tPMBpOoSOQFGUTYeUBIpybBD8V3obdt3fstv+kx1HUrJrOU4CTICl5/e+7+P3HgnGBaOlHAz+2dv/5ODw086Dh93Pv/iy68eHT1+V+aVwOQM5ywX72NUEkYzciapZOR9IQjiMSPn8dVPtn4+J6KkeTaSy4KMOZpkNKUYSZOKjg6uidwiqSOgovM/ASQCQmHC0iJW1Cg6HvAgA5TcC1DYPuyU0YTAXCaug7hMO6QtDQA72qBXoX/XZi96SpgY2i4ReFyBcAmncSKfoy1JeZBdIGR3xbrRBrUQVfRbq4H/M4qP29kBudwZ2Lj2PpD8L7KpOxhHfzo2eKfs0QCY103GkZtrPI2m9IlZM0ZpVnzFaip3UG2bI73R4qj2SzesXVtrM32p6NOZ7tdOr+xf6zdo5rvG0v6FyvrjgMki8KHckokAhAnuQSpfTvkdbKNgMZ5VSWposepOZkIDnFiKnftd4p2+vfI2r9jmq359sWhyX1z/tgtL30qAfM2mBzcdBm9KzwR357I9RbDQtTur9n0IjAmKtrU7XfpV7Jrf1wp/rmagtoB6BbmZgM6leKywjbl+KG/k+4EF36NeibgI1atcYthxsT8Sle3p7LA0t0LcI9axS9Oh4cDpwD7gZhE1w7DXPMDraFzDJcVJjFDZXkRDgo5VkhIihnRXViVpED4Ck3IhQkzxEk5Vu5a1ODEZBKQ5sL8MglcdpOhEC/LJY8N0jZSbtdsclftopLpj2NFs6KSJMP1QmnFgMyBvWNBQgXBki1NgLCgxivAU2TGLM1NbDZ6Y5kpYXMi241gPlZl6lZvWYq5bpMXdaNdKEhGrnHOcqS7xRMEadsmZAUVUza/UtX8a59ZM5LcpmdGtJRiTMBZ3QDFGUgntq502n6mE7t2FPxOzQYK8Ma5/LYhAMhfGSX25aLNhE/iNvdf0XUiarZEmbLelnAHTjJ1LXpBM6fr4sbwkMJ6IvCpahm/wnVEjgFKzDTWetGk1wpzScPtM3gzePT8Nvz9/tuL41evm/P6wHvifev5Xuj94L3yfvG3pmHD/4+Pdw73C/s+h86PzR+bOG7u81nK+91tP56z9G0Lzs</latexit> Conditional random fields (CRFs) exp( Ψ ( w , y )) ■ Decoding: Direct application of Viterbi! P ( y | w ) = P exp( Ψ ( w , y 0 )) y 0 ∈ Y ( w ) 6

  13. <latexit sha1_base64="Hl4SzGDfq/v1/mRkJxB+iyWEgs4=">AH9nicnVTPb9s2FayZG69X0137IVbEFha3cDqCmyXAkW3wy7tPMBpOoSOQFGUTYeUBIpybBD8V3obdt3fstv+kx1HUrJrOU4CTICl5/e+7+P3HgnGBaOlHAz+2dv/5ODw086Dh93Pv/iy68eHT1+V+aVwOQM5ywX72NUEkYzciapZOR9IQjiMSPn8dVPtn4+J6KkeTaSy4KMOZpkNKUYSZOKjg6uidwiqSOgovM/ASQCQmHC0iJW1Cg6HvAgA5TcC1DYPuyU0YTAXCaug7hMO6QtDQA72qBXoX/XZi96SpgY2i4ReFyBcAmncSKfoy1JeZBdIGR3xbrRBrUQVfRbq4H/M4qP29kBudwZ2Lj2PpD8L7KpOxhHfzo2eKfs0QCY103GkZtrPI2m9IlZM0ZpVnzFaip3UG2bI73R4qj2SzesXVtrM32p6NOZ7tdOr+xf6zdo5rvG0v6FyvrjgMki8KHckokAhAnuQSpfTvkdbKNgMZ5VSWposepOZkIDnFiKnftd4p2+vfI2r9jmq359sWhyX1z/tgtL30qAfM2mBzcdBm9KzwR357I9RbDQtTur9n0IjAmKtrU7XfpV7Jrf1wp/rmagtoB6BbmZgM6leKywjbl+KG/k+4EF36NeibgI1atcYthxsT8Sle3p7LA0t0LcI9axS9Oh4cDpwD7gZhE1w7DXPMDraFzDJcVJjFDZXkRDgo5VkhIihnRXViVpED4Ck3IhQkzxEk5Vu5a1ODEZBKQ5sL8MglcdpOhEC/LJY8N0jZSbtdsclftopLpj2NFs6KSJMP1QmnFgMyBvWNBQgXBki1NgLCgxivAU2TGLM1NbDZ6Y5kpYXMi241gPlZl6lZvWYq5bpMXdaNdKEhGrnHOcqS7xRMEadsmZAUVUza/UtX8a59ZM5LcpmdGtJRiTMBZ3QDFGUgntq502n6mE7t2FPxOzQYK8Ma5/LYhAMhfGSX25aLNhE/iNvdf0XUiarZEmbLelnAHTjJ1LXpBM6fr4sbwkMJ6IvCpahm/wnVEjgFKzDTWetGk1wpzScPtM3gzePT8Nvz9/tuL41evm/P6wHvifev5Xuj94L3yfvG3pmHD/4+Pdw73C/s+h86PzR+bOG7u81nK+91tP56z9G0Lzs</latexit> Conditional random fields (CRFs) exp( Ψ ( w , y )) ■ Decoding: Direct application of Viterbi! P ( y | w ) = P exp( Ψ ( w , y 0 )) y 0 ∈ Y ( w ) 6

  14. <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="Hl4SzGDfq/v1/mRkJxB+iyWEgs4=">AH9nicnVTPb9s2FayZG69X0137IVbEFha3cDqCmyXAkW3wy7tPMBpOoSOQFGUTYeUBIpybBD8V3obdt3fstv+kx1HUrJrOU4CTICl5/e+7+P3HgnGBaOlHAz+2dv/5ODw086Dh93Pv/iy68eHT1+V+aVwOQM5ywX72NUEkYzciapZOR9IQjiMSPn8dVPtn4+J6KkeTaSy4KMOZpkNKUYSZOKjg6uidwiqSOgovM/ASQCQmHC0iJW1Cg6HvAgA5TcC1DYPuyU0YTAXCaug7hMO6QtDQA72qBXoX/XZi96SpgY2i4ReFyBcAmncSKfoy1JeZBdIGR3xbrRBrUQVfRbq4H/M4qP29kBudwZ2Lj2PpD8L7KpOxhHfzo2eKfs0QCY103GkZtrPI2m9IlZM0ZpVnzFaip3UG2bI73R4qj2SzesXVtrM32p6NOZ7tdOr+xf6zdo5rvG0v6FyvrjgMki8KHckokAhAnuQSpfTvkdbKNgMZ5VSWposepOZkIDnFiKnftd4p2+vfI2r9jmq359sWhyX1z/tgtL30qAfM2mBzcdBm9KzwR357I9RbDQtTur9n0IjAmKtrU7XfpV7Jrf1wp/rmagtoB6BbmZgM6leKywjbl+KG/k+4EF36NeibgI1atcYthxsT8Sle3p7LA0t0LcI9axS9Oh4cDpwD7gZhE1w7DXPMDraFzDJcVJjFDZXkRDgo5VkhIihnRXViVpED4Ck3IhQkzxEk5Vu5a1ODEZBKQ5sL8MglcdpOhEC/LJY8N0jZSbtdsclftopLpj2NFs6KSJMP1QmnFgMyBvWNBQgXBki1NgLCgxivAU2TGLM1NbDZ6Y5kpYXMi241gPlZl6lZvWYq5bpMXdaNdKEhGrnHOcqS7xRMEadsmZAUVUza/UtX8a59ZM5LcpmdGtJRiTMBZ3QDFGUgntq502n6mE7t2FPxOzQYK8Ma5/LYhAMhfGSX25aLNhE/iNvdf0XUiarZEmbLelnAHTjJ1LXpBM6fr4sbwkMJ6IvCpahm/wnVEjgFKzDTWetGk1wpzScPtM3gzePT8Nvz9/tuL41evm/P6wHvifev5Xuj94L3yfvG3pmHD/4+Pdw73C/s+h86PzR+bOG7u81nK+91tP56z9G0Lzs</latexit> Conditional random fields (CRFs) exp( Ψ ( w , y )) ■ Decoding: Direct application of Viterbi! P ( y | w ) = P exp( Ψ ( w , y 0 )) y 0 ∈ Y ( w ) ˆ y = argmax log P ( y | w ) y 6

  15. <latexit sha1_base64="Hl4SzGDfq/v1/mRkJxB+iyWEgs4=">AH9nicnVTPb9s2FayZG69X0137IVbEFha3cDqCmyXAkW3wy7tPMBpOoSOQFGUTYeUBIpybBD8V3obdt3fstv+kx1HUrJrOU4CTICl5/e+7+P3HgnGBaOlHAz+2dv/5ODw086Dh93Pv/iy68eHT1+V+aVwOQM5ywX72NUEkYzciapZOR9IQjiMSPn8dVPtn4+J6KkeTaSy4KMOZpkNKUYSZOKjg6uidwiqSOgovM/ASQCQmHC0iJW1Cg6HvAgA5TcC1DYPuyU0YTAXCaug7hMO6QtDQA72qBXoX/XZi96SpgY2i4ReFyBcAmncSKfoy1JeZBdIGR3xbrRBrUQVfRbq4H/M4qP29kBudwZ2Lj2PpD8L7KpOxhHfzo2eKfs0QCY103GkZtrPI2m9IlZM0ZpVnzFaip3UG2bI73R4qj2SzesXVtrM32p6NOZ7tdOr+xf6zdo5rvG0v6FyvrjgMki8KHckokAhAnuQSpfTvkdbKNgMZ5VSWposepOZkIDnFiKnftd4p2+vfI2r9jmq359sWhyX1z/tgtL30qAfM2mBzcdBm9KzwR357I9RbDQtTur9n0IjAmKtrU7XfpV7Jrf1wp/rmagtoB6BbmZgM6leKywjbl+KG/k+4EF36NeibgI1atcYthxsT8Sle3p7LA0t0LcI9axS9Oh4cDpwD7gZhE1w7DXPMDraFzDJcVJjFDZXkRDgo5VkhIihnRXViVpED4Ck3IhQkzxEk5Vu5a1ODEZBKQ5sL8MglcdpOhEC/LJY8N0jZSbtdsclftopLpj2NFs6KSJMP1QmnFgMyBvWNBQgXBki1NgLCgxivAU2TGLM1NbDZ6Y5kpYXMi241gPlZl6lZvWYq5bpMXdaNdKEhGrnHOcqS7xRMEadsmZAUVUza/UtX8a59ZM5LcpmdGtJRiTMBZ3QDFGUgntq502n6mE7t2FPxOzQYK8Ma5/LYhAMhfGSX25aLNhE/iNvdf0XUiarZEmbLelnAHTjJ1LXpBM6fr4sbwkMJ6IvCpahm/wnVEjgFKzDTWetGk1wpzScPtM3gzePT8Nvz9/tuL41evm/P6wHvifev5Xuj94L3yfvG3pmHD/4+Pdw73C/s+h86PzR+bOG7u81nK+91tP56z9G0Lzs</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="DypLI6QWGLOj7fmyWz0DYpUsnmg=">AFfXiclVNdb9MwFM3KCqN8bfDIi2Gq1rKuagbSeJk0AUJI0SR2JfmUjmO01qz48h2tlWxSv8RP4GfwCcpO2atSDNUpyb63Pf6JkgYVbrT+bVUubNcvXtv5X7twcNHj5+srj09UiKVmBxiwYQ8CZAijMbkUFPNyEkiCeIBI8fB+fvs/PiCSEVF/FWPEtLjaBDTiGKknau/tvS7VodRsw4ObStkD2Htkm2AVQpRwyqlWfcN3fvNHGz6NgNAPSQaWQBxKHQeEtkpw6jPs83wLd98Wat3m0UrJDTEBSwPEkETaQXCWNRKa1pQk5O4Nl5TGAHKkhxgxczrJ27T/INrImG5dZDKDni+osb/lNxv7YBN8bALHPUS6kGdtxonkgKOr/tQFmRiASRPAbBdqi9ATpaOx0qxbWznJrftxzbVxTdZfXe+0O/kC84Y/Nta98er21yoShgKnMQaM6TUmd9JdM8gqSlmxNZgqkiC8DkakDNnxogT1TP5dFpQd54QREK6J9Yg985GMSVGvHAIbMq1M2zLno7CzV0dueoXGSahLjIlGUMqAFyEYdhFQSrNnIGQhL6rQCPERu5rT7IdytzaQZEnZBdLkQzHtGRXn2kqSA23LwVFoDUoSk0sOEdx+MrACHKRiGJUMp0NszRxF7Ur1Z4QRM1bt2UkhENhaQDGiPGSKRhtpXd7jXUMN9r8ANxFyTJgVP9OSESaSGdkmLErLuwAXwBM/N/SBpPkc4sl2VyAa6YrC8iIbGxewxoQgMBlKkSUnwXHwu1BGgyF1DgSflsALhptS/OZPzxtF23/d3v7yZn3v3XheV7zn3kuv4fnejrfnfK63qGHK6eV75UflZ/Lf6r1aqvaLqCVpXHM6+0qjt/AWSk5PE=</latexit> Conditional random fields (CRFs) exp( Ψ ( w , y )) ■ Decoding: Direct application of Viterbi! P ( y | w ) = P exp( Ψ ( w , y 0 )) y 0 ∈ Y ( w ) y = argmax ˆ log P ( y | w ) y X exp Ψ ( y 0 , w ) = argmax Ψ ( y , w ) − log y y 0 ∈ Y ( w ) 6

  16. <latexit sha1_base64="Hl4SzGDfq/v1/mRkJxB+iyWEgs4=">AH9nicnVTPb9s2FayZG69X0137IVbEFha3cDqCmyXAkW3wy7tPMBpOoSOQFGUTYeUBIpybBD8V3obdt3fstv+kx1HUrJrOU4CTICl5/e+7+P3HgnGBaOlHAz+2dv/5ODw086Dh93Pv/iy68eHT1+V+aVwOQM5ywX72NUEkYzciapZOR9IQjiMSPn8dVPtn4+J6KkeTaSy4KMOZpkNKUYSZOKjg6uidwiqSOgovM/ASQCQmHC0iJW1Cg6HvAgA5TcC1DYPuyU0YTAXCaug7hMO6QtDQA72qBXoX/XZi96SpgY2i4ReFyBcAmncSKfoy1JeZBdIGR3xbrRBrUQVfRbq4H/M4qP29kBudwZ2Lj2PpD8L7KpOxhHfzo2eKfs0QCY103GkZtrPI2m9IlZM0ZpVnzFaip3UG2bI73R4qj2SzesXVtrM32p6NOZ7tdOr+xf6zdo5rvG0v6FyvrjgMki8KHckokAhAnuQSpfTvkdbKNgMZ5VSWposepOZkIDnFiKnftd4p2+vfI2r9jmq359sWhyX1z/tgtL30qAfM2mBzcdBm9KzwR357I9RbDQtTur9n0IjAmKtrU7XfpV7Jrf1wp/rmagtoB6BbmZgM6leKywjbl+KG/k+4EF36NeibgI1atcYthxsT8Sle3p7LA0t0LcI9axS9Oh4cDpwD7gZhE1w7DXPMDraFzDJcVJjFDZXkRDgo5VkhIihnRXViVpED4Ck3IhQkzxEk5Vu5a1ODEZBKQ5sL8MglcdpOhEC/LJY8N0jZSbtdsclftopLpj2NFs6KSJMP1QmnFgMyBvWNBQgXBki1NgLCgxivAU2TGLM1NbDZ6Y5kpYXMi241gPlZl6lZvWYq5bpMXdaNdKEhGrnHOcqS7xRMEadsmZAUVUza/UtX8a59ZM5LcpmdGtJRiTMBZ3QDFGUgntq502n6mE7t2FPxOzQYK8Ma5/LYhAMhfGSX25aLNhE/iNvdf0XUiarZEmbLelnAHTjJ1LXpBM6fr4sbwkMJ6IvCpahm/wnVEjgFKzDTWetGk1wpzScPtM3gzePT8Nvz9/tuL41evm/P6wHvifev5Xuj94L3yfvG3pmHD/4+Pdw73C/s+h86PzR+bOG7u81nK+91tP56z9G0Lzs</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="DypLI6QWGLOj7fmyWz0DYpUsnmg=">AFfXiclVNdb9MwFM3KCqN8bfDIi2Gq1rKuagbSeJk0AUJI0SR2JfmUjmO01qz48h2tlWxSv8RP4GfwCcpO2atSDNUpyb63Pf6JkgYVbrT+bVUubNcvXtv5X7twcNHj5+srj09UiKVmBxiwYQ8CZAijMbkUFPNyEkiCeIBI8fB+fvs/PiCSEVF/FWPEtLjaBDTiGKknau/tvS7VodRsw4ObStkD2Htkm2AVQpRwyqlWfcN3fvNHGz6NgNAPSQaWQBxKHQeEtkpw6jPs83wLd98Wat3m0UrJDTEBSwPEkETaQXCWNRKa1pQk5O4Nl5TGAHKkhxgxczrJ27T/INrImG5dZDKDni+osb/lNxv7YBN8bALHPUS6kGdtxonkgKOr/tQFmRiASRPAbBdqi9ATpaOx0qxbWznJrftxzbVxTdZfXe+0O/kC84Y/Nta98er21yoShgKnMQaM6TUmd9JdM8gqSlmxNZgqkiC8DkakDNnxogT1TP5dFpQd54QREK6J9Yg985GMSVGvHAIbMq1M2zLno7CzV0dueoXGSahLjIlGUMqAFyEYdhFQSrNnIGQhL6rQCPERu5rT7IdytzaQZEnZBdLkQzHtGRXn2kqSA23LwVFoDUoSk0sOEdx+MrACHKRiGJUMp0NszRxF7Ur1Z4QRM1bt2UkhENhaQDGiPGSKRhtpXd7jXUMN9r8ANxFyTJgVP9OSESaSGdkmLErLuwAXwBM/N/SBpPkc4sl2VyAa6YrC8iIbGxewxoQgMBlKkSUnwXHwu1BGgyF1DgSflsALhptS/OZPzxtF23/d3v7yZn3v3XheV7zn3kuv4fnejrfnfK63qGHK6eV75UflZ/Lf6r1aqvaLqCVpXHM6+0qjt/AWSk5PE=</latexit> Conditional random fields (CRFs) exp( Ψ ( w , y )) ■ Decoding: Direct application of Viterbi! P ( y | w ) = P exp( Ψ ( w , y 0 )) y 0 ∈ Y ( w ) y = argmax ˆ log P ( y | w ) y X exp Ψ ( y 0 , w ) = argmax Ψ ( y , w ) − log y y 0 ∈ Y ( w ) 6

  17. <latexit sha1_base64="DypLI6QWGLOj7fmyWz0DYpUsnmg=">AFfXiclVNdb9MwFM3KCqN8bfDIi2Gq1rKuagbSeJk0AUJI0SR2JfmUjmO01qz48h2tlWxSv8RP4GfwCcpO2atSDNUpyb63Pf6JkgYVbrT+bVUubNcvXtv5X7twcNHj5+srj09UiKVmBxiwYQ8CZAijMbkUFPNyEkiCeIBI8fB+fvs/PiCSEVF/FWPEtLjaBDTiGKknau/tvS7VodRsw4ObStkD2Htkm2AVQpRwyqlWfcN3fvNHGz6NgNAPSQaWQBxKHQeEtkpw6jPs83wLd98Wat3m0UrJDTEBSwPEkETaQXCWNRKa1pQk5O4Nl5TGAHKkhxgxczrJ27T/INrImG5dZDKDni+osb/lNxv7YBN8bALHPUS6kGdtxonkgKOr/tQFmRiASRPAbBdqi9ATpaOx0qxbWznJrftxzbVxTdZfXe+0O/kC84Y/Nta98er21yoShgKnMQaM6TUmd9JdM8gqSlmxNZgqkiC8DkakDNnxogT1TP5dFpQd54QREK6J9Yg985GMSVGvHAIbMq1M2zLno7CzV0dueoXGSahLjIlGUMqAFyEYdhFQSrNnIGQhL6rQCPERu5rT7IdytzaQZEnZBdLkQzHtGRXn2kqSA23LwVFoDUoSk0sOEdx+MrACHKRiGJUMp0NszRxF7Ur1Z4QRM1bt2UkhENhaQDGiPGSKRhtpXd7jXUMN9r8ANxFyTJgVP9OSESaSGdkmLErLuwAXwBM/N/SBpPkc4sl2VyAa6YrC8iIbGxewxoQgMBlKkSUnwXHwu1BGgyF1DgSflsALhptS/OZPzxtF23/d3v7yZn3v3XheV7zn3kuv4fnejrfnfK63qGHK6eV75UflZ/Lf6r1aqvaLqCVpXHM6+0qjt/AWSk5PE=</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="Hl4SzGDfq/v1/mRkJxB+iyWEgs4=">AH9nicnVTPb9s2FayZG69X0137IVbEFha3cDqCmyXAkW3wy7tPMBpOoSOQFGUTYeUBIpybBD8V3obdt3fstv+kx1HUrJrOU4CTICl5/e+7+P3HgnGBaOlHAz+2dv/5ODw086Dh93Pv/iy68eHT1+V+aVwOQM5ywX72NUEkYzciapZOR9IQjiMSPn8dVPtn4+J6KkeTaSy4KMOZpkNKUYSZOKjg6uidwiqSOgovM/ASQCQmHC0iJW1Cg6HvAgA5TcC1DYPuyU0YTAXCaug7hMO6QtDQA72qBXoX/XZi96SpgY2i4ReFyBcAmncSKfoy1JeZBdIGR3xbrRBrUQVfRbq4H/M4qP29kBudwZ2Lj2PpD8L7KpOxhHfzo2eKfs0QCY103GkZtrPI2m9IlZM0ZpVnzFaip3UG2bI73R4qj2SzesXVtrM32p6NOZ7tdOr+xf6zdo5rvG0v6FyvrjgMki8KHckokAhAnuQSpfTvkdbKNgMZ5VSWposepOZkIDnFiKnftd4p2+vfI2r9jmq359sWhyX1z/tgtL30qAfM2mBzcdBm9KzwR357I9RbDQtTur9n0IjAmKtrU7XfpV7Jrf1wp/rmagtoB6BbmZgM6leKywjbl+KG/k+4EF36NeibgI1atcYthxsT8Sle3p7LA0t0LcI9axS9Oh4cDpwD7gZhE1w7DXPMDraFzDJcVJjFDZXkRDgo5VkhIihnRXViVpED4Ck3IhQkzxEk5Vu5a1ODEZBKQ5sL8MglcdpOhEC/LJY8N0jZSbtdsclftopLpj2NFs6KSJMP1QmnFgMyBvWNBQgXBki1NgLCgxivAU2TGLM1NbDZ6Y5kpYXMi241gPlZl6lZvWYq5bpMXdaNdKEhGrnHOcqS7xRMEadsmZAUVUza/UtX8a59ZM5LcpmdGtJRiTMBZ3QDFGUgntq502n6mE7t2FPxOzQYK8Ma5/LYhAMhfGSX25aLNhE/iNvdf0XUiarZEmbLelnAHTjJ1LXpBM6fr4sbwkMJ6IvCpahm/wnVEjgFKzDTWetGk1wpzScPtM3gzePT8Nvz9/tuL41evm/P6wHvifev5Xuj94L3yfvG3pmHD/4+Pdw73C/s+h86PzR+bOG7u81nK+91tP56z9G0Lzs</latexit> <latexit sha1_base64="DypLI6QWGLOj7fmyWz0DYpUsnmg=">AFfXiclVNdb9MwFM3KCqN8bfDIi2Gq1rKuagbSeJk0AUJI0SR2JfmUjmO01qz48h2tlWxSv8RP4GfwCcpO2atSDNUpyb63Pf6JkgYVbrT+bVUubNcvXtv5X7twcNHj5+srj09UiKVmBxiwYQ8CZAijMbkUFPNyEkiCeIBI8fB+fvs/PiCSEVF/FWPEtLjaBDTiGKknau/tvS7VodRsw4ObStkD2Htkm2AVQpRwyqlWfcN3fvNHGz6NgNAPSQaWQBxKHQeEtkpw6jPs83wLd98Wat3m0UrJDTEBSwPEkETaQXCWNRKa1pQk5O4Nl5TGAHKkhxgxczrJ27T/INrImG5dZDKDni+osb/lNxv7YBN8bALHPUS6kGdtxonkgKOr/tQFmRiASRPAbBdqi9ATpaOx0qxbWznJrftxzbVxTdZfXe+0O/kC84Y/Nta98er21yoShgKnMQaM6TUmd9JdM8gqSlmxNZgqkiC8DkakDNnxogT1TP5dFpQd54QREK6J9Yg985GMSVGvHAIbMq1M2zLno7CzV0dueoXGSahLjIlGUMqAFyEYdhFQSrNnIGQhL6rQCPERu5rT7IdytzaQZEnZBdLkQzHtGRXn2kqSA23LwVFoDUoSk0sOEdx+MrACHKRiGJUMp0NszRxF7Ur1Z4QRM1bt2UkhENhaQDGiPGSKRhtpXd7jXUMN9r8ANxFyTJgVP9OSESaSGdkmLErLuwAXwBM/N/SBpPkc4sl2VyAa6YrC8iIbGxewxoQgMBlKkSUnwXHwu1BGgyF1DgSflsALhptS/OZPzxtF23/d3v7yZn3v3XheV7zn3kuv4fnejrfnfK63qGHK6eV75UflZ/Lf6r1aqvaLqCVpXHM6+0qjt/AWSk5PE=</latexit> Conditional random fields (CRFs) exp( Ψ ( w , y )) ■ Decoding: Direct application of Viterbi! P ( y | w ) = P exp( Ψ ( w , y 0 )) y 0 ∈ Y ( w ) ˆ y = argmax log P ( y | w ) y X exp Ψ ( y 0 , w ) = argmax Ψ ( y , w ) − log y y 0 ∈ Y ( w ) same for all settings of y = argmax Ψ ( y , w ) y 6

  18. <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="DypLI6QWGLOj7fmyWz0DYpUsnmg=">AFfXiclVNdb9MwFM3KCqN8bfDIi2Gq1rKuagbSeJk0AUJI0SR2JfmUjmO01qz48h2tlWxSv8RP4GfwCcpO2atSDNUpyb63Pf6JkgYVbrT+bVUubNcvXtv5X7twcNHj5+srj09UiKVmBxiwYQ8CZAijMbkUFPNyEkiCeIBI8fB+fvs/PiCSEVF/FWPEtLjaBDTiGKknau/tvS7VodRsw4ObStkD2Htkm2AVQpRwyqlWfcN3fvNHGz6NgNAPSQaWQBxKHQeEtkpw6jPs83wLd98Wat3m0UrJDTEBSwPEkETaQXCWNRKa1pQk5O4Nl5TGAHKkhxgxczrJ27T/INrImG5dZDKDni+osb/lNxv7YBN8bALHPUS6kGdtxonkgKOr/tQFmRiASRPAbBdqi9ATpaOx0qxbWznJrftxzbVxTdZfXe+0O/kC84Y/Nta98er21yoShgKnMQaM6TUmd9JdM8gqSlmxNZgqkiC8DkakDNnxogT1TP5dFpQd54QREK6J9Yg985GMSVGvHAIbMq1M2zLno7CzV0dueoXGSahLjIlGUMqAFyEYdhFQSrNnIGQhL6rQCPERu5rT7IdytzaQZEnZBdLkQzHtGRXn2kqSA23LwVFoDUoSk0sOEdx+MrACHKRiGJUMp0NszRxF7Ur1Z4QRM1bt2UkhENhaQDGiPGSKRhtpXd7jXUMN9r8ANxFyTJgVP9OSESaSGdkmLErLuwAXwBM/N/SBpPkc4sl2VyAa6YrC8iIbGxewxoQgMBlKkSUnwXHwu1BGgyF1DgSflsALhptS/OZPzxtF23/d3v7yZn3v3XheV7zn3kuv4fnejrfnfK63qGHK6eV75UflZ/Lf6r1aqvaLqCVpXHM6+0qjt/AWSk5PE=</latexit> <latexit sha1_base64="Hl4SzGDfq/v1/mRkJxB+iyWEgs4=">AH9nicnVTPb9s2FayZG69X0137IVbEFha3cDqCmyXAkW3wy7tPMBpOoSOQFGUTYeUBIpybBD8V3obdt3fstv+kx1HUrJrOU4CTICl5/e+7+P3HgnGBaOlHAz+2dv/5ODw086Dh93Pv/iy68eHT1+V+aVwOQM5ywX72NUEkYzciapZOR9IQjiMSPn8dVPtn4+J6KkeTaSy4KMOZpkNKUYSZOKjg6uidwiqSOgovM/ASQCQmHC0iJW1Cg6HvAgA5TcC1DYPuyU0YTAXCaug7hMO6QtDQA72qBXoX/XZi96SpgY2i4ReFyBcAmncSKfoy1JeZBdIGR3xbrRBrUQVfRbq4H/M4qP29kBudwZ2Lj2PpD8L7KpOxhHfzo2eKfs0QCY103GkZtrPI2m9IlZM0ZpVnzFaip3UG2bI73R4qj2SzesXVtrM32p6NOZ7tdOr+xf6zdo5rvG0v6FyvrjgMki8KHckokAhAnuQSpfTvkdbKNgMZ5VSWposepOZkIDnFiKnftd4p2+vfI2r9jmq359sWhyX1z/tgtL30qAfM2mBzcdBm9KzwR357I9RbDQtTur9n0IjAmKtrU7XfpV7Jrf1wp/rmagtoB6BbmZgM6leKywjbl+KG/k+4EF36NeibgI1atcYthxsT8Sle3p7LA0t0LcI9axS9Oh4cDpwD7gZhE1w7DXPMDraFzDJcVJjFDZXkRDgo5VkhIihnRXViVpED4Ck3IhQkzxEk5Vu5a1ODEZBKQ5sL8MglcdpOhEC/LJY8N0jZSbtdsclftopLpj2NFs6KSJMP1QmnFgMyBvWNBQgXBki1NgLCgxivAU2TGLM1NbDZ6Y5kpYXMi241gPlZl6lZvWYq5bpMXdaNdKEhGrnHOcqS7xRMEadsmZAUVUza/UtX8a59ZM5LcpmdGtJRiTMBZ3QDFGUgntq502n6mE7t2FPxOzQYK8Ma5/LYhAMhfGSX25aLNhE/iNvdf0XUiarZEmbLelnAHTjJ1LXpBM6fr4sbwkMJ6IvCpahm/wnVEjgFKzDTWetGk1wpzScPtM3gzePT8Nvz9/tuL41evm/P6wHvifev5Xuj94L3yfvG3pmHD/4+Pdw73C/s+h86PzR+bOG7u81nK+91tP56z9G0Lzs</latexit> <latexit sha1_base64="DypLI6QWGLOj7fmyWz0DYpUsnmg=">AFfXiclVNdb9MwFM3KCqN8bfDIi2Gq1rKuagbSeJk0AUJI0SR2JfmUjmO01qz48h2tlWxSv8RP4GfwCcpO2atSDNUpyb63Pf6JkgYVbrT+bVUubNcvXtv5X7twcNHj5+srj09UiKVmBxiwYQ8CZAijMbkUFPNyEkiCeIBI8fB+fvs/PiCSEVF/FWPEtLjaBDTiGKknau/tvS7VodRsw4ObStkD2Htkm2AVQpRwyqlWfcN3fvNHGz6NgNAPSQaWQBxKHQeEtkpw6jPs83wLd98Wat3m0UrJDTEBSwPEkETaQXCWNRKa1pQk5O4Nl5TGAHKkhxgxczrJ27T/INrImG5dZDKDni+osb/lNxv7YBN8bALHPUS6kGdtxonkgKOr/tQFmRiASRPAbBdqi9ATpaOx0qxbWznJrftxzbVxTdZfXe+0O/kC84Y/Nta98er21yoShgKnMQaM6TUmd9JdM8gqSlmxNZgqkiC8DkakDNnxogT1TP5dFpQd54QREK6J9Yg985GMSVGvHAIbMq1M2zLno7CzV0dueoXGSahLjIlGUMqAFyEYdhFQSrNnIGQhL6rQCPERu5rT7IdytzaQZEnZBdLkQzHtGRXn2kqSA23LwVFoDUoSk0sOEdx+MrACHKRiGJUMp0NszRxF7Ur1Z4QRM1bt2UkhENhaQDGiPGSKRhtpXd7jXUMN9r8ANxFyTJgVP9OSESaSGdkmLErLuwAXwBM/N/SBpPkc4sl2VyAa6YrC8iIbGxewxoQgMBlKkSUnwXHwu1BGgyF1DgSflsALhptS/OZPzxtF23/d3v7yZn3v3XheV7zn3kuv4fnejrfnfK63qGHK6eV75UflZ/Lf6r1aqvaLqCVpXHM6+0qjt/AWSk5PE=</latexit> Conditional random fields (CRFs) exp( Ψ ( w , y )) ■ Decoding: Direct application of Viterbi! P ( y | w ) = P exp( Ψ ( w , y 0 )) y 0 ∈ Y ( w ) ˆ y = argmax log P ( y | w ) y X exp Ψ ( y 0 , w ) = argmax Ψ ( y , w ) − log y y 0 ∈ Y ( w ) same for all settings of y = argmax Ψ ( y , w ) y M +1 X = argmax ψ ( w , y m , y m − 1 , m ) y m =1 6

  19. <latexit sha1_base64="HUJof5rs93v8xuaMgJAQitvYE=">AGNXiclVPbhMxEN02BEq4tfDIi6GKmtCLsgWpCKlSuQghVRVBohdUp5HX602s2uV7W0bWf4hfoBv4YE3xCu/gHc3SbNQNRSvJPxzJnj45kgYVTpVuv73HzlRvXmrYXbtTt3791/sLj08ECJVGKyjwUT8ihAijAak31NSNHiSIB4wcBqdvs/PDMyIVFfFnPUhIh6NeTCOKkXau7tL81odthVtwICbc7sGsu/ANsE2gCrlkFOteoavu3bE7O36tsAOo+0cgCiEOh85TIjhEGXZ5thq/7h9v1urtRoEKOQ1BEZYXiCTCBpKLpDGLQtOaEoXcveK0hAjnQfI2a+jOo27V+AVjKka18ymYievlBjd91vNnbBKnjfBA67j3RBz9oME8keRxfdsQsy0QMjEcCkCrX6rPAR1cGQaibXeo5ybUEusVYuwWZVzZD/X4Ft1eWNSa/DhBNwXeO/2nOg06IXJ7n0Lv5aVbuLy62NVr7AtOEPjWVvuNquyUMBU45iTVmSKljv5XojkFSU8yIrcFUkQThU9Qjx86MESeqY/KpsqDuPCGIhHS/WIPcO5lhEFdqwAMXmYmvrp5lzlnx6mOXnYMjZNUkxgXhaKUAS1ANqIgpJgzQbOQFhSxXgPnKzot0gO50nyvQJOyO6fBHMO0ZFefUSpYDbcvJFcdEalCQm51hwjuLwmYER4pQNQhKhlOlsCKORPUuvtfCMJmo3RiSEQ2FpD0aI8ZIpG2ld3u09cw32vwHXEPJMmeY/0xIRJpIR2Tokete7AefJL1i/1XJI3Hkc4sX8vkBNxlMl1EQmJji5FhQhEY9KRIkxLhqfycqANAkXuGIp6U04oI16X+1Z6cNg42N/znG5ufXizvBn264L32HvqNTzf2/J2vA9e29v3cGWpslXZqbyufqv+qP6s/ipC5+eGOY+80qr+/gO+OCZh</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="DypLI6QWGLOj7fmyWz0DYpUsnmg=">AFfXiclVNdb9MwFM3KCqN8bfDIi2Gq1rKuagbSeJk0AUJI0SR2JfmUjmO01qz48h2tlWxSv8RP4GfwCcpO2atSDNUpyb63Pf6JkgYVbrT+bVUubNcvXtv5X7twcNHj5+srj09UiKVmBxiwYQ8CZAijMbkUFPNyEkiCeIBI8fB+fvs/PiCSEVF/FWPEtLjaBDTiGKknau/tvS7VodRsw4ObStkD2Htkm2AVQpRwyqlWfcN3fvNHGz6NgNAPSQaWQBxKHQeEtkpw6jPs83wLd98Wat3m0UrJDTEBSwPEkETaQXCWNRKa1pQk5O4Nl5TGAHKkhxgxczrJ27T/INrImG5dZDKDni+osb/lNxv7YBN8bALHPUS6kGdtxonkgKOr/tQFmRiASRPAbBdqi9ATpaOx0qxbWznJrftxzbVxTdZfXe+0O/kC84Y/Nta98er21yoShgKnMQaM6TUmd9JdM8gqSlmxNZgqkiC8DkakDNnxogT1TP5dFpQd54QREK6J9Yg985GMSVGvHAIbMq1M2zLno7CzV0dueoXGSahLjIlGUMqAFyEYdhFQSrNnIGQhL6rQCPERu5rT7IdytzaQZEnZBdLkQzHtGRXn2kqSA23LwVFoDUoSk0sOEdx+MrACHKRiGJUMp0NszRxF7Ur1Z4QRM1bt2UkhENhaQDGiPGSKRhtpXd7jXUMN9r8ANxFyTJgVP9OSESaSGdkmLErLuwAXwBM/N/SBpPkc4sl2VyAa6YrC8iIbGxewxoQgMBlKkSUnwXHwu1BGgyF1DgSflsALhptS/OZPzxtF23/d3v7yZn3v3XheV7zn3kuv4fnejrfnfK63qGHK6eV75UflZ/Lf6r1aqvaLqCVpXHM6+0qjt/AWSk5PE=</latexit> <latexit sha1_base64="Hl4SzGDfq/v1/mRkJxB+iyWEgs4=">AH9nicnVTPb9s2FayZG69X0137IVbEFha3cDqCmyXAkW3wy7tPMBpOoSOQFGUTYeUBIpybBD8V3obdt3fstv+kx1HUrJrOU4CTICl5/e+7+P3HgnGBaOlHAz+2dv/5ODw086Dh93Pv/iy68eHT1+V+aVwOQM5ywX72NUEkYzciapZOR9IQjiMSPn8dVPtn4+J6KkeTaSy4KMOZpkNKUYSZOKjg6uidwiqSOgovM/ASQCQmHC0iJW1Cg6HvAgA5TcC1DYPuyU0YTAXCaug7hMO6QtDQA72qBXoX/XZi96SpgY2i4ReFyBcAmncSKfoy1JeZBdIGR3xbrRBrUQVfRbq4H/M4qP29kBudwZ2Lj2PpD8L7KpOxhHfzo2eKfs0QCY103GkZtrPI2m9IlZM0ZpVnzFaip3UG2bI73R4qj2SzesXVtrM32p6NOZ7tdOr+xf6zdo5rvG0v6FyvrjgMki8KHckokAhAnuQSpfTvkdbKNgMZ5VSWposepOZkIDnFiKnftd4p2+vfI2r9jmq359sWhyX1z/tgtL30qAfM2mBzcdBm9KzwR357I9RbDQtTur9n0IjAmKtrU7XfpV7Jrf1wp/rmagtoB6BbmZgM6leKywjbl+KG/k+4EF36NeibgI1atcYthxsT8Sle3p7LA0t0LcI9axS9Oh4cDpwD7gZhE1w7DXPMDraFzDJcVJjFDZXkRDgo5VkhIihnRXViVpED4Ck3IhQkzxEk5Vu5a1ODEZBKQ5sL8MglcdpOhEC/LJY8N0jZSbtdsclftopLpj2NFs6KSJMP1QmnFgMyBvWNBQgXBki1NgLCgxivAU2TGLM1NbDZ6Y5kpYXMi241gPlZl6lZvWYq5bpMXdaNdKEhGrnHOcqS7xRMEadsmZAUVUza/UtX8a59ZM5LcpmdGtJRiTMBZ3QDFGUgntq502n6mE7t2FPxOzQYK8Ma5/LYhAMhfGSX25aLNhE/iNvdf0XUiarZEmbLelnAHTjJ1LXpBM6fr4sbwkMJ6IvCpahm/wnVEjgFKzDTWetGk1wpzScPtM3gzePT8Nvz9/tuL41evm/P6wHvifev5Xuj94L3yfvG3pmHD/4+Pdw73C/s+h86PzR+bOG7u81nK+91tP56z9G0Lzs</latexit> <latexit sha1_base64="DypLI6QWGLOj7fmyWz0DYpUsnmg=">AFfXiclVNdb9MwFM3KCqN8bfDIi2Gq1rKuagbSeJk0AUJI0SR2JfmUjmO01qz48h2tlWxSv8RP4GfwCcpO2atSDNUpyb63Pf6JkgYVbrT+bVUubNcvXtv5X7twcNHj5+srj09UiKVmBxiwYQ8CZAijMbkUFPNyEkiCeIBI8fB+fvs/PiCSEVF/FWPEtLjaBDTiGKknau/tvS7VodRsw4ObStkD2Htkm2AVQpRwyqlWfcN3fvNHGz6NgNAPSQaWQBxKHQeEtkpw6jPs83wLd98Wat3m0UrJDTEBSwPEkETaQXCWNRKa1pQk5O4Nl5TGAHKkhxgxczrJ27T/INrImG5dZDKDni+osb/lNxv7YBN8bALHPUS6kGdtxonkgKOr/tQFmRiASRPAbBdqi9ATpaOx0qxbWznJrftxzbVxTdZfXe+0O/kC84Y/Nta98er21yoShgKnMQaM6TUmd9JdM8gqSlmxNZgqkiC8DkakDNnxogT1TP5dFpQd54QREK6J9Yg985GMSVGvHAIbMq1M2zLno7CzV0dueoXGSahLjIlGUMqAFyEYdhFQSrNnIGQhL6rQCPERu5rT7IdytzaQZEnZBdLkQzHtGRXn2kqSA23LwVFoDUoSk0sOEdx+MrACHKRiGJUMp0NszRxF7Ur1Z4QRM1bt2UkhENhaQDGiPGSKRhtpXd7jXUMN9r8ANxFyTJgVP9OSESaSGdkmLErLuwAXwBM/N/SBpPkc4sl2VyAa6YrC8iIbGxewxoQgMBlKkSUnwXHwu1BGgyF1DgSflsALhptS/OZPzxtF23/d3v7yZn3v3XheV7zn3kuv4fnejrfnfK63qGHK6eV75UflZ/Lf6r1aqvaLqCVpXHM6+0qjt/AWSk5PE=</latexit> Conditional random fields (CRFs) exp( Ψ ( w , y )) ■ Decoding: Direct application of Viterbi! P ( y | w ) = P exp( Ψ ( w , y 0 )) y 0 ∈ Y ( w ) ˆ y = argmax log P ( y | w ) y X exp Ψ ( y 0 , w ) = argmax Ψ ( y , w ) − log y y 0 ∈ Y ( w ) same for all settings of y = argmax Ψ ( y , w ) y M +1 X = argmax ψ ( w , y m , y m − 1 , m ) = s m ( y m , y m − 1 ) y m =1 6

  20. <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="HUJof5rs93v8xuaMgJAQitvYE=">AGNXiclVPbhMxEN02BEq4tfDIi6GKmtCLsgWpCKlSuQghVRVBohdUp5HX602s2uV7W0bWf4hfoBv4YE3xCu/gHc3SbNQNRSvJPxzJnj45kgYVTpVuv73HzlRvXmrYXbtTt3791/sLj08ECJVGKyjwUT8ihAijAak31NSNHiSIB4wcBqdvs/PDMyIVFfFnPUhIh6NeTCOKkXau7tL81odthVtwICbc7sGsu/ANsE2gCrlkFOteoavu3bE7O36tsAOo+0cgCiEOh85TIjhEGXZ5thq/7h9v1urtRoEKOQ1BEZYXiCTCBpKLpDGLQtOaEoXcveK0hAjnQfI2a+jOo27V+AVjKka18ymYievlBjd91vNnbBKnjfBA67j3RBz9oME8keRxfdsQsy0QMjEcCkCrX6rPAR1cGQaibXeo5ybUEusVYuwWZVzZD/X4Ft1eWNSa/DhBNwXeO/2nOg06IXJ7n0Lv5aVbuLy62NVr7AtOEPjWVvuNquyUMBU45iTVmSKljv5XojkFSU8yIrcFUkQThU9Qjx86MESeqY/KpsqDuPCGIhHS/WIPcO5lhEFdqwAMXmYmvrp5lzlnx6mOXnYMjZNUkxgXhaKUAS1ANqIgpJgzQbOQFhSxXgPnKzot0gO50nyvQJOyO6fBHMO0ZFefUSpYDbcvJFcdEalCQm51hwjuLwmYER4pQNQhKhlOlsCKORPUuvtfCMJmo3RiSEQ2FpD0aI8ZIpG2ld3u09cw32vwHXEPJMmeY/0xIRJpIR2Tokete7AefJL1i/1XJI3Hkc4sX8vkBNxlMl1EQmJji5FhQhEY9KRIkxLhqfycqANAkXuGIp6U04oI16X+1Z6cNg42N/znG5ufXizvBn264L32HvqNTzf2/J2vA9e29v3cGWpslXZqbyufqv+qP6s/ipC5+eGOY+80qr+/gO+OCZh</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> Conditional random fields (CRFs) ■ Decoding: Direct application of Viterbi! M +1 X = argmax ψ ( w , y m , y m − 1 , m ) = s m ( y m , y m − 1 ) ˆ y y m =1 7

  21. <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="cFZeFhMRSODTiOyXYHFD+C/9s=">AGOHiclVPdbtMwFM5WCiP8bcAdN4apWst+1Awk0KRJE5sQ0jRJPaD5hI5jtNas5PIdrZVlh+JF+BNuOMOcsT4CRt16wFaZbqnB5/5zufj8JUkalard/zM3XbtVv31m46967/+Dho8Wlx0cyQmhzhiTgJkCSMxuRQUcXISoI4gEjx8HZbn5+fE6EpEn8WQ1S0uWoF9OIYqSsy1+a/+Y2YEfSJgy4vjBrIP8OTAtsAygzDhnlVElf823PfNUHq57JAVD1iUIGQBwmqgiJzJh4PN803zds/94y210miUr5DQEJaxIEAmENSXaXOWhJbRFQmFe8UmpTGAHKk+Rkx/GeVtmX8QreRMN75kOoGevlBzf91rNfBKnjfApa7j1Qpz5icE4keR5f+2AVZ0gOjIoDJKriNWfCR1MFQal6u9YLlxgW54lq5IpuVNWe+QW2pc+bk+6WCyfofO1tHVjS6aKXJ0XpZ+CviZhK4i8utzfaxQLThjc0lp3h6tgGFzBMcMZJrDBDUp567VR1NRKYkaMCzNJUoTPUI+cWjNGnMiuLgbLgIb1hCBKhP3FChTeyQiNuJQDHlhkXn95/Sx3zjo7zVT0tqtpnGaKxLhMFGUMqATkUwpCKghWbGANhAW1WgHuIzsuys6ybiJNH3CzomqXgTzrpZRkb0iKeCmGnxZXtSFgsTkAiecozh8qWGEOGWDkEQoYyqfw2hkz6rXWnhOUzks3ZiSEQUTQXs0RoyRSMF8q7rtp69gsbtwj9gHEuTAqv6YEoFUIqySsk2NfbAefJ63jPkfksZjpDWr19KFAHuZvC5JSmJtyqlhiSQw6IkSyuCp+ILoZYARfYZSjyphpUI26Xe9Z6cNo42N7xXG5ufXi/vBv264LzHnhNB3PePsOB+cjnPo4NrT2lZt7ZX/17/Wf9V/1C5+eGMU+cyqr/+Qtf4ih5</latexit> <latexit sha1_base64="HUJof5rs93v8xuaMgJAQitvYE=">AGNXiclVPbhMxEN02BEq4tfDIi6GKmtCLsgWpCKlSuQghVRVBohdUp5HX602s2uV7W0bWf4hfoBv4YE3xCu/gHc3SbNQNRSvJPxzJnj45kgYVTpVuv73HzlRvXmrYXbtTt3791/sLj08ECJVGKyjwUT8ihAijAak31NSNHiSIB4wcBqdvs/PDMyIVFfFnPUhIh6NeTCOKkXau7tL81odthVtwICbc7sGsu/ANsE2gCrlkFOteoavu3bE7O36tsAOo+0cgCiEOh85TIjhEGXZ5thq/7h9v1urtRoEKOQ1BEZYXiCTCBpKLpDGLQtOaEoXcveK0hAjnQfI2a+jOo27V+AVjKka18ymYievlBjd91vNnbBKnjfBA67j3RBz9oME8keRxfdsQsy0QMjEcCkCrX6rPAR1cGQaibXeo5ybUEusVYuwWZVzZD/X4Ft1eWNSa/DhBNwXeO/2nOg06IXJ7n0Lv5aVbuLy62NVr7AtOEPjWVvuNquyUMBU45iTVmSKljv5XojkFSU8yIrcFUkQThU9Qjx86MESeqY/KpsqDuPCGIhHS/WIPcO5lhEFdqwAMXmYmvrp5lzlnx6mOXnYMjZNUkxgXhaKUAS1ANqIgpJgzQbOQFhSxXgPnKzot0gO50nyvQJOyO6fBHMO0ZFefUSpYDbcvJFcdEalCQm51hwjuLwmYER4pQNQhKhlOlsCKORPUuvtfCMJmo3RiSEQ2FpD0aI8ZIpG2ld3u09cw32vwHXEPJMmeY/0xIRJpIR2Tokete7AefJL1i/1XJI3Hkc4sX8vkBNxlMl1EQmJji5FhQhEY9KRIkxLhqfycqANAkXuGIp6U04oI16X+1Z6cNg42N/znG5ufXizvBn264L32HvqNTzf2/J2vA9e29v3cGWpslXZqbyufqv+qP6s/ipC5+eGOY+80qr+/gO+OCZh</latexit> Conditional random fields (CRFs) ■ Decoding: Direct application of Viterbi! M +1 X = argmax ψ ( w , y m , y m − 1 , m ) = s m ( y m , y m − 1 ) ˆ y y m =1 M +1 X max y 1: M Ψ ( w , y 1: M ) = max s m ( y m , y m − 1 ) y 1: M m =1 7

  22. <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="HUJof5rs93v8xuaMgJAQitvYE=">AGNXiclVPbhMxEN02BEq4tfDIi6GKmtCLsgWpCKlSuQghVRVBohdUp5HX602s2uV7W0bWf4hfoBv4YE3xCu/gHc3SbNQNRSvJPxzJnj45kgYVTpVuv73HzlRvXmrYXbtTt3791/sLj08ECJVGKyjwUT8ihAijAak31NSNHiSIB4wcBqdvs/PDMyIVFfFnPUhIh6NeTCOKkXau7tL81odthVtwICbc7sGsu/ANsE2gCrlkFOteoavu3bE7O36tsAOo+0cgCiEOh85TIjhEGXZ5thq/7h9v1urtRoEKOQ1BEZYXiCTCBpKLpDGLQtOaEoXcveK0hAjnQfI2a+jOo27V+AVjKka18ymYievlBjd91vNnbBKnjfBA67j3RBz9oME8keRxfdsQsy0QMjEcCkCrX6rPAR1cGQaibXeo5ybUEusVYuwWZVzZD/X4Ft1eWNSa/DhBNwXeO/2nOg06IXJ7n0Lv5aVbuLy62NVr7AtOEPjWVvuNquyUMBU45iTVmSKljv5XojkFSU8yIrcFUkQThU9Qjx86MESeqY/KpsqDuPCGIhHS/WIPcO5lhEFdqwAMXmYmvrp5lzlnx6mOXnYMjZNUkxgXhaKUAS1ANqIgpJgzQbOQFhSxXgPnKzot0gO50nyvQJOyO6fBHMO0ZFefUSpYDbcvJFcdEalCQm51hwjuLwmYER4pQNQhKhlOlsCKORPUuvtfCMJmo3RiSEQ2FpD0aI8ZIpG2ld3u09cw32vwHXEPJMmeY/0xIRJpIR2Tokete7AefJL1i/1XJI3Hkc4sX8vkBNxlMl1EQmJji5FhQhEY9KRIkxLhqfycqANAkXuGIp6U04oI16X+1Z6cNg42N/znG5ufXizvBn264L32HvqNTzf2/J2vA9e29v3cGWpslXZqbyufqv+qP6s/ipC5+eGOY+80qr+/gO+OCZh</latexit> <latexit sha1_base64="cFZeFhMRSODTiOyXYHFD+C/9s=">AGOHiclVPdbtMwFM5WCiP8bcAdN4apWst+1Awk0KRJE5sQ0jRJPaD5hI5jtNas5PIdrZVlh+JF+BNuOMOcsT4CRt16wFaZbqnB5/5zufj8JUkalard/zM3XbtVv31m46967/+Dho8Wlx0cyQmhzhiTgJkCSMxuRQUcXISoI4gEjx8HZbn5+fE6EpEn8WQ1S0uWoF9OIYqSsy1+a/+Y2YEfSJgy4vjBrIP8OTAtsAygzDhnlVElf823PfNUHq57JAVD1iUIGQBwmqgiJzJh4PN803zds/94y210miUr5DQEJaxIEAmENSXaXOWhJbRFQmFe8UmpTGAHKk+Rkx/GeVtmX8QreRMN75kOoGevlBzf91rNfBKnjfApa7j1Qpz5icE4keR5f+2AVZ0gOjIoDJKriNWfCR1MFQal6u9YLlxgW54lq5IpuVNWe+QW2pc+bk+6WCyfofO1tHVjS6aKXJ0XpZ+CviZhK4i8utzfaxQLThjc0lp3h6tgGFzBMcMZJrDBDUp567VR1NRKYkaMCzNJUoTPUI+cWjNGnMiuLgbLgIb1hCBKhP3FChTeyQiNuJQDHlhkXn95/Sx3zjo7zVT0tqtpnGaKxLhMFGUMqATkUwpCKghWbGANhAW1WgHuIzsuys6ybiJNH3CzomqXgTzrpZRkb0iKeCmGnxZXtSFgsTkAiecozh8qWGEOGWDkEQoYyqfw2hkz6rXWnhOUzks3ZiSEQUTQXs0RoyRSMF8q7rtp69gsbtwj9gHEuTAqv6YEoFUIqySsk2NfbAefJ63jPkfksZjpDWr19KFAHuZvC5JSmJtyqlhiSQw6IkSyuCp+ILoZYARfYZSjyphpUI26Xe9Z6cNo42N7xXG5ufXi/vBv264LzHnhNB3PePsOB+cjnPo4NrT2lZt7ZX/17/Wf9V/1C5+eGMU+cyqr/+Qtf4ih5</latexit> Conditional random fields (CRFs) ■ Decoding: Direct application of Viterbi! M +1 X = argmax ψ ( w , y m , y m − 1 , m ) = s m ( y m , y m − 1 ) ˆ y y m =1 score of best tag sequence M +1 of length M X max y 1: M Ψ ( w , y 1: M ) = max s m ( y m , y m − 1 ) y 1: M m =1 7

  23. <latexit sha1_base64="WGAjEtshRFBM14UTBXAPpJKzjY=">AGpHiclVRdb9MwFM1WKN8bfCEeDFMZS37oBlIKRJEyCENCaKoNvQ3EWO46TW7CSynW1VFIlfwyv8Hf4NdtJ2TVOQFqnO7fW5xf23FjRqXqdP4sLNauXa/fWLrZuHX7zt17yv3D2SUCEx6OGKROHKRJIyGpKeoYuQoFgRxl5FD9/SdmT8I0LSKPymhjHpcxSE1KcYKZ1yVmoPG03YlbQFXZ6eZxvAvIdZG+wAKBMOGeVUSflO3Z2ku6v25kBQDUgCmUAYi9SeYmfTRiGDjdDyjdt/Y+3G81uq2CFnHqgOUCvkA4heQibs2z0M7SkoU8vaZFaQgR2qAEUu/j3Xb2T+I1gzTlRcZT6GrC2rtbdrt1h5YBx/aQHMPkCrsZnhRCLg6MKZpCLAjBuApjuQqM5Dz62OhxZNe3azFmu3JBLrVLsnmqhvkKHdiRDm9Np3UOTvE5qf1mX7NWu17M5L2fg59xUVWZLdNFuhUG24IMhQEjzyUeZBXGan1qpTmKotVpZzl1c5WJ39ANbBHwao1erOyqKAXoQTkKFGZLy2O7Eqp8ioShmJGvARJIY4VMUkGMdhogT2U/zS5yBps54wI+E/oUK5NnpihRxKYfc1Uiz13J2ziTnzR0nyn/dT2kYJ4qEuBDyEwZUBMwXAXhUEKzYUAcIC6q9AjxA+moq/d3QuzolMyDsjKjyQjDvp9LP1UuWXJ6Viy+KhTagICE5xHnKPSepdBHnLKhR3yUMGXuvD+O5/VrwzujsRy1bkLJiIKRoAENEWPEV9AM5bR+DRTMxwZ8T/QGCbKvX+OiUAqEtpJcSUyvWEBfGyOTPY/JA0nSB2Wl5XmBvRiTF+imIRpVtxQFkC3UBESVwyXKnPjWoC5OtKPCkXFYg9Cm1Z89kNTjY3rJfbG1/ebm6+3Z0XpesR9YTq2XZ1itr1/poda2ehWs/aj9rv2q/60/rn+pf670CurgwqnlglZ76yV/XA02e</latexit> <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="cFZeFhMRSODTiOyXYHFD+C/9s=">AGOHiclVPdbtMwFM5WCiP8bcAdN4apWst+1Awk0KRJE5sQ0jRJPaD5hI5jtNas5PIdrZVlh+JF+BNuOMOcsT4CRt16wFaZbqnB5/5zufj8JUkalard/zM3XbtVv31m46967/+Dho8Wlx0cyQmhzhiTgJkCSMxuRQUcXISoI4gEjx8HZbn5+fE6EpEn8WQ1S0uWoF9OIYqSsy1+a/+Y2YEfSJgy4vjBrIP8OTAtsAygzDhnlVElf823PfNUHq57JAVD1iUIGQBwmqgiJzJh4PN803zds/94y210miUr5DQEJaxIEAmENSXaXOWhJbRFQmFe8UmpTGAHKk+Rkx/GeVtmX8QreRMN75kOoGevlBzf91rNfBKnjfApa7j1Qpz5icE4keR5f+2AVZ0gOjIoDJKriNWfCR1MFQal6u9YLlxgW54lq5IpuVNWe+QW2pc+bk+6WCyfofO1tHVjS6aKXJ0XpZ+CviZhK4i8utzfaxQLThjc0lp3h6tgGFzBMcMZJrDBDUp567VR1NRKYkaMCzNJUoTPUI+cWjNGnMiuLgbLgIb1hCBKhP3FChTeyQiNuJQDHlhkXn95/Sx3zjo7zVT0tqtpnGaKxLhMFGUMqATkUwpCKghWbGANhAW1WgHuIzsuys6ybiJNH3CzomqXgTzrpZRkb0iKeCmGnxZXtSFgsTkAiecozh8qWGEOGWDkEQoYyqfw2hkz6rXWnhOUzks3ZiSEQUTQXs0RoyRSMF8q7rtp69gsbtwj9gHEuTAqv6YEoFUIqySsk2NfbAefJ63jPkfksZjpDWr19KFAHuZvC5JSmJtyqlhiSQw6IkSyuCp+ILoZYARfYZSjyphpUI26Xe9Z6cNo42N7xXG5ufXi/vBv264LzHnhNB3PePsOB+cjnPo4NrT2lZt7ZX/17/Wf9V/1C5+eGMU+cyqr/+Qtf4ih5</latexit> <latexit sha1_base64="HUJof5rs93v8xuaMgJAQitvYE=">AGNXiclVPbhMxEN02BEq4tfDIi6GKmtCLsgWpCKlSuQghVRVBohdUp5HX602s2uV7W0bWf4hfoBv4YE3xCu/gHc3SbNQNRSvJPxzJnj45kgYVTpVuv73HzlRvXmrYXbtTt3791/sLj08ECJVGKyjwUT8ihAijAak31NSNHiSIB4wcBqdvs/PDMyIVFfFnPUhIh6NeTCOKkXau7tL81odthVtwICbc7sGsu/ANsE2gCrlkFOteoavu3bE7O36tsAOo+0cgCiEOh85TIjhEGXZ5thq/7h9v1urtRoEKOQ1BEZYXiCTCBpKLpDGLQtOaEoXcveK0hAjnQfI2a+jOo27V+AVjKka18ymYievlBjd91vNnbBKnjfBA67j3RBz9oME8keRxfdsQsy0QMjEcCkCrX6rPAR1cGQaibXeo5ybUEusVYuwWZVzZD/X4Ft1eWNSa/DhBNwXeO/2nOg06IXJ7n0Lv5aVbuLy62NVr7AtOEPjWVvuNquyUMBU45iTVmSKljv5XojkFSU8yIrcFUkQThU9Qjx86MESeqY/KpsqDuPCGIhHS/WIPcO5lhEFdqwAMXmYmvrp5lzlnx6mOXnYMjZNUkxgXhaKUAS1ANqIgpJgzQbOQFhSxXgPnKzot0gO50nyvQJOyO6fBHMO0ZFefUSpYDbcvJFcdEalCQm51hwjuLwmYER4pQNQhKhlOlsCKORPUuvtfCMJmo3RiSEQ2FpD0aI8ZIpG2ld3u09cw32vwHXEPJMmeY/0xIRJpIR2Tokete7AefJL1i/1XJI3Hkc4sX8vkBNxlMl1EQmJji5FhQhEY9KRIkxLhqfycqANAkXuGIp6U04oI16X+1Z6cNg42N/znG5ufXizvBn264L32HvqNTzf2/J2vA9e29v3cGWpslXZqbyufqv+qP6s/ipC5+eGOY+80qr+/gO+OCZh</latexit> Conditional random fields (CRFs) ■ Decoding: Direct application of Viterbi! M +1 X = argmax ψ ( w , y m , y m − 1 , m ) = s m ( y m , y m − 1 ) ˆ y y m =1 score of best tag sequence M +1 of length M X max y 1: M Ψ ( w , y 1: M ) = max s m ( y m , y m − 1 ) y 1: M m =1 M X = max y M s M +1 ( h / s i , y M ) + max s m ( y m , y m − 1 ) y 1: M − 1 m =1 7

  24. <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="cFZeFhMRSODTiOyXYHFD+C/9s=">AGOHiclVPdbtMwFM5WCiP8bcAdN4apWst+1Awk0KRJE5sQ0jRJPaD5hI5jtNas5PIdrZVlh+JF+BNuOMOcsT4CRt16wFaZbqnB5/5zufj8JUkalard/zM3XbtVv31m46967/+Dho8Wlx0cyQmhzhiTgJkCSMxuRQUcXISoI4gEjx8HZbn5+fE6EpEn8WQ1S0uWoF9OIYqSsy1+a/+Y2YEfSJgy4vjBrIP8OTAtsAygzDhnlVElf823PfNUHq57JAVD1iUIGQBwmqgiJzJh4PN803zds/94y210miUr5DQEJaxIEAmENSXaXOWhJbRFQmFe8UmpTGAHKk+Rkx/GeVtmX8QreRMN75kOoGevlBzf91rNfBKnjfApa7j1Qpz5icE4keR5f+2AVZ0gOjIoDJKriNWfCR1MFQal6u9YLlxgW54lq5IpuVNWe+QW2pc+bk+6WCyfofO1tHVjS6aKXJ0XpZ+CviZhK4i8utzfaxQLThjc0lp3h6tgGFzBMcMZJrDBDUp567VR1NRKYkaMCzNJUoTPUI+cWjNGnMiuLgbLgIb1hCBKhP3FChTeyQiNuJQDHlhkXn95/Sx3zjo7zVT0tqtpnGaKxLhMFGUMqATkUwpCKghWbGANhAW1WgHuIzsuys6ybiJNH3CzomqXgTzrpZRkb0iKeCmGnxZXtSFgsTkAiecozh8qWGEOGWDkEQoYyqfw2hkz6rXWnhOUzks3ZiSEQUTQXs0RoyRSMF8q7rtp69gsbtwj9gHEuTAqv6YEoFUIqySsk2NfbAefJ63jPkfksZjpDWr19KFAHuZvC5JSmJtyqlhiSQw6IkSyuCp+ILoZYARfYZSjyphpUI26Xe9Z6cNo42N7xXG5ufXi/vBv264LzHnhNB3PePsOB+cjnPo4NrT2lZt7ZX/17/Wf9V/1C5+eGMU+cyqr/+Qtf4ih5</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="WGAjEtshRFBM14UTBXAPpJKzjY=">AGpHiclVRdb9MwFM1WKN8bfCEeDFMZS37oBlIKRJEyCENCaKoNvQ3EWO46TW7CSynW1VFIlfwyv8Hf4NdtJ2TVOQFqnO7fW5xf23FjRqXqdP4sLNauXa/fWLrZuHX7zt17yv3D2SUCEx6OGKROHKRJIyGpKeoYuQoFgRxl5FD9/SdmT8I0LSKPymhjHpcxSE1KcYKZ1yVmoPG03YlbQFXZ6eZxvAvIdZG+wAKBMOGeVUSflO3Z2ku6v25kBQDUgCmUAYi9SeYmfTRiGDjdDyjdt/Y+3G81uq2CFnHqgOUCvkA4heQibs2z0M7SkoU8vaZFaQgR2qAEUu/j3Xb2T+I1gzTlRcZT6GrC2rtbdrt1h5YBx/aQHMPkCrsZnhRCLg6MKZpCLAjBuApjuQqM5Dz62OhxZNe3azFmu3JBLrVLsnmqhvkKHdiRDm9Np3UOTvE5qf1mX7NWu17M5L2fg59xUVWZLdNFuhUG24IMhQEjzyUeZBXGan1qpTmKotVpZzl1c5WJ39ANbBHwao1erOyqKAXoQTkKFGZLy2O7Eqp8ioShmJGvARJIY4VMUkGMdhogT2U/zS5yBps54wI+E/oUK5NnpihRxKYfc1Uiz13J2ziTnzR0nyn/dT2kYJ4qEuBDyEwZUBMwXAXhUEKzYUAcIC6q9AjxA+moq/d3QuzolMyDsjKjyQjDvp9LP1UuWXJ6Viy+KhTagICE5xHnKPSepdBHnLKhR3yUMGXuvD+O5/VrwzujsRy1bkLJiIKRoAENEWPEV9AM5bR+DRTMxwZ8T/QGCbKvX+OiUAqEtpJcSUyvWEBfGyOTPY/JA0nSB2Wl5XmBvRiTF+imIRpVtxQFkC3UBESVwyXKnPjWoC5OtKPCkXFYg9Cm1Z89kNTjY3rJfbG1/ebm6+3Z0XpesR9YTq2XZ1itr1/poda2ehWs/aj9rv2q/60/rn+pf670CurgwqnlglZ76yV/XA02e</latexit> <latexit sha1_base64="HUJof5rs93v8xuaMgJAQitvYE=">AGNXiclVPbhMxEN02BEq4tfDIi6GKmtCLsgWpCKlSuQghVRVBohdUp5HX602s2uV7W0bWf4hfoBv4YE3xCu/gHc3SbNQNRSvJPxzJnj45kgYVTpVuv73HzlRvXmrYXbtTt3791/sLj08ECJVGKyjwUT8ihAijAak31NSNHiSIB4wcBqdvs/PDMyIVFfFnPUhIh6NeTCOKkXau7tL81odthVtwICbc7sGsu/ANsE2gCrlkFOteoavu3bE7O36tsAOo+0cgCiEOh85TIjhEGXZ5thq/7h9v1urtRoEKOQ1BEZYXiCTCBpKLpDGLQtOaEoXcveK0hAjnQfI2a+jOo27V+AVjKka18ymYievlBjd91vNnbBKnjfBA67j3RBz9oME8keRxfdsQsy0QMjEcCkCrX6rPAR1cGQaibXeo5ybUEusVYuwWZVzZD/X4Ft1eWNSa/DhBNwXeO/2nOg06IXJ7n0Lv5aVbuLy62NVr7AtOEPjWVvuNquyUMBU45iTVmSKljv5XojkFSU8yIrcFUkQThU9Qjx86MESeqY/KpsqDuPCGIhHS/WIPcO5lhEFdqwAMXmYmvrp5lzlnx6mOXnYMjZNUkxgXhaKUAS1ANqIgpJgzQbOQFhSxXgPnKzot0gO50nyvQJOyO6fBHMO0ZFefUSpYDbcvJFcdEalCQm51hwjuLwmYER4pQNQhKhlOlsCKORPUuvtfCMJmo3RiSEQ2FpD0aI8ZIpG2ld3u09cw32vwHXEPJMmeY/0xIRJpIR2Tokete7AefJL1i/1XJI3Hkc4sX8vkBNxlMl1EQmJji5FhQhEY9KRIkxLhqfycqANAkXuGIp6U04oI16X+1Z6cNg42N/znG5ufXizvBn264L32HvqNTzf2/J2vA9e29v3cGWpslXZqbyufqv+qP6s/ipC5+eGOY+80qr+/gO+OCZh</latexit> Conditional random fields (CRFs) ■ Decoding: Direct application of Viterbi! M +1 X = argmax ψ ( w , y m , y m − 1 , m ) = s m ( y m , y m − 1 ) ˆ y y m =1 score of best tag sequence M +1 of length M X max y 1: M Ψ ( w , y 1: M ) = max s m ( y m , y m − 1 ) score of best tag sequence y 1: M of length M-1 m =1 M X = max y M s M +1 ( h / s i , y M ) + max s m ( y m , y m − 1 ) y 1: M − 1 m =1 7

  25. <latexit sha1_base64="HUJof5rs93v8xuaMgJAQitvYE=">AGNXiclVPbhMxEN02BEq4tfDIi6GKmtCLsgWpCKlSuQghVRVBohdUp5HX602s2uV7W0bWf4hfoBv4YE3xCu/gHc3SbNQNRSvJPxzJnj45kgYVTpVuv73HzlRvXmrYXbtTt3791/sLj08ECJVGKyjwUT8ihAijAak31NSNHiSIB4wcBqdvs/PDMyIVFfFnPUhIh6NeTCOKkXau7tL81odthVtwICbc7sGsu/ANsE2gCrlkFOteoavu3bE7O36tsAOo+0cgCiEOh85TIjhEGXZ5thq/7h9v1urtRoEKOQ1BEZYXiCTCBpKLpDGLQtOaEoXcveK0hAjnQfI2a+jOo27V+AVjKka18ymYievlBjd91vNnbBKnjfBA67j3RBz9oME8keRxfdsQsy0QMjEcCkCrX6rPAR1cGQaibXeo5ybUEusVYuwWZVzZD/X4Ft1eWNSa/DhBNwXeO/2nOg06IXJ7n0Lv5aVbuLy62NVr7AtOEPjWVvuNquyUMBU45iTVmSKljv5XojkFSU8yIrcFUkQThU9Qjx86MESeqY/KpsqDuPCGIhHS/WIPcO5lhEFdqwAMXmYmvrp5lzlnx6mOXnYMjZNUkxgXhaKUAS1ANqIgpJgzQbOQFhSxXgPnKzot0gO50nyvQJOyO6fBHMO0ZFefUSpYDbcvJFcdEalCQm51hwjuLwmYER4pQNQhKhlOlsCKORPUuvtfCMJmo3RiSEQ2FpD0aI8ZIpG2ld3u09cw32vwHXEPJMmeY/0xIRJpIR2Tokete7AefJL1i/1XJI3Hkc4sX8vkBNxlMl1EQmJji5FhQhEY9KRIkxLhqfycqANAkXuGIp6U04oI16X+1Z6cNg42N/znG5ufXizvBn264L32HvqNTzf2/J2vA9e29v3cGWpslXZqbyufqv+qP6s/ipC5+eGOY+80qr+/gO+OCZh</latexit> <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="cFZeFhMRSODTiOyXYHFD+C/9s=">AGOHiclVPdbtMwFM5WCiP8bcAdN4apWst+1Awk0KRJE5sQ0jRJPaD5hI5jtNas5PIdrZVlh+JF+BNuOMOcsT4CRt16wFaZbqnB5/5zufj8JUkalard/zM3XbtVv31m46967/+Dho8Wlx0cyQmhzhiTgJkCSMxuRQUcXISoI4gEjx8HZbn5+fE6EpEn8WQ1S0uWoF9OIYqSsy1+a/+Y2YEfSJgy4vjBrIP8OTAtsAygzDhnlVElf823PfNUHq57JAVD1iUIGQBwmqgiJzJh4PN803zds/94y210miUr5DQEJaxIEAmENSXaXOWhJbRFQmFe8UmpTGAHKk+Rkx/GeVtmX8QreRMN75kOoGevlBzf91rNfBKnjfApa7j1Qpz5icE4keR5f+2AVZ0gOjIoDJKriNWfCR1MFQal6u9YLlxgW54lq5IpuVNWe+QW2pc+bk+6WCyfofO1tHVjS6aKXJ0XpZ+CviZhK4i8utzfaxQLThjc0lp3h6tgGFzBMcMZJrDBDUp567VR1NRKYkaMCzNJUoTPUI+cWjNGnMiuLgbLgIb1hCBKhP3FChTeyQiNuJQDHlhkXn95/Sx3zjo7zVT0tqtpnGaKxLhMFGUMqATkUwpCKghWbGANhAW1WgHuIzsuys6ybiJNH3CzomqXgTzrpZRkb0iKeCmGnxZXtSFgsTkAiecozh8qWGEOGWDkEQoYyqfw2hkz6rXWnhOUzks3ZiSEQUTQXs0RoyRSMF8q7rtp69gsbtwj9gHEuTAqv6YEoFUIqySsk2NfbAefJ63jPkfksZjpDWr19KFAHuZvC5JSmJtyqlhiSQw6IkSyuCp+ILoZYARfYZSjyphpUI26Xe9Z6cNo42N7xXG5ufXi/vBv264LzHnhNB3PePsOB+cjnPo4NrT2lZt7ZX/17/Wf9V/1C5+eGMU+cyqr/+Qtf4ih5</latexit> <latexit sha1_base64="WGAjEtshRFBM14UTBXAPpJKzjY=">AGpHiclVRdb9MwFM1WKN8bfCEeDFMZS37oBlIKRJEyCENCaKoNvQ3EWO46TW7CSynW1VFIlfwyv8Hf4NdtJ2TVOQFqnO7fW5xf23FjRqXqdP4sLNauXa/fWLrZuHX7zt17yv3D2SUCEx6OGKROHKRJIyGpKeoYuQoFgRxl5FD9/SdmT8I0LSKPymhjHpcxSE1KcYKZ1yVmoPG03YlbQFXZ6eZxvAvIdZG+wAKBMOGeVUSflO3Z2ku6v25kBQDUgCmUAYi9SeYmfTRiGDjdDyjdt/Y+3G81uq2CFnHqgOUCvkA4heQibs2z0M7SkoU8vaZFaQgR2qAEUu/j3Xb2T+I1gzTlRcZT6GrC2rtbdrt1h5YBx/aQHMPkCrsZnhRCLg6MKZpCLAjBuApjuQqM5Dz62OhxZNe3azFmu3JBLrVLsnmqhvkKHdiRDm9Np3UOTvE5qf1mX7NWu17M5L2fg59xUVWZLdNFuhUG24IMhQEjzyUeZBXGan1qpTmKotVpZzl1c5WJ39ANbBHwao1erOyqKAXoQTkKFGZLy2O7Eqp8ioShmJGvARJIY4VMUkGMdhogT2U/zS5yBps54wI+E/oUK5NnpihRxKYfc1Uiz13J2ziTnzR0nyn/dT2kYJ4qEuBDyEwZUBMwXAXhUEKzYUAcIC6q9AjxA+moq/d3QuzolMyDsjKjyQjDvp9LP1UuWXJ6Viy+KhTagICE5xHnKPSepdBHnLKhR3yUMGXuvD+O5/VrwzujsRy1bkLJiIKRoAENEWPEV9AM5bR+DRTMxwZ8T/QGCbKvX+OiUAqEtpJcSUyvWEBfGyOTPY/JA0nSB2Wl5XmBvRiTF+imIRpVtxQFkC3UBESVwyXKnPjWoC5OtKPCkXFYg9Cm1Z89kNTjY3rJfbG1/ebm6+3Z0XpesR9YTq2XZ1itr1/poda2ehWs/aj9rv2q/60/rn+pf670CurgwqnlglZ76yV/XA02e</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> Conditional random fields (CRFs) ■ Decoding: Direct application of Viterbi! M +1 X = argmax ψ ( w , y m , y m − 1 , m ) = s m ( y m , y m − 1 ) ˆ y y m =1 score of best tag sequence M +1 of length M X max y 1: M Ψ ( w , y 1: M ) = max s m ( y m , y m − 1 ) score of best tag sequence y 1: M of length M-1 m =1 M X = max y M s M +1 ( h / s i , y M ) + max s m ( y m , y m − 1 ) y 1: M − 1 score of m =1 most probable extension y M 7

  26. <latexit sha1_base64="cFZeFhMRSODTiOyXYHFD+C/9s=">AGOHiclVPdbtMwFM5WCiP8bcAdN4apWst+1Awk0KRJE5sQ0jRJPaD5hI5jtNas5PIdrZVlh+JF+BNuOMOcsT4CRt16wFaZbqnB5/5zufj8JUkalard/zM3XbtVv31m46967/+Dho8Wlx0cyQmhzhiTgJkCSMxuRQUcXISoI4gEjx8HZbn5+fE6EpEn8WQ1S0uWoF9OIYqSsy1+a/+Y2YEfSJgy4vjBrIP8OTAtsAygzDhnlVElf823PfNUHq57JAVD1iUIGQBwmqgiJzJh4PN803zds/94y210miUr5DQEJaxIEAmENSXaXOWhJbRFQmFe8UmpTGAHKk+Rkx/GeVtmX8QreRMN75kOoGevlBzf91rNfBKnjfApa7j1Qpz5icE4keR5f+2AVZ0gOjIoDJKriNWfCR1MFQal6u9YLlxgW54lq5IpuVNWe+QW2pc+bk+6WCyfofO1tHVjS6aKXJ0XpZ+CviZhK4i8utzfaxQLThjc0lp3h6tgGFzBMcMZJrDBDUp567VR1NRKYkaMCzNJUoTPUI+cWjNGnMiuLgbLgIb1hCBKhP3FChTeyQiNuJQDHlhkXn95/Sx3zjo7zVT0tqtpnGaKxLhMFGUMqATkUwpCKghWbGANhAW1WgHuIzsuys6ybiJNH3CzomqXgTzrpZRkb0iKeCmGnxZXtSFgsTkAiecozh8qWGEOGWDkEQoYyqfw2hkz6rXWnhOUzks3ZiSEQUTQXs0RoyRSMF8q7rtp69gsbtwj9gHEuTAqv6YEoFUIqySsk2NfbAefJ63jPkfksZjpDWr19KFAHuZvC5JSmJtyqlhiSQw6IkSyuCp+ILoZYARfYZSjyphpUI26Xe9Z6cNo42N7xXG5ufXi/vBv264LzHnhNB3PePsOB+cjnPo4NrT2lZt7ZX/17/Wf9V/1C5+eGMU+cyqr/+Qtf4ih5</latexit> <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="WGAjEtshRFBM14UTBXAPpJKzjY=">AGpHiclVRdb9MwFM1WKN8bfCEeDFMZS37oBlIKRJEyCENCaKoNvQ3EWO46TW7CSynW1VFIlfwyv8Hf4NdtJ2TVOQFqnO7fW5xf23FjRqXqdP4sLNauXa/fWLrZuHX7zt17yv3D2SUCEx6OGKROHKRJIyGpKeoYuQoFgRxl5FD9/SdmT8I0LSKPymhjHpcxSE1KcYKZ1yVmoPG03YlbQFXZ6eZxvAvIdZG+wAKBMOGeVUSflO3Z2ku6v25kBQDUgCmUAYi9SeYmfTRiGDjdDyjdt/Y+3G81uq2CFnHqgOUCvkA4heQibs2z0M7SkoU8vaZFaQgR2qAEUu/j3Xb2T+I1gzTlRcZT6GrC2rtbdrt1h5YBx/aQHMPkCrsZnhRCLg6MKZpCLAjBuApjuQqM5Dz62OhxZNe3azFmu3JBLrVLsnmqhvkKHdiRDm9Np3UOTvE5qf1mX7NWu17M5L2fg59xUVWZLdNFuhUG24IMhQEjzyUeZBXGan1qpTmKotVpZzl1c5WJ39ANbBHwao1erOyqKAXoQTkKFGZLy2O7Eqp8ioShmJGvARJIY4VMUkGMdhogT2U/zS5yBps54wI+E/oUK5NnpihRxKYfc1Uiz13J2ziTnzR0nyn/dT2kYJ4qEuBDyEwZUBMwXAXhUEKzYUAcIC6q9AjxA+moq/d3QuzolMyDsjKjyQjDvp9LP1UuWXJ6Viy+KhTagICE5xHnKPSepdBHnLKhR3yUMGXuvD+O5/VrwzujsRy1bkLJiIKRoAENEWPEV9AM5bR+DRTMxwZ8T/QGCbKvX+OiUAqEtpJcSUyvWEBfGyOTPY/JA0nSB2Wl5XmBvRiTF+imIRpVtxQFkC3UBESVwyXKnPjWoC5OtKPCkXFYg9Cm1Z89kNTjY3rJfbG1/ebm6+3Z0XpesR9YTq2XZ1itr1/poda2ehWs/aj9rv2q/60/rn+pf670CurgwqnlglZ76yV/XA02e</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="HUJof5rs93v8xuaMgJAQitvYE=">AGNXiclVPbhMxEN02BEq4tfDIi6GKmtCLsgWpCKlSuQghVRVBohdUp5HX602s2uV7W0bWf4hfoBv4YE3xCu/gHc3SbNQNRSvJPxzJnj45kgYVTpVuv73HzlRvXmrYXbtTt3791/sLj08ECJVGKyjwUT8ihAijAak31NSNHiSIB4wcBqdvs/PDMyIVFfFnPUhIh6NeTCOKkXau7tL81odthVtwICbc7sGsu/ANsE2gCrlkFOteoavu3bE7O36tsAOo+0cgCiEOh85TIjhEGXZ5thq/7h9v1urtRoEKOQ1BEZYXiCTCBpKLpDGLQtOaEoXcveK0hAjnQfI2a+jOo27V+AVjKka18ymYievlBjd91vNnbBKnjfBA67j3RBz9oME8keRxfdsQsy0QMjEcCkCrX6rPAR1cGQaibXeo5ybUEusVYuwWZVzZD/X4Ft1eWNSa/DhBNwXeO/2nOg06IXJ7n0Lv5aVbuLy62NVr7AtOEPjWVvuNquyUMBU45iTVmSKljv5XojkFSU8yIrcFUkQThU9Qjx86MESeqY/KpsqDuPCGIhHS/WIPcO5lhEFdqwAMXmYmvrp5lzlnx6mOXnYMjZNUkxgXhaKUAS1ANqIgpJgzQbOQFhSxXgPnKzot0gO50nyvQJOyO6fBHMO0ZFefUSpYDbcvJFcdEalCQm51hwjuLwmYER4pQNQhKhlOlsCKORPUuvtfCMJmo3RiSEQ2FpD0aI8ZIpG2ld3u09cw32vwHXEPJMmeY/0xIRJpIR2Tokete7AefJL1i/1XJI3Hkc4sX8vkBNxlMl1EQmJji5FhQhEY9KRIkxLhqfycqANAkXuGIp6U04oI16X+1Z6cNg42N/znG5ufXizvBn264L32HvqNTzf2/J2vA9e29v3cGWpslXZqbyufqv+qP6s/ipC5+eGOY+80qr+/gO+OCZh</latexit> Conditional random fields (CRFs) ■ Decoding: Direct application of Viterbi! M +1 X = argmax ψ ( w , y m , y m − 1 , m ) = s m ( y m , y m − 1 ) ˆ y y m =1 same subproblem! score of best tag sequence M +1 of length M X max y 1: M Ψ ( w , y 1: M ) = max s m ( y m , y m − 1 ) score of best tag sequence y 1: M of length M-1 m =1 M X = max y M s M +1 ( h / s i , y M ) + max s m ( y m , y m − 1 ) y 1: M − 1 score of m =1 most probable extension y M 7

  27. <latexit sha1_base64="cFZeFhMRSODTiOyXYHFD+C/9s=">AGOHiclVPdbtMwFM5WCiP8bcAdN4apWst+1Awk0KRJE5sQ0jRJPaD5hI5jtNas5PIdrZVlh+JF+BNuOMOcsT4CRt16wFaZbqnB5/5zufj8JUkalard/zM3XbtVv31m46967/+Dho8Wlx0cyQmhzhiTgJkCSMxuRQUcXISoI4gEjx8HZbn5+fE6EpEn8WQ1S0uWoF9OIYqSsy1+a/+Y2YEfSJgy4vjBrIP8OTAtsAygzDhnlVElf823PfNUHq57JAVD1iUIGQBwmqgiJzJh4PN803zds/94y210miUr5DQEJaxIEAmENSXaXOWhJbRFQmFe8UmpTGAHKk+Rkx/GeVtmX8QreRMN75kOoGevlBzf91rNfBKnjfApa7j1Qpz5icE4keR5f+2AVZ0gOjIoDJKriNWfCR1MFQal6u9YLlxgW54lq5IpuVNWe+QW2pc+bk+6WCyfofO1tHVjS6aKXJ0XpZ+CviZhK4i8utzfaxQLThjc0lp3h6tgGFzBMcMZJrDBDUp567VR1NRKYkaMCzNJUoTPUI+cWjNGnMiuLgbLgIb1hCBKhP3FChTeyQiNuJQDHlhkXn95/Sx3zjo7zVT0tqtpnGaKxLhMFGUMqATkUwpCKghWbGANhAW1WgHuIzsuys6ybiJNH3CzomqXgTzrpZRkb0iKeCmGnxZXtSFgsTkAiecozh8qWGEOGWDkEQoYyqfw2hkz6rXWnhOUzks3ZiSEQUTQXs0RoyRSMF8q7rtp69gsbtwj9gHEuTAqv6YEoFUIqySsk2NfbAefJ63jPkfksZjpDWr19KFAHuZvC5JSmJtyqlhiSQw6IkSyuCp+ILoZYARfYZSjyphpUI26Xe9Z6cNo42N7xXG5ufXi/vBv264LzHnhNB3PePsOB+cjnPo4NrT2lZt7ZX/17/Wf9V/1C5+eGMU+cyqr/+Qtf4ih5</latexit> <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="WGAjEtshRFBM14UTBXAPpJKzjY=">AGpHiclVRdb9MwFM1WKN8bfCEeDFMZS37oBlIKRJEyCENCaKoNvQ3EWO46TW7CSynW1VFIlfwyv8Hf4NdtJ2TVOQFqnO7fW5xf23FjRqXqdP4sLNauXa/fWLrZuHX7zt17yv3D2SUCEx6OGKROHKRJIyGpKeoYuQoFgRxl5FD9/SdmT8I0LSKPymhjHpcxSE1KcYKZ1yVmoPG03YlbQFXZ6eZxvAvIdZG+wAKBMOGeVUSflO3Z2ku6v25kBQDUgCmUAYi9SeYmfTRiGDjdDyjdt/Y+3G81uq2CFnHqgOUCvkA4heQibs2z0M7SkoU8vaZFaQgR2qAEUu/j3Xb2T+I1gzTlRcZT6GrC2rtbdrt1h5YBx/aQHMPkCrsZnhRCLg6MKZpCLAjBuApjuQqM5Dz62OhxZNe3azFmu3JBLrVLsnmqhvkKHdiRDm9Np3UOTvE5qf1mX7NWu17M5L2fg59xUVWZLdNFuhUG24IMhQEjzyUeZBXGan1qpTmKotVpZzl1c5WJ39ANbBHwao1erOyqKAXoQTkKFGZLy2O7Eqp8ioShmJGvARJIY4VMUkGMdhogT2U/zS5yBps54wI+E/oUK5NnpihRxKYfc1Uiz13J2ziTnzR0nyn/dT2kYJ4qEuBDyEwZUBMwXAXhUEKzYUAcIC6q9AjxA+moq/d3QuzolMyDsjKjyQjDvp9LP1UuWXJ6Viy+KhTagICE5xHnKPSepdBHnLKhR3yUMGXuvD+O5/VrwzujsRy1bkLJiIKRoAENEWPEV9AM5bR+DRTMxwZ8T/QGCbKvX+OiUAqEtpJcSUyvWEBfGyOTPY/JA0nSB2Wl5XmBvRiTF+imIRpVtxQFkC3UBESVwyXKnPjWoC5OtKPCkXFYg9Cm1Z89kNTjY3rJfbG1/ebm6+3Z0XpesR9YTq2XZ1itr1/poda2ehWs/aj9rv2q/60/rn+pf670CurgwqnlglZ76yV/XA02e</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="HUJof5rs93v8xuaMgJAQitvYE=">AGNXiclVPbhMxEN02BEq4tfDIi6GKmtCLsgWpCKlSuQghVRVBohdUp5HX602s2uV7W0bWf4hfoBv4YE3xCu/gHc3SbNQNRSvJPxzJnj45kgYVTpVuv73HzlRvXmrYXbtTt3791/sLj08ECJVGKyjwUT8ihAijAak31NSNHiSIB4wcBqdvs/PDMyIVFfFnPUhIh6NeTCOKkXau7tL81odthVtwICbc7sGsu/ANsE2gCrlkFOteoavu3bE7O36tsAOo+0cgCiEOh85TIjhEGXZ5thq/7h9v1urtRoEKOQ1BEZYXiCTCBpKLpDGLQtOaEoXcveK0hAjnQfI2a+jOo27V+AVjKka18ymYievlBjd91vNnbBKnjfBA67j3RBz9oME8keRxfdsQsy0QMjEcCkCrX6rPAR1cGQaibXeo5ybUEusVYuwWZVzZD/X4Ft1eWNSa/DhBNwXeO/2nOg06IXJ7n0Lv5aVbuLy62NVr7AtOEPjWVvuNquyUMBU45iTVmSKljv5XojkFSU8yIrcFUkQThU9Qjx86MESeqY/KpsqDuPCGIhHS/WIPcO5lhEFdqwAMXmYmvrp5lzlnx6mOXnYMjZNUkxgXhaKUAS1ANqIgpJgzQbOQFhSxXgPnKzot0gO50nyvQJOyO6fBHMO0ZFefUSpYDbcvJFcdEalCQm51hwjuLwmYER4pQNQhKhlOlsCKORPUuvtfCMJmo3RiSEQ2FpD0aI8ZIpG2ld3u09cw32vwHXEPJMmeY/0xIRJpIR2Tokete7AefJL1i/1XJI3Hkc4sX8vkBNxlMl1EQmJji5FhQhEY9KRIkxLhqfycqANAkXuGIp6U04oI16X+1Z6cNg42N/znG5ufXizvBn264L32HvqNTzf2/J2vA9e29v3cGWpslXZqbyufqv+qP6s/ipC5+eGOY+80qr+/gO+OCZh</latexit> Conditional random fields (CRFs) ■ Decoding: Direct application of Viterbi! ψ includes transition and emission/unary scores M +1 X = argmax ψ ( w , y m , y m − 1 , m ) = s m ( y m , y m − 1 ) ˆ y y m =1 same subproblem! score of best tag sequence M +1 of length M X max y 1: M Ψ ( w , y 1: M ) = max s m ( y m , y m − 1 ) score of best tag sequence y 1: M of length M-1 m =1 M X = max y M s M +1 ( h / s i , y M ) + max s m ( y m , y m − 1 ) y 1: M − 1 score of m =1 most probable extension y M 7

  28. Learning in CRFs 8

  29. <latexit sha1_base64="qyQNuw9LW72v28UgaIfcr0GnkE=">AG6HiclVTdbtMwFM5WKP8bXDJjWGqlrB1NAMJBJo0AUJI06BIbAPNXeW6TmpmJ5HtbKusvAN3iFuehZfgFbiFB8BOuq5pC9Is1Tk9/s53Ph8fu5swKlWz+XNuvnLpcvXKwtXates3bt5aXLq9J+NUYLKLYxaLj10kCaMR2VUMfIxEQTxLiP73aOXdn3/mAhJ4+iDGiSkzVEY0YBipIyrs1Rp1+qwJakLu1yfZGvAfgeZBzYBlCmHjHKqZEfzT871DurfmYBUPWJQhmAuBerPCTIRgyDreT5g3f/ONerd5yC1bIaQ8UsDxBIBDWkJwm7iwJXqZLEnL3iklKIwA5Un2MmP50ltfL/kG0YpkuvMlkD29IXe74XvuNlgFrz1guPtIFfKyzHIiEXJ02hm5ItDcFYEMF6FWn0W/EzqYCjVlquRs1y4IOdcK+dks7Ja5gtUYFN2uDvuNj4xtfR/rMdwzpd9WIlr/0M/ISKGVkm40yUqYUFu5ChKGTkoYQiN/Iwm2t1OpchK2ebkQsSxozORoGjFve2dJaH2qVe6UQLz3NQ3BCvs7jcXG/mA0wb/tBYdoaj1VmaF7AX45STSGpDzwm4lqayQUxYxkNZhKkiB8hEJyYMwIcSLbOn8IMlA3nh4IYmF+kQK5dzxCIy7lgHcN0vaLnFyzlrB6kKnrY1jZJUkQgXiYKUARUD+6qAHhUEKzYwBsKCGq0A95G53sq8PaYzxtL0CTsmqrwRzNtaBn2kqQuz8rBp8VGa1CQiJzgmHMU9R5oGCBO2aBHApQyZd+N4MyeVa+13jFN5LB0I0pGFIwFDWmEGCOBgnYqu82nr2A+1+ArYg5IkB2j+l1CBFKxMEqKa5WZAwvhPdt12f+QNBohjVnels4FmM3YusQJiXRW3HIWSwK7oYjTpCR4Kj4XaghQYI6hwJNyWIEwXepP9uS0sbex7j9a3j/eHnrxbBfF5y7zn3HdXznibPlvHFazq6DKz8qvyq/K3+qn6tfql+r3wro/Nw5o5TGtXvfwFvtGeb</latexit> Learning in CRFs ■ As with logistic regression, weights θ are learned by minimizing negative log likelihood: N log P ( y ( i ) | w ( i ) ; ✓ ) X ` = − i =1 8

  30. <latexit sha1_base64="5MA9J47fV/N83fC3Lg9LbvGCH4=">AHpHiclVRtb9MwEA6jUCivg0+IL4ZpLGFsNIAEAk1CgBDSGAzBXtDcVa7rtBZ2EtnOWGVF4tfwFf4O/wa/tFvTdqBFanM53z3Pc3e2OzmjUjWbf87Mna2dO1+/cLFx6fKVq9euz9/YlkhMNnCGcvEbgdJwmhKthRVjOzmgiDeYWSn8+21Xd85IELSLP2iBjlpcdRLaUIxUsbVnq/daizCTUlD2OH6e/kA2PegjMAagLgkFOlWxrvhaX+3pjOS5tAFR9olAJIO5myqUk5RHCoM3tn+YrsfniUWNxM/SokNMu8GOIBEIa0gO83CWhKjUFQnOvWRIaQogR6qPEdNfR7xReQLQkU6dZH5WPR0QeH6ShyF62AZvI2Awe4j5eWVpcVEosfRYfvIBVnWA6MmgPEuNBZnhY+kDoZSbtWHMqpG3KMtXQMNovVIp+iA2uyzcNxt/HBMby2jp9vGNTprvsV1/sZ8RMqZrBM5pks0wsbHEKG0h4jDyUznBplmt5msuAVdlmcUHCmBG64gOpDfxQGea+DmlUGan3vAD+iESNiWzwn+Pj0e98l9O/wnDN7OfHv0wzW8ARhIVjrE60pM5zXGBgvb6KmpfX2iuNt0Dpo14aCwEw2ezPT8nYDfDBSepwgxJuRc3c9XSCiKGSkbsJAkR/gb6pE9Y6aIE9nS7hIrwaLxdEGSCfNLFXDe8QyNuJQD3jGRtmA5uWads9b2CpU8a2ma5oUiKfZEScGAyoC9EUGXCoIVGxgDYUGNVoD7yFxNytybZg+M0fQJOyCqWgjmLS0Tx16R1OFlNfnQF9qAgqTkO84R2n3voYJ4pQNuiRBVP2zktG9qx+Pege0FwOW3cEyYiCmRkbTRGzA3dTr7rdUP1oG/ANMQMSZMOo/pgTgVQmjBJ/JZRmYD14x56Y8l+RND2KNGa1LO0EmGJsX7KcpLr025RlksBOT2RFXhE8le+EGgCUmDH4eFJN8xFml8aTe3La2H60Gj9efTpycLV8P9eiG4HdwNwiAOngYvg3fBZrAV4NqP2s/ar9rv+r36+/rn+pYPnTszLkZVJ76/l85mKrY</latexit> <latexit sha1_base64="qyQNuw9LW72v28UgaIfcr0GnkE=">AG6HiclVTdbtMwFM5WKP8bXDJjWGqlrB1NAMJBJo0AUJI06BIbAPNXeW6TmpmJ5HtbKusvAN3iFuehZfgFbiFB8BOuq5pC9Is1Tk9/s53Ph8fu5swKlWz+XNuvnLpcvXKwtXates3bt5aXLq9J+NUYLKLYxaLj10kCaMR2VUMfIxEQTxLiP73aOXdn3/mAhJ4+iDGiSkzVEY0YBipIyrs1Rp1+qwJakLu1yfZGvAfgeZBzYBlCmHjHKqZEfzT871DurfmYBUPWJQhmAuBerPCTIRgyDreT5g3f/ONerd5yC1bIaQ8UsDxBIBDWkJwm7iwJXqZLEnL3iklKIwA5Un2MmP50ltfL/kG0YpkuvMlkD29IXe74XvuNlgFrz1guPtIFfKyzHIiEXJ02hm5ItDcFYEMF6FWn0W/EzqYCjVlquRs1y4IOdcK+dks7Ja5gtUYFN2uDvuNj4xtfR/rMdwzpd9WIlr/0M/ISKGVkm40yUqYUFu5ChKGTkoYQiN/Iwm2t1OpchK2ebkQsSxozORoGjFve2dJaH2qVe6UQLz3NQ3BCvs7jcXG/mA0wb/tBYdoaj1VmaF7AX45STSGpDzwm4lqayQUxYxkNZhKkiB8hEJyYMwIcSLbOn8IMlA3nh4IYmF+kQK5dzxCIy7lgHcN0vaLnFyzlrB6kKnrY1jZJUkQgXiYKUARUD+6qAHhUEKzYwBsKCGq0A95G53sq8PaYzxtL0CTsmqrwRzNtaBn2kqQuz8rBp8VGa1CQiJzgmHMU9R5oGCBO2aBHApQyZd+N4MyeVa+13jFN5LB0I0pGFIwFDWmEGCOBgnYqu82nr2A+1+ArYg5IkB2j+l1CBFKxMEqKa5WZAwvhPdt12f+QNBohjVnels4FmM3YusQJiXRW3HIWSwK7oYjTpCR4Kj4XaghQYI6hwJNyWIEwXepP9uS0sbex7j9a3j/eHnrxbBfF5y7zn3HdXznibPlvHFazq6DKz8qvyq/K3+qn6tfql+r3wro/Nw5o5TGtXvfwFvtGeb</latexit> Learning in CRFs ■ As with logistic regression, weights θ are learned by minimizing negative log likelihood: N log P ( y ( i ) | w ( i ) ; ✓ ) X ` = − i =1 N ⇣ ⌘ X X θ · f ( w ( i ) , y ( i ) ) + log θ · f ( w ( i ) , y 0 ) = − exp i =1 y 0 ∈ Y ( w ( i ) ) 8

  31. <latexit sha1_base64="qyQNuw9LW72v28UgaIfcr0GnkE=">AG6HiclVTdbtMwFM5WKP8bXDJjWGqlrB1NAMJBJo0AUJI06BIbAPNXeW6TmpmJ5HtbKusvAN3iFuehZfgFbiFB8BOuq5pC9Is1Tk9/s53Ph8fu5swKlWz+XNuvnLpcvXKwtXates3bt5aXLq9J+NUYLKLYxaLj10kCaMR2VUMfIxEQTxLiP73aOXdn3/mAhJ4+iDGiSkzVEY0YBipIyrs1Rp1+qwJakLu1yfZGvAfgeZBzYBlCmHjHKqZEfzT871DurfmYBUPWJQhmAuBerPCTIRgyDreT5g3f/ONerd5yC1bIaQ8UsDxBIBDWkJwm7iwJXqZLEnL3iklKIwA5Un2MmP50ltfL/kG0YpkuvMlkD29IXe74XvuNlgFrz1guPtIFfKyzHIiEXJ02hm5ItDcFYEMF6FWn0W/EzqYCjVlquRs1y4IOdcK+dks7Ja5gtUYFN2uDvuNj4xtfR/rMdwzpd9WIlr/0M/ISKGVkm40yUqYUFu5ChKGTkoYQiN/Iwm2t1OpchK2ebkQsSxozORoGjFve2dJaH2qVe6UQLz3NQ3BCvs7jcXG/mA0wb/tBYdoaj1VmaF7AX45STSGpDzwm4lqayQUxYxkNZhKkiB8hEJyYMwIcSLbOn8IMlA3nh4IYmF+kQK5dzxCIy7lgHcN0vaLnFyzlrB6kKnrY1jZJUkQgXiYKUARUD+6qAHhUEKzYwBsKCGq0A95G53sq8PaYzxtL0CTsmqrwRzNtaBn2kqQuz8rBp8VGa1CQiJzgmHMU9R5oGCBO2aBHApQyZd+N4MyeVa+13jFN5LB0I0pGFIwFDWmEGCOBgnYqu82nr2A+1+ArYg5IkB2j+l1CBFKxMEqKa5WZAwvhPdt12f+QNBohjVnels4FmM3YusQJiXRW3HIWSwK7oYjTpCR4Kj4XaghQYI6hwJNyWIEwXepP9uS0sbex7j9a3j/eHnrxbBfF5y7zn3HdXznibPlvHFazq6DKz8qvyq/K3+qn6tfql+r3wro/Nw5o5TGtXvfwFvtGeb</latexit> <latexit sha1_base64="5MA9J47fV/N83fC3Lg9LbvGCH4=">AHpHiclVRtb9MwEA6jUCivg0+IL4ZpLGFsNIAEAk1CgBDSGAzBXtDcVa7rtBZ2EtnOWGVF4tfwFf4O/wa/tFvTdqBFanM53z3Pc3e2OzmjUjWbf87Mna2dO1+/cLFx6fKVq9euz9/YlkhMNnCGcvEbgdJwmhKthRVjOzmgiDeYWSn8+21Xd85IELSLP2iBjlpcdRLaUIxUsbVnq/daizCTUlD2OH6e/kA2PegjMAagLgkFOlWxrvhaX+3pjOS5tAFR9olAJIO5myqUk5RHCoM3tn+YrsfniUWNxM/SokNMu8GOIBEIa0gO83CWhKjUFQnOvWRIaQogR6qPEdNfR7xReQLQkU6dZH5WPR0QeH6ShyF62AZvI2Awe4j5eWVpcVEosfRYfvIBVnWA6MmgPEuNBZnhY+kDoZSbtWHMqpG3KMtXQMNovVIp+iA2uyzcNxt/HBMby2jp9vGNTprvsV1/sZ8RMqZrBM5pks0wsbHEKG0h4jDyUznBplmt5msuAVdlmcUHCmBG64gOpDfxQGea+DmlUGan3vAD+iESNiWzwn+Pj0e98l9O/wnDN7OfHv0wzW8ARhIVjrE60pM5zXGBgvb6KmpfX2iuNt0Dpo14aCwEw2ezPT8nYDfDBSepwgxJuRc3c9XSCiKGSkbsJAkR/gb6pE9Y6aIE9nS7hIrwaLxdEGSCfNLFXDe8QyNuJQD3jGRtmA5uWads9b2CpU8a2ma5oUiKfZEScGAyoC9EUGXCoIVGxgDYUGNVoD7yFxNytybZg+M0fQJOyCqWgjmLS0Tx16R1OFlNfnQF9qAgqTkO84R2n3voYJ4pQNuiRBVP2zktG9qx+Pege0FwOW3cEyYiCmRkbTRGzA3dTr7rdUP1oG/ANMQMSZMOo/pgTgVQmjBJ/JZRmYD14x56Y8l+RND2KNGa1LO0EmGJsX7KcpLr025RlksBOT2RFXhE8le+EGgCUmDH4eFJN8xFml8aTe3La2H60Gj9efTpycLV8P9eiG4HdwNwiAOngYvg3fBZrAV4NqP2s/ar9rv+r36+/rn+pYPnTszLkZVJ76/l85mKrY</latexit> Learning in CRFs ■ As with logistic regression, weights θ are learned by minimizing negative log likelihood: N log P ( y ( i ) | w ( i ) ; ✓ ) X ` = − i =1 sum over all possible labelings N ⇣ ⌘ X X θ · f ( w ( i ) , y ( i ) ) + log θ · f ( w ( i ) , y 0 ) = − exp i =1 y 0 ∈ Y ( w ( i ) ) 8

  32. <latexit sha1_base64="qyQNuw9LW72v28UgaIfcr0GnkE=">AG6HiclVTdbtMwFM5WKP8bXDJjWGqlrB1NAMJBJo0AUJI06BIbAPNXeW6TmpmJ5HtbKusvAN3iFuehZfgFbiFB8BOuq5pC9Is1Tk9/s53Ph8fu5swKlWz+XNuvnLpcvXKwtXates3bt5aXLq9J+NUYLKLYxaLj10kCaMR2VUMfIxEQTxLiP73aOXdn3/mAhJ4+iDGiSkzVEY0YBipIyrs1Rp1+qwJakLu1yfZGvAfgeZBzYBlCmHjHKqZEfzT871DurfmYBUPWJQhmAuBerPCTIRgyDreT5g3f/ONerd5yC1bIaQ8UsDxBIBDWkJwm7iwJXqZLEnL3iklKIwA5Un2MmP50ltfL/kG0YpkuvMlkD29IXe74XvuNlgFrz1guPtIFfKyzHIiEXJ02hm5ItDcFYEMF6FWn0W/EzqYCjVlquRs1y4IOdcK+dks7Ja5gtUYFN2uDvuNj4xtfR/rMdwzpd9WIlr/0M/ISKGVkm40yUqYUFu5ChKGTkoYQiN/Iwm2t1OpchK2ebkQsSxozORoGjFve2dJaH2qVe6UQLz3NQ3BCvs7jcXG/mA0wb/tBYdoaj1VmaF7AX45STSGpDzwm4lqayQUxYxkNZhKkiB8hEJyYMwIcSLbOn8IMlA3nh4IYmF+kQK5dzxCIy7lgHcN0vaLnFyzlrB6kKnrY1jZJUkQgXiYKUARUD+6qAHhUEKzYwBsKCGq0A95G53sq8PaYzxtL0CTsmqrwRzNtaBn2kqQuz8rBp8VGa1CQiJzgmHMU9R5oGCBO2aBHApQyZd+N4MyeVa+13jFN5LB0I0pGFIwFDWmEGCOBgnYqu82nr2A+1+ArYg5IkB2j+l1CBFKxMEqKa5WZAwvhPdt12f+QNBohjVnels4FmM3YusQJiXRW3HIWSwK7oYjTpCR4Kj4XaghQYI6hwJNyWIEwXepP9uS0sbex7j9a3j/eHnrxbBfF5y7zn3HdXznibPlvHFazq6DKz8qvyq/K3+qn6tfql+r3wro/Nw5o5TGtXvfwFvtGeb</latexit> <latexit sha1_base64="5MA9J47fV/N83fC3Lg9LbvGCH4=">AHpHiclVRtb9MwEA6jUCivg0+IL4ZpLGFsNIAEAk1CgBDSGAzBXtDcVa7rtBZ2EtnOWGVF4tfwFf4O/wa/tFvTdqBFanM53z3Pc3e2OzmjUjWbf87Mna2dO1+/cLFx6fKVq9euz9/YlkhMNnCGcvEbgdJwmhKthRVjOzmgiDeYWSn8+21Xd85IELSLP2iBjlpcdRLaUIxUsbVnq/daizCTUlD2OH6e/kA2PegjMAagLgkFOlWxrvhaX+3pjOS5tAFR9olAJIO5myqUk5RHCoM3tn+YrsfniUWNxM/SokNMu8GOIBEIa0gO83CWhKjUFQnOvWRIaQogR6qPEdNfR7xReQLQkU6dZH5WPR0QeH6ShyF62AZvI2Awe4j5eWVpcVEosfRYfvIBVnWA6MmgPEuNBZnhY+kDoZSbtWHMqpG3KMtXQMNovVIp+iA2uyzcNxt/HBMby2jp9vGNTprvsV1/sZ8RMqZrBM5pks0wsbHEKG0h4jDyUznBplmt5msuAVdlmcUHCmBG64gOpDfxQGea+DmlUGan3vAD+iESNiWzwn+Pj0e98l9O/wnDN7OfHv0wzW8ARhIVjrE60pM5zXGBgvb6KmpfX2iuNt0Dpo14aCwEw2ezPT8nYDfDBSepwgxJuRc3c9XSCiKGSkbsJAkR/gb6pE9Y6aIE9nS7hIrwaLxdEGSCfNLFXDe8QyNuJQD3jGRtmA5uWads9b2CpU8a2ma5oUiKfZEScGAyoC9EUGXCoIVGxgDYUGNVoD7yFxNytybZg+M0fQJOyCqWgjmLS0Tx16R1OFlNfnQF9qAgqTkO84R2n3voYJ4pQNuiRBVP2zktG9qx+Pege0FwOW3cEyYiCmRkbTRGzA3dTr7rdUP1oG/ANMQMSZMOo/pgTgVQmjBJ/JZRmYD14x56Y8l+RND2KNGa1LO0EmGJsX7KcpLr025RlksBOT2RFXhE8le+EGgCUmDH4eFJN8xFml8aTe3La2H60Gj9efTpycLV8P9eiG4HdwNwiAOngYvg3fBZrAV4NqP2s/ar9rv+r36+/rn+pYPnTszLkZVJ76/l85mKrY</latexit> Learning in CRFs ■ As with logistic regression, weights θ are learned by minimizing negative log likelihood: N log P ( y ( i ) | w ( i ) ; ✓ ) X ` = − i =1 sum over all possible labelings N ⇣ ⌘ X X θ · f ( w ( i ) , y ( i ) ) + log θ · f ( w ( i ) , y 0 ) = − exp i =1 y 0 ∈ Y ( w ( i ) ) ■ Can be computed efficiently using forward algorithm . 8

  33. Learning in CRFs 9

  34. Learning in CRFs ■ Likelihood can be computed efficiently using forward algorithm . 9

  35. <latexit sha1_base64="KhLME4V4Sic9fp1LltTVpr0QY=">AKNXicrVXdbuNEFPZ2YbMJf9vlkpuBqpNtyUuSItAkcqvkFaFItHdRZ2sNbHyezO2NbMpNvI8gvxAjwLF9whbnkFzsw4WTtxCxVYSnJy5nznO38+Myk4U3o4/O3W1u3Xr/Tu9sfvPHmW2+/c2/7/mOVz2VMz+Kc5/LphCjKWUbPNOcPi0kJWLC6ZPJi6/M+ZMLKhXLs5/0oqBjQaYZS1lMNKi7Tu/DHbxqWI+nojyZfUAmd9FaARwmouMGeCaRWVYhRWz8qT/bAyBljPqCYVwnGSawtJq5WHRSTMVykOQvgngsHuqe+8YsES5MwsQSpJXGJ6WfhdIQRV2QrBqveAlGUIC6JnMeHlz0veoLrC0Z7xdOMki4b1ZkL+o4Mw8B+hfRtgMD3jGgXlUZn0ROBbmMVirM8ylaFgE1qzDY7TJfhrqoQzXlOrBeblyQV72XjnrYjWeb1CBkYqE31SDjf8RWX42Ql43ay6O7G17Bfi6KDZR0HKiFMfYxJ9mU048UlawMO1v8kFztpsXVyYcg6BHjhDZgy/bzXzWemzoNVSp/kcuVfExtuCo394fx+WSz3zyZwRfeh+Zu9r2FuAjhNtd9gtaRXc8L7giWbznQwIQXM+LKsnxZmkUtoiGwx5kJkGBVJQBILN1zFY9a0FZDS1knixk8X9vogbuJp5KB1eyF5Z7fW9WCT4KhNYO3+IwnMBxNUoTo7q/Ubg+dWY0K5JmYIq6VsWxg9t2uGw5pPViozI805+Y8jZ53DlUwhm1y1WE9Ov8yhP+V8PprBI2iezvDw6F90KYQ1sKOVz+n0faWxEkezwXNdMyJUufhsNDjkjNYk6rAZ4rWpD4BZnScxAzAi0Zl/ZWrdAuaBKU5hI+mUZW20SURCi1EBOwNC+gWj8zyq6z87lOPx2XLCvmaxI0rnHOkcmSsaJUzSWPMFCSWDGJF8YxANzRc5FCnBs2M8guq24nEYlyq1LK3QpqIqg2+dIkOsKQZfRnQpAs+bDEKRGMLxKakjnX5hJOl3JXvR4kF6xQdelWLjnVOIc1wjLCzQKyW6itkvGrZoB/pCgyQ9gah/KgkOpcQibujKmjYFL9vVnh1nSXLVpYgtMqbQCQjKlLXtCsrNza5LmieDKV+bxoBbyBt4GCA5JCG5w9bcOcBUxpuD6Tm8Ljo8Pw48OjHz/ZOf6ynte73nveB57vhd5D79j7zjv1zry4t9172DvufdH/tf97/4/+n85061aNedrPf2/gYLQpVa</latexit> Learning in CRFs ■ Likelihood can be computed efficiently using forward algorithm . Define: m X X α m ( y m ) = exp s n ( y n , y n − 1 ) n =1 y 1: m − 1 9

  36. <latexit sha1_base64="mezX0sqYlDlGzh2M6vW+sE82j1s=">AKNXicrVXbtGEGWcNorUW9w+5mVbwzBZx67oBkiRQIDTGwoEbl2gzgVehViRS2mT3SWxu3QsEPyh/kC/pQ9K/raX+heKIWUqKRGS0DSaHZmzsyZ4ewkp0Sq4fD3a1vX3n3Ru9mf/De+x98+NGt7Y8fy6wQMT6LM5qJpxMkMSUcnymiKH6aC4zYhOInk5fmPMnF1hIkvFf1DzHY4amnKQkRkqrou0bvw524akPpyw8lV1B5jfeRWAEYCyYJASRpSMSjYKq+flyX5YGQOoZlihCsA4yZR1SatlhHnEzFfJDkL9jwWD3VPfRYWMJMCZWYBUoLiE+DL3u1IqrKVglXvaVDCAWRIzWJEy2cL3KDaEGjPRLpykXnDer0g/9FBGPiPwD74PgA69gwpl15VmZhITBm6jJYqSLMpWJAmiwMdrvMF6nO61QNXQc2ypUJeR1r73WwLlQT+QoMjGTE/KZa62AjXlSG90901HXW3YnlvsN+JYsOlFU/7aW5MY+pIhPKf5CQmEF62aw9texdLA2WhcWxJTqRA+cITGP7a+bz0SdBqdM8AO4Vsfm23MFb3h/nvyDL/bMFbOi+bv5672s3NwEUp8pvoFrQzZj6fYGCTGfK1I9oPkOl8Xb0mSRWRYNiD3gpkIGZMS1A7dEckvkJs9cZMnCzYTpcH1LDuQKkUbOtW6vLt7ZrXQ9WAc4agNYu/8IoueDMCxBXZ3V+o3Bc6sxwVQhM4TVQrYtjF7YNUP1mk+WKjMjzTn7jyNXnQOVTDW2TYT06/zKF/xXwzdcIGEW3doaHQ/uAdSGshR2vfk6j7S0BkywuGOYqpkjK83CYq3GJhCIxdUAFhLnKH6JpvhcixzploxLe6tWYFdrEpBmQn+4Alb9CgRk3LOJtrSvIBy9cwou87OC5V+NS4JzwuFeyA0oIClQFzRYOECBwrOtcCigXRuYJ4hnQ3lL7INU8NmBmF1i1C4nZuJSpRW+lNGFV2/nSFTqAnP8Ks4YQz5vIQpYoTOE5yigipzCacLuYuvO8kFyWVN3TIkxQpmeo0QjqhZQHYLtdV2ybhVM4DfYt0gU901j/lWCVCZ2Ju6Mq3bAp/NSs8OpNloQvLbXYLqu0CehiDC9ZjnlZubVJM4nhZCqyIm8lvOZvE9UBUKrb4Ox281Z6CkNV2dyXh8dBh+eXj0892d46/reb3p3fY+83wv9O5x94P3ql35sW97d693nHvYf+3/h/9P/t/OdOta7XPJ17r6f/9D9AplSI=</latexit> <latexit sha1_base64="KhLME4V4Sic9fp1LltTVpr0QY=">AKNXicrVXdbuNEFPZ2YbMJf9vlkpuBqpNtyUuSItAkcqvkFaFItHdRZ2sNbHyezO2NbMpNvI8gvxAjwLF9whbnkFzsw4WTtxCxVYSnJy5nznO38+Myk4U3o4/O3W1u3Xr/Tu9sfvPHmW2+/c2/7/mOVz2VMz+Kc5/LphCjKWUbPNOcPi0kJWLC6ZPJi6/M+ZMLKhXLs5/0oqBjQaYZS1lMNKi7Tu/DHbxqWI+nojyZfUAmd9FaARwmouMGeCaRWVYhRWz8qT/bAyBljPqCYVwnGSawtJq5WHRSTMVykOQvgngsHuqe+8YsES5MwsQSpJXGJ6WfhdIQRV2QrBqveAlGUIC6JnMeHlz0veoLrC0Z7xdOMki4b1ZkL+o4Mw8B+hfRtgMD3jGgXlUZn0ROBbmMVirM8ylaFgE1qzDY7TJfhrqoQzXlOrBeblyQV72XjnrYjWeb1CBkYqE31SDjf8RWX42Ql43ay6O7G17Bfi6KDZR0HKiFMfYxJ9mU048UlawMO1v8kFztpsXVyYcg6BHjhDZgy/bzXzWemzoNVSp/kcuVfExtuCo394fx+WSz3zyZwRfeh+Zu9r2FuAjhNtd9gtaRXc8L7giWbznQwIQXM+LKsnxZmkUtoiGwx5kJkGBVJQBILN1zFY9a0FZDS1knixk8X9vogbuJp5KB1eyF5Z7fW9WCT4KhNYO3+IwnMBxNUoTo7q/Ubg+dWY0K5JmYIq6VsWxg9t2uGw5pPViozI805+Y8jZ53DlUwhm1y1WE9Ov8yhP+V8PprBI2iezvDw6F90KYQ1sKOVz+n0faWxEkezwXNdMyJUufhsNDjkjNYk6rAZ4rWpD4BZnScxAzAi0Zl/ZWrdAuaBKU5hI+mUZW20SURCi1EBOwNC+gWj8zyq6z87lOPx2XLCvmaxI0rnHOkcmSsaJUzSWPMFCSWDGJF8YxANzRc5FCnBs2M8guq24nEYlyq1LK3QpqIqg2+dIkOsKQZfRnQpAs+bDEKRGMLxKakjnX5hJOl3JXvR4kF6xQdelWLjnVOIc1wjLCzQKyW6itkvGrZoB/pCgyQ9gah/KgkOpcQibujKmjYFL9vVnh1nSXLVpYgtMqbQCQjKlLXtCsrNza5LmieDKV+bxoBbyBt4GCA5JCG5w9bcOcBUxpuD6Tm8Ljo8Pw48OjHz/ZOf6ynte73nveB57vhd5D79j7zjv1zry4t9172DvufdH/tf97/4/+n85061aNedrPf2/gYLQpVa</latexit> Learning in CRFs ■ Likelihood can be computed efficiently using forward algorithm . Define: m m X X X Y α m ( y m ) = exp s n ( y n , y n − 1 ) = exp s n ( y n , y n − 1 ) n =1 n =1 y 1: m − 1 y 1: m − 1 9

  37. <latexit sha1_base64="1jW+4tvq9pLDCdP9LJYNMcCYaQ=">AKNXicrVb9s2Fazra69W7M9oVbERamtRKB3RYSC7YkCRLQOWtkPoCrRE2WxJSiCpNIagP7Q/sN+yh70Ne91fGC+yK9lylgATYPv48JzvfOeiw0lOiVTD4R+3t56+53bvTv9wbvf/Bh3e3P3oqs0LE+CzOaCaeT5DElHB8poi+HkuMGITip9NXn1jzp9dYCFJxn9R8xyPGZpykpIYKa2Ktm/NtiFp5L4cMLK19V9YH7nVQBGAMqCQUoYUTIq2SisXpQn+2FlDKCaYUqAOMkU9YlrZYI84iZr5IdhPofCwa7p75DhYwkwJnZAKlAcQnxZe53UQiqskXBqvd0UMIBZEjNYkTLXxdxg2oD0J5BunGSecN6PSH/yUEY+E/APvg+ABp7hpSjV1UGE4kpQ5fRUgVpNgWLIoBmFQa7XeYLqvOaqinXgUW5cUHeYO29AeuKapBvUIGRjJjfVGsdbOBFZfjliUZdr7o7sbXvsF9h0RFl1U976VoYx9SxKcUP5BQWMG6mVj767E0WDtaVyIKdVED5whMY/tpr5ovRJ0Gqp0zwG7hWxfFvu4D/eH+e/KJb7ZxPY0H3d/PXe125uAihOld+IaoNujqnfFyjIdKZM/ojmM+TqsnhbmlVktomiD3gJkMGZMS1A7eF5MumdbrmIksWfgany/dqEuT6SCPnWbdXJ+/MVroerDM9auNbu01srxlEzwdhWI6Oav1G4PnVmOCqUJmCKuFbFsYvbRrhuo1nyxVZkac/bdeRq97ByqYKy3yabDenSuSeF/DXj1NQJG0d2d4eHQPmBdCGthx6uf02h7S8AkiwuGuYopkvI8HOZqXCKhSExNYCFxDmKX6EpPtciR7ol49LeqhXY1ZoEpJnQH6A1TY9SsSknLOJtjQvoFw9M8qus/NCpV+MS8LzQmEeu0BpQYHKgLmiQUIEjhWdawHFgmiuIJ4h3Q2lL3Jdp0aYGaYXWLUTidm4lKmN3qI0YVXb+dIlOoACc/w6zhDPmshClihM4TnKCKnMJpwu5q173kwuSy7p0S0iKFcz0GiEcUbOA7BZq+2ScatmAL/FukECn2jWP+VYIJUJzcTdUZVu2BR+YlZ4dZUl4UtLbTKi0BnYypS5ZjXlZubdJMYjiZiqzIW4TX/C1RDYBS3QZnj9tuzkJPabg6k+vC06PD8OHh0c+f7x/Xc/rHe+e96ne6H3yDv2fvBOvTMv7m3HvWOe1/1f+/2f+r/7cz3bpV+3zstZ7+P/8CrWyU6g=</latexit> <latexit sha1_base64="KhLME4V4Sic9fp1LltTVpr0QY=">AKNXicrVXdbuNEFPZ2YbMJf9vlkpuBqpNtyUuSItAkcqvkFaFItHdRZ2sNbHyezO2NbMpNvI8gvxAjwLF9whbnkFzsw4WTtxCxVYSnJy5nznO38+Myk4U3o4/O3W1u3Xr/Tu9sfvPHmW2+/c2/7/mOVz2VMz+Kc5/LphCjKWUbPNOcPi0kJWLC6ZPJi6/M+ZMLKhXLs5/0oqBjQaYZS1lMNKi7Tu/DHbxqWI+nojyZfUAmd9FaARwmouMGeCaRWVYhRWz8qT/bAyBljPqCYVwnGSawtJq5WHRSTMVykOQvgngsHuqe+8YsES5MwsQSpJXGJ6WfhdIQRV2QrBqveAlGUIC6JnMeHlz0veoLrC0Z7xdOMki4b1ZkL+o4Mw8B+hfRtgMD3jGgXlUZn0ROBbmMVirM8ylaFgE1qzDY7TJfhrqoQzXlOrBeblyQV72XjnrYjWeb1CBkYqE31SDjf8RWX42Ql43ay6O7G17Bfi6KDZR0HKiFMfYxJ9mU048UlawMO1v8kFztpsXVyYcg6BHjhDZgy/bzXzWemzoNVSp/kcuVfExtuCo394fx+WSz3zyZwRfeh+Zu9r2FuAjhNtd9gtaRXc8L7giWbznQwIQXM+LKsnxZmkUtoiGwx5kJkGBVJQBILN1zFY9a0FZDS1knixk8X9vogbuJp5KB1eyF5Z7fW9WCT4KhNYO3+IwnMBxNUoTo7q/Ubg+dWY0K5JmYIq6VsWxg9t2uGw5pPViozI805+Y8jZ53DlUwhm1y1WE9Ov8yhP+V8PprBI2iezvDw6F90KYQ1sKOVz+n0faWxEkezwXNdMyJUufhsNDjkjNYk6rAZ4rWpD4BZnScxAzAi0Zl/ZWrdAuaBKU5hI+mUZW20SURCi1EBOwNC+gWj8zyq6z87lOPx2XLCvmaxI0rnHOkcmSsaJUzSWPMFCSWDGJF8YxANzRc5FCnBs2M8guq24nEYlyq1LK3QpqIqg2+dIkOsKQZfRnQpAs+bDEKRGMLxKakjnX5hJOl3JXvR4kF6xQdelWLjnVOIc1wjLCzQKyW6itkvGrZoB/pCgyQ9gah/KgkOpcQibujKmjYFL9vVnh1nSXLVpYgtMqbQCQjKlLXtCsrNza5LmieDKV+bxoBbyBt4GCA5JCG5w9bcOcBUxpuD6Tm8Ljo8Pw48OjHz/ZOf6ynte73nveB57vhd5D79j7zjv1zry4t9172DvufdH/tf97/4/+n85061aNedrPf2/gYLQpVa</latexit> <latexit sha1_base64="mezX0sqYlDlGzh2M6vW+sE82j1s=">AKNXicrVXbtGEGWcNorUW9w+5mVbwzBZx67oBkiRQIDTGwoEbl2gzgVehViRS2mT3SWxu3QsEPyh/kC/pQ9K/raX+heKIWUqKRGS0DSaHZmzsyZ4ewkp0Sq4fD3a1vX3n3Ru9mf/De+x98+NGt7Y8fy6wQMT6LM5qJpxMkMSUcnymiKH6aC4zYhOInk5fmPMnF1hIkvFf1DzHY4amnKQkRkqrou0bvw524akPpyw8lV1B5jfeRWAEYCyYJASRpSMSjYKq+flyX5YGQOoZlihCsA4yZR1SatlhHnEzFfJDkL9jwWD3VPfRYWMJMCZWYBUoLiE+DL3u1IqrKVglXvaVDCAWRIzWJEy2cL3KDaEGjPRLpykXnDer0g/9FBGPiPwD74PgA69gwpl15VmZhITBm6jJYqSLMpWJAmiwMdrvMF6nO61QNXQc2ypUJeR1r73WwLlQT+QoMjGTE/KZa62AjXlSG90901HXW3YnlvsN+JYsOlFU/7aW5MY+pIhPKf5CQmEF62aw9texdLA2WhcWxJTqRA+cITGP7a+bz0SdBqdM8AO4Vsfm23MFb3h/nvyDL/bMFbOi+bv5672s3NwEUp8pvoFrQzZj6fYGCTGfK1I9oPkOl8Xb0mSRWRYNiD3gpkIGZMS1A7dEckvkJs9cZMnCzYTpcH1LDuQKkUbOtW6vLt7ZrXQ9WAc4agNYu/8IoueDMCxBXZ3V+o3Bc6sxwVQhM4TVQrYtjF7YNUP1mk+WKjMjzTn7jyNXnQOVTDW2TYT06/zKF/xXwzdcIGEW3doaHQ/uAdSGshR2vfk6j7S0BkywuGOYqpkjK83CYq3GJhCIxdUAFhLnKH6JpvhcixzploxLe6tWYFdrEpBmQn+4Alb9CgRk3LOJtrSvIBy9cwou87OC5V+NS4JzwuFeyA0oIClQFzRYOECBwrOtcCigXRuYJ4hnQ3lL7INU8NmBmF1i1C4nZuJSpRW+lNGFV2/nSFTqAnP8Ks4YQz5vIQpYoTOE5yigipzCacLuYuvO8kFyWVN3TIkxQpmeo0QjqhZQHYLtdV2ybhVM4DfYt0gU901j/lWCVCZ2Ju6Mq3bAp/NSs8OpNloQvLbXYLqu0CehiDC9ZjnlZubVJM4nhZCqyIm8lvOZvE9UBUKrb4Ox281Z6CkNV2dyXh8dBh+eXj0892d46/reb3p3fY+83wv9O5x94P3ql35sW97d693nHvYf+3/h/9P/t/OdOta7XPJ17r6f/9D9AplSI=</latexit> Learning in CRFs ■ Likelihood can be computed efficiently using forward algorithm . Define: m m X X X Y α m ( y m ) = exp s n ( y n , y n − 1 ) = exp s n ( y n , y n − 1 ) n =1 n =1 y 1: m − 1 y 1: m − 1 ■ As in decoding / Viterbi, can be decomposed into recursive substructure: m − 1 X X Y = (exp s m ( y m , y m − 1 )) exp s n ( y n , y n − 1 ) n =1 y m − 1 y 1: m − 2 9

  38. <latexit sha1_base64="1jW+4tvq9pLDCdP9LJYNMcCYaQ=">AKNXicrVb9s2Fazra69W7M9oVbERamtRKB3RYSC7YkCRLQOWtkPoCrRE2WxJSiCpNIagP7Q/sN+yh70Ne91fGC+yK9lylgATYPv48JzvfOeiw0lOiVTD4R+3t56+53bvTv9wbvf/Bh3e3P3oqs0LE+CzOaCaeT5DElHB8poi+HkuMGITip9NXn1jzp9dYCFJxn9R8xyPGZpykpIYKa2Ktm/NtiFp5L4cMLK19V9YH7nVQBGAMqCQUoYUTIq2SisXpQn+2FlDKCaYUqAOMkU9YlrZYI84iZr5IdhPofCwa7p75DhYwkwJnZAKlAcQnxZe53UQiqskXBqvd0UMIBZEjNYkTLXxdxg2oD0J5BunGSecN6PSH/yUEY+E/APvg+ABp7hpSjV1UGE4kpQ5fRUgVpNgWLIoBmFQa7XeYLqvOaqinXgUW5cUHeYO29AeuKapBvUIGRjJjfVGsdbOBFZfjliUZdr7o7sbXvsF9h0RFl1U976VoYx9SxKcUP5BQWMG6mVj767E0WDtaVyIKdVED5whMY/tpr5ovRJ0Gqp0zwG7hWxfFvu4D/eH+e/KJb7ZxPY0H3d/PXe125uAihOld+IaoNujqnfFyjIdKZM/ojmM+TqsnhbmlVktomiD3gJkMGZMS1A7eF5MumdbrmIksWfgany/dqEuT6SCPnWbdXJ+/MVroerDM9auNbu01srxlEzwdhWI6Oav1G4PnVmOCqUJmCKuFbFsYvbRrhuo1nyxVZkac/bdeRq97ByqYKy3yabDenSuSeF/DXj1NQJG0d2d4eHQPmBdCGthx6uf02h7S8AkiwuGuYopkvI8HOZqXCKhSExNYCFxDmKX6EpPtciR7ol49LeqhXY1ZoEpJnQH6A1TY9SsSknLOJtjQvoFw9M8qus/NCpV+MS8LzQmEeu0BpQYHKgLmiQUIEjhWdawHFgmiuIJ4h3Q2lL3Jdp0aYGaYXWLUTidm4lKmN3qI0YVXb+dIlOoACc/w6zhDPmshClihM4TnKCKnMJpwu5q173kwuSy7p0S0iKFcz0GiEcUbOA7BZq+2ScatmAL/FukECn2jWP+VYIJUJzcTdUZVu2BR+YlZ4dZUl4UtLbTKi0BnYypS5ZjXlZubdJMYjiZiqzIW4TX/C1RDYBS3QZnj9tuzkJPabg6k+vC06PD8OHh0c+f7x/Xc/rHe+e96ne6H3yDv2fvBOvTMv7m3HvWOe1/1f+/2f+r/7cz3bpV+3zstZ7+P/8CrWyU6g=</latexit> <latexit sha1_base64="S+C2pUnFb9NeQcjrf+w7RtILE54=">AJH3iclVXdbts2FazJfW8v2a73A27IjcNJmVDtjQIkCxDcOAIlsGLG2H0BVoibKJkpRAUmkMQW/RF9jT7G7YbZ9lNzskZceylS0RYJs+PN/3nR8ecVxwps1w+O7Oxnvb27d7X3Q/Cjz/59N72Z891XqEniU5z9XLMdGUM0nPDOcviwUJWLM6Yvx6+/t/osLqjTL5W9mVtCRIBPJMpYQA6Z4e/Ntfxefahbisaje1A+R/Z3VA3SMsC4F5kwo+NKHEf1q+pkP6qtAzZTakiNcJLmxkGyesEwi4X9qsRBP/EoL97GnpWLFiKvJsTyBRJKkwvi7ArhEFdtUJw5j0QZRJhQcw0Ibz6fa47qK8h2rNMt06yWPJeTyh8dhANwmdoH/04QMA9JcaHV9eWk6iJIJfxwoR5PkHzIqDlKvR3u9znoc6aUG25DhzLrQtyxbV3RdalaplvUYFjHYtw2Qw2vMQXV9HjE2Bdr7rfcbXv8F+JokNlFQcoqIV1DjEncsLpVxort3Awq7W/rgVkbUuLUw5h0APvCOzj+3mvmqCtmg1VJveYL8iLh4W3D0P/Pj8fNi+X8ugWu6D81f730D8yeA08yES6pO9HpNmBes2GRqbP6EF1Pi6zKflqsqsfCVdGKuA1pMxRIxIA0hVSLprWCS1Uns5xlqcLe7MgbsJ07KFNfyF7fS9sG6wFbwPldI3JDTgfTFCNmuScNVwcvPjezvBw6B60voiaxU7QPKfx9obCaZ6UgkqTcKL1eTQszKgiyrCE07qPS0LkrwmE3oOS0lAfFS5+6NGu2BJUZYr+EiDnHUZURGh9UyMwdMeNb26Z41de+elyb4dVUwWpaEy8UJZyZHJkb2MUMoUTQyfwYIkikGsKJkSuBUMXFnQ+SWZKeUX1LQTScSo0plTb4U0FnUbfOkT7WNFJX2T5EIQmT6ocEYE47OUZqTkxl432XzdVa+H6QUrdFO6BSWnBucwMEwSbkfNzVvb7MbJD1Uf/0ChQYqeQNS/FQRkyuIxL+Na2jYBN+3L6v6vzyZXHjCsp1W5QKAZGxd8oLKqvYvCJ5riscTlZdFK+A1vAsUCEgGbfD+tA3zHnBKo9Uzub54fnQYPTo8+vXrnafNe1F3wRfBmEQR8EzwNfgpOg7Mg2fxn6/7Wg6393h+9P3t/9f72rht3GsznQevpvfsXiMUzJw=</latexit> <latexit sha1_base64="mezX0sqYlDlGzh2M6vW+sE82j1s=">AKNXicrVXbtGEGWcNorUW9w+5mVbwzBZx67oBkiRQIDTGwoEbl2gzgVehViRS2mT3SWxu3QsEPyh/kC/pQ9K/raX+heKIWUqKRGS0DSaHZmzsyZ4ewkp0Sq4fD3a1vX3n3Ru9mf/De+x98+NGt7Y8fy6wQMT6LM5qJpxMkMSUcnymiKH6aC4zYhOInk5fmPMnF1hIkvFf1DzHY4amnKQkRkqrou0bvw524akPpyw8lV1B5jfeRWAEYCyYJASRpSMSjYKq+flyX5YGQOoZlihCsA4yZR1SatlhHnEzFfJDkL9jwWD3VPfRYWMJMCZWYBUoLiE+DL3u1IqrKVglXvaVDCAWRIzWJEy2cL3KDaEGjPRLpykXnDer0g/9FBGPiPwD74PgA69gwpl15VmZhITBm6jJYqSLMpWJAmiwMdrvMF6nO61QNXQc2ypUJeR1r73WwLlQT+QoMjGTE/KZa62AjXlSG90901HXW3YnlvsN+JYsOlFU/7aW5MY+pIhPKf5CQmEF62aw9texdLA2WhcWxJTqRA+cITGP7a+bz0SdBqdM8AO4Vsfm23MFb3h/nvyDL/bMFbOi+bv5672s3NwEUp8pvoFrQzZj6fYGCTGfK1I9oPkOl8Xb0mSRWRYNiD3gpkIGZMS1A7dEckvkJs9cZMnCzYTpcH1LDuQKkUbOtW6vLt7ZrXQ9WAc4agNYu/8IoueDMCxBXZ3V+o3Bc6sxwVQhM4TVQrYtjF7YNUP1mk+WKjMjzTn7jyNXnQOVTDW2TYT06/zKF/xXwzdcIGEW3doaHQ/uAdSGshR2vfk6j7S0BkywuGOYqpkjK83CYq3GJhCIxdUAFhLnKH6JpvhcixzploxLe6tWYFdrEpBmQn+4Alb9CgRk3LOJtrSvIBy9cwou87OC5V+NS4JzwuFeyA0oIClQFzRYOECBwrOtcCigXRuYJ4hnQ3lL7INU8NmBmF1i1C4nZuJSpRW+lNGFV2/nSFTqAnP8Ks4YQz5vIQpYoTOE5yigipzCacLuYuvO8kFyWVN3TIkxQpmeo0QjqhZQHYLtdV2ybhVM4DfYt0gU901j/lWCVCZ2Ju6Mq3bAp/NSs8OpNloQvLbXYLqu0CehiDC9ZjnlZubVJM4nhZCqyIm8lvOZvE9UBUKrb4Ox281Z6CkNV2dyXh8dBh+eXj0892d46/reb3p3fY+83wv9O5x94P3ql35sW97d693nHvYf+3/h/9P/t/OdOta7XPJ17r6f/9D9AplSI=</latexit> <latexit sha1_base64="KhLME4V4Sic9fp1LltTVpr0QY=">AKNXicrVXdbuNEFPZ2YbMJf9vlkpuBqpNtyUuSItAkcqvkFaFItHdRZ2sNbHyezO2NbMpNvI8gvxAjwLF9whbnkFzsw4WTtxCxVYSnJy5nznO38+Myk4U3o4/O3W1u3Xr/Tu9sfvPHmW2+/c2/7/mOVz2VMz+Kc5/LphCjKWUbPNOcPi0kJWLC6ZPJi6/M+ZMLKhXLs5/0oqBjQaYZS1lMNKi7Tu/DHbxqWI+nojyZfUAmd9FaARwmouMGeCaRWVYhRWz8qT/bAyBljPqCYVwnGSawtJq5WHRSTMVykOQvgngsHuqe+8YsES5MwsQSpJXGJ6WfhdIQRV2QrBqveAlGUIC6JnMeHlz0veoLrC0Z7xdOMki4b1ZkL+o4Mw8B+hfRtgMD3jGgXlUZn0ROBbmMVirM8ylaFgE1qzDY7TJfhrqoQzXlOrBeblyQV72XjnrYjWeb1CBkYqE31SDjf8RWX42Ql43ay6O7G17Bfi6KDZR0HKiFMfYxJ9mU048UlawMO1v8kFztpsXVyYcg6BHjhDZgy/bzXzWemzoNVSp/kcuVfExtuCo394fx+WSz3zyZwRfeh+Zu9r2FuAjhNtd9gtaRXc8L7giWbznQwIQXM+LKsnxZmkUtoiGwx5kJkGBVJQBILN1zFY9a0FZDS1knixk8X9vogbuJp5KB1eyF5Z7fW9WCT4KhNYO3+IwnMBxNUoTo7q/Ubg+dWY0K5JmYIq6VsWxg9t2uGw5pPViozI805+Y8jZ53DlUwhm1y1WE9Ov8yhP+V8PprBI2iezvDw6F90KYQ1sKOVz+n0faWxEkezwXNdMyJUufhsNDjkjNYk6rAZ4rWpD4BZnScxAzAi0Zl/ZWrdAuaBKU5hI+mUZW20SURCi1EBOwNC+gWj8zyq6z87lOPx2XLCvmaxI0rnHOkcmSsaJUzSWPMFCSWDGJF8YxANzRc5FCnBs2M8guq24nEYlyq1LK3QpqIqg2+dIkOsKQZfRnQpAs+bDEKRGMLxKakjnX5hJOl3JXvR4kF6xQdelWLjnVOIc1wjLCzQKyW6itkvGrZoB/pCgyQ9gah/KgkOpcQibujKmjYFL9vVnh1nSXLVpYgtMqbQCQjKlLXtCsrNza5LmieDKV+bxoBbyBt4GCA5JCG5w9bcOcBUxpuD6Tm8Ljo8Pw48OjHz/ZOf6ynte73nveB57vhd5D79j7zjv1zry4t9172DvufdH/tf97/4/+n85061aNedrPf2/gYLQpVa</latexit> Learning in CRFs ■ Likelihood can be computed efficiently using forward algorithm . Define: m m X X X Y α m ( y m ) = exp s n ( y n , y n − 1 ) = exp s n ( y n , y n − 1 ) n =1 n =1 y 1: m − 1 y 1: m − 1 ■ As in decoding / Viterbi, can be decomposed into recursive substructure: m − 1 X X Y = (exp s m ( y m , y m − 1 )) exp s n ( y n , y n − 1 ) n =1 y m − 1 y 1: m − 2 X = (exp s m ( y m , y m − 1 )) × α m − 1 ( y m − 1 ) y m − 1 9

  39. <latexit sha1_base64="1jW+4tvq9pLDCdP9LJYNMcCYaQ=">AKNXicrVb9s2Fazra69W7M9oVbERamtRKB3RYSC7YkCRLQOWtkPoCrRE2WxJSiCpNIagP7Q/sN+yh70Ne91fGC+yK9lylgATYPv48JzvfOeiw0lOiVTD4R+3t56+53bvTv9wbvf/Bh3e3P3oqs0LE+CzOaCaeT5DElHB8poi+HkuMGITip9NXn1jzp9dYCFJxn9R8xyPGZpykpIYKa2Ktm/NtiFp5L4cMLK19V9YH7nVQBGAMqCQUoYUTIq2SisXpQn+2FlDKCaYUqAOMkU9YlrZYI84iZr5IdhPofCwa7p75DhYwkwJnZAKlAcQnxZe53UQiqskXBqvd0UMIBZEjNYkTLXxdxg2oD0J5BunGSecN6PSH/yUEY+E/APvg+ABp7hpSjV1UGE4kpQ5fRUgVpNgWLIoBmFQa7XeYLqvOaqinXgUW5cUHeYO29AeuKapBvUIGRjJjfVGsdbOBFZfjliUZdr7o7sbXvsF9h0RFl1U976VoYx9SxKcUP5BQWMG6mVj767E0WDtaVyIKdVED5whMY/tpr5ovRJ0Gqp0zwG7hWxfFvu4D/eH+e/KJb7ZxPY0H3d/PXe125uAihOld+IaoNujqnfFyjIdKZM/ojmM+TqsnhbmlVktomiD3gJkMGZMS1A7eF5MumdbrmIksWfgany/dqEuT6SCPnWbdXJ+/MVroerDM9auNbu01srxlEzwdhWI6Oav1G4PnVmOCqUJmCKuFbFsYvbRrhuo1nyxVZkac/bdeRq97ByqYKy3yabDenSuSeF/DXj1NQJG0d2d4eHQPmBdCGthx6uf02h7S8AkiwuGuYopkvI8HOZqXCKhSExNYCFxDmKX6EpPtciR7ol49LeqhXY1ZoEpJnQH6A1TY9SsSknLOJtjQvoFw9M8qus/NCpV+MS8LzQmEeu0BpQYHKgLmiQUIEjhWdawHFgmiuIJ4h3Q2lL3Jdp0aYGaYXWLUTidm4lKmN3qI0YVXb+dIlOoACc/w6zhDPmshClihM4TnKCKnMJpwu5q173kwuSy7p0S0iKFcz0GiEcUbOA7BZq+2ScatmAL/FukECn2jWP+VYIJUJzcTdUZVu2BR+YlZ4dZUl4UtLbTKi0BnYypS5ZjXlZubdJMYjiZiqzIW4TX/C1RDYBS3QZnj9tuzkJPabg6k+vC06PD8OHh0c+f7x/Xc/rHe+e96ne6H3yDv2fvBOvTMv7m3HvWOe1/1f+/2f+r/7cz3bpV+3zstZ7+P/8CrWyU6g=</latexit> <latexit sha1_base64="S+C2pUnFb9NeQcjrf+w7RtILE54=">AJH3iclVXdbts2FazJfW8v2a73A27IjcNJmVDtjQIkCxDcOAIlsGLG2H0BVoibKJkpRAUmkMQW/RF9jT7G7YbZ9lNzskZceylS0RYJs+PN/3nR8ecVxwps1w+O7Oxnvb27d7X3Q/Cjz/59N72Z891XqEniU5z9XLMdGUM0nPDOcviwUJWLM6Yvx6+/t/osLqjTL5W9mVtCRIBPJMpYQA6Z4e/Ntfxefahbisaje1A+R/Z3VA3SMsC4F5kwo+NKHEf1q+pkP6qtAzZTakiNcJLmxkGyesEwi4X9qsRBP/EoL97GnpWLFiKvJsTyBRJKkwvi7ArhEFdtUJw5j0QZRJhQcw0Ibz6fa47qK8h2rNMt06yWPJeTyh8dhANwmdoH/04QMA9JcaHV9eWk6iJIJfxwoR5PkHzIqDlKvR3u9znoc6aUG25DhzLrQtyxbV3RdalaplvUYFjHYtw2Qw2vMQXV9HjE2Bdr7rfcbXv8F+JokNlFQcoqIV1DjEncsLpVxort3Awq7W/rgVkbUuLUw5h0APvCOzj+3mvmqCtmg1VJveYL8iLh4W3D0P/Pj8fNi+X8ugWu6D81f730D8yeA08yES6pO9HpNmBes2GRqbP6EF1Pi6zKflqsqsfCVdGKuA1pMxRIxIA0hVSLprWCS1Uns5xlqcLe7MgbsJ07KFNfyF7fS9sG6wFbwPldI3JDTgfTFCNmuScNVwcvPjezvBw6B60voiaxU7QPKfx9obCaZ6UgkqTcKL1eTQszKgiyrCE07qPS0LkrwmE3oOS0lAfFS5+6NGu2BJUZYr+EiDnHUZURGh9UyMwdMeNb26Z41de+elyb4dVUwWpaEy8UJZyZHJkb2MUMoUTQyfwYIkikGsKJkSuBUMXFnQ+SWZKeUX1LQTScSo0plTb4U0FnUbfOkT7WNFJX2T5EIQmT6ocEYE47OUZqTkxl432XzdVa+H6QUrdFO6BSWnBucwMEwSbkfNzVvb7MbJD1Uf/0ChQYqeQNS/FQRkyuIxL+Na2jYBN+3L6v6vzyZXHjCsp1W5QKAZGxd8oLKqvYvCJ5riscTlZdFK+A1vAsUCEgGbfD+tA3zHnBKo9Uzub54fnQYPTo8+vXrnafNe1F3wRfBmEQR8EzwNfgpOg7Mg2fxn6/7Wg6393h+9P3t/9f72rht3GsznQevpvfsXiMUzJw=</latexit> <latexit sha1_base64="mezX0sqYlDlGzh2M6vW+sE82j1s=">AKNXicrVXbtGEGWcNorUW9w+5mVbwzBZx67oBkiRQIDTGwoEbl2gzgVehViRS2mT3SWxu3QsEPyh/kC/pQ9K/raX+heKIWUqKRGS0DSaHZmzsyZ4ewkp0Sq4fD3a1vX3n3Ru9mf/De+x98+NGt7Y8fy6wQMT6LM5qJpxMkMSUcnymiKH6aC4zYhOInk5fmPMnF1hIkvFf1DzHY4amnKQkRkqrou0bvw524akPpyw8lV1B5jfeRWAEYCyYJASRpSMSjYKq+flyX5YGQOoZlihCsA4yZR1SatlhHnEzFfJDkL9jwWD3VPfRYWMJMCZWYBUoLiE+DL3u1IqrKVglXvaVDCAWRIzWJEy2cL3KDaEGjPRLpykXnDer0g/9FBGPiPwD74PgA69gwpl15VmZhITBm6jJYqSLMpWJAmiwMdrvMF6nO61QNXQc2ypUJeR1r73WwLlQT+QoMjGTE/KZa62AjXlSG90901HXW3YnlvsN+JYsOlFU/7aW5MY+pIhPKf5CQmEF62aw9texdLA2WhcWxJTqRA+cITGP7a+bz0SdBqdM8AO4Vsfm23MFb3h/nvyDL/bMFbOi+bv5672s3NwEUp8pvoFrQzZj6fYGCTGfK1I9oPkOl8Xb0mSRWRYNiD3gpkIGZMS1A7dEckvkJs9cZMnCzYTpcH1LDuQKkUbOtW6vLt7ZrXQ9WAc4agNYu/8IoueDMCxBXZ3V+o3Bc6sxwVQhM4TVQrYtjF7YNUP1mk+WKjMjzTn7jyNXnQOVTDW2TYT06/zKF/xXwzdcIGEW3doaHQ/uAdSGshR2vfk6j7S0BkywuGOYqpkjK83CYq3GJhCIxdUAFhLnKH6JpvhcixzploxLe6tWYFdrEpBmQn+4Alb9CgRk3LOJtrSvIBy9cwou87OC5V+NS4JzwuFeyA0oIClQFzRYOECBwrOtcCigXRuYJ4hnQ3lL7INU8NmBmF1i1C4nZuJSpRW+lNGFV2/nSFTqAnP8Ks4YQz5vIQpYoTOE5yigipzCacLuYuvO8kFyWVN3TIkxQpmeo0QjqhZQHYLtdV2ybhVM4DfYt0gU901j/lWCVCZ2Ju6Mq3bAp/NSs8OpNloQvLbXYLqu0CehiDC9ZjnlZubVJM4nhZCqyIm8lvOZvE9UBUKrb4Ox281Z6CkNV2dyXh8dBh+eXj0892d46/reb3p3fY+83wv9O5x94P3ql35sW97d693nHvYf+3/h/9P/t/OdOta7XPJ17r6f/9D9AplSI=</latexit> <latexit sha1_base64="KhLME4V4Sic9fp1LltTVpr0QY=">AKNXicrVXdbuNEFPZ2YbMJf9vlkpuBqpNtyUuSItAkcqvkFaFItHdRZ2sNbHyezO2NbMpNvI8gvxAjwLF9whbnkFzsw4WTtxCxVYSnJy5nznO38+Myk4U3o4/O3W1u3Xr/Tu9sfvPHmW2+/c2/7/mOVz2VMz+Kc5/LphCjKWUbPNOcPi0kJWLC6ZPJi6/M+ZMLKhXLs5/0oqBjQaYZS1lMNKi7Tu/DHbxqWI+nojyZfUAmd9FaARwmouMGeCaRWVYhRWz8qT/bAyBljPqCYVwnGSawtJq5WHRSTMVykOQvgngsHuqe+8YsES5MwsQSpJXGJ6WfhdIQRV2QrBqveAlGUIC6JnMeHlz0veoLrC0Z7xdOMki4b1ZkL+o4Mw8B+hfRtgMD3jGgXlUZn0ROBbmMVirM8ylaFgE1qzDY7TJfhrqoQzXlOrBeblyQV72XjnrYjWeb1CBkYqE31SDjf8RWX42Ql43ay6O7G17Bfi6KDZR0HKiFMfYxJ9mU048UlawMO1v8kFztpsXVyYcg6BHjhDZgy/bzXzWemzoNVSp/kcuVfExtuCo394fx+WSz3zyZwRfeh+Zu9r2FuAjhNtd9gtaRXc8L7giWbznQwIQXM+LKsnxZmkUtoiGwx5kJkGBVJQBILN1zFY9a0FZDS1knixk8X9vogbuJp5KB1eyF5Z7fW9WCT4KhNYO3+IwnMBxNUoTo7q/Ubg+dWY0K5JmYIq6VsWxg9t2uGw5pPViozI805+Y8jZ53DlUwhm1y1WE9Ov8yhP+V8PprBI2iezvDw6F90KYQ1sKOVz+n0faWxEkezwXNdMyJUufhsNDjkjNYk6rAZ4rWpD4BZnScxAzAi0Zl/ZWrdAuaBKU5hI+mUZW20SURCi1EBOwNC+gWj8zyq6z87lOPx2XLCvmaxI0rnHOkcmSsaJUzSWPMFCSWDGJF8YxANzRc5FCnBs2M8guq24nEYlyq1LK3QpqIqg2+dIkOsKQZfRnQpAs+bDEKRGMLxKakjnX5hJOl3JXvR4kF6xQdelWLjnVOIc1wjLCzQKyW6itkvGrZoB/pCgyQ9gah/KgkOpcQibujKmjYFL9vVnh1nSXLVpYgtMqbQCQjKlLXtCsrNza5LmieDKV+bxoBbyBt4GCA5JCG5w9bcOcBUxpuD6Tm8Ljo8Pw48OjHz/ZOf6ynte73nveB57vhd5D79j7zjv1zry4t9172DvufdH/tf97/4/+n85061aNedrPf2/gYLQpVa</latexit> Learning in CRFs ■ Likelihood can be computed efficiently using forward algorithm . Define: m m X X X Y α m ( y m ) = exp s n ( y n , y n − 1 ) = exp s n ( y n , y n − 1 ) n =1 n =1 y 1: m − 1 y 1: m − 1 ■ As in decoding / Viterbi, can be decomposed into recursive substructure: m − 1 X X Y = (exp s m ( y m , y m − 1 )) exp s n ( y n , y n − 1 ) n =1 y m − 1 y 1: m − 2 X = (exp s m ( y m , y m − 1 )) × α m − 1 ( y m − 1 ) y m − 1 sum instead of max 9

  40. <latexit sha1_base64="1jW+4tvq9pLDCdP9LJYNMcCYaQ=">AKNXicrVb9s2Fazra69W7M9oVbERamtRKB3RYSC7YkCRLQOWtkPoCrRE2WxJSiCpNIagP7Q/sN+yh70Ne91fGC+yK9lylgATYPv48JzvfOeiw0lOiVTD4R+3t56+53bvTv9wbvf/Bh3e3P3oqs0LE+CzOaCaeT5DElHB8poi+HkuMGITip9NXn1jzp9dYCFJxn9R8xyPGZpykpIYKa2Ktm/NtiFp5L4cMLK19V9YH7nVQBGAMqCQUoYUTIq2SisXpQn+2FlDKCaYUqAOMkU9YlrZYI84iZr5IdhPofCwa7p75DhYwkwJnZAKlAcQnxZe53UQiqskXBqvd0UMIBZEjNYkTLXxdxg2oD0J5BunGSecN6PSH/yUEY+E/APvg+ABp7hpSjV1UGE4kpQ5fRUgVpNgWLIoBmFQa7XeYLqvOaqinXgUW5cUHeYO29AeuKapBvUIGRjJjfVGsdbOBFZfjliUZdr7o7sbXvsF9h0RFl1U976VoYx9SxKcUP5BQWMG6mVj767E0WDtaVyIKdVED5whMY/tpr5ovRJ0Gqp0zwG7hWxfFvu4D/eH+e/KJb7ZxPY0H3d/PXe125uAihOld+IaoNujqnfFyjIdKZM/ojmM+TqsnhbmlVktomiD3gJkMGZMS1A7eF5MumdbrmIksWfgany/dqEuT6SCPnWbdXJ+/MVroerDM9auNbu01srxlEzwdhWI6Oav1G4PnVmOCqUJmCKuFbFsYvbRrhuo1nyxVZkac/bdeRq97ByqYKy3yabDenSuSeF/DXj1NQJG0d2d4eHQPmBdCGthx6uf02h7S8AkiwuGuYopkvI8HOZqXCKhSExNYCFxDmKX6EpPtciR7ol49LeqhXY1ZoEpJnQH6A1TY9SsSknLOJtjQvoFw9M8qus/NCpV+MS8LzQmEeu0BpQYHKgLmiQUIEjhWdawHFgmiuIJ4h3Q2lL3Jdp0aYGaYXWLUTidm4lKmN3qI0YVXb+dIlOoACc/w6zhDPmshClihM4TnKCKnMJpwu5q173kwuSy7p0S0iKFcz0GiEcUbOA7BZq+2ScatmAL/FukECn2jWP+VYIJUJzcTdUZVu2BR+YlZ4dZUl4UtLbTKi0BnYypS5ZjXlZubdJMYjiZiqzIW4TX/C1RDYBS3QZnj9tuzkJPabg6k+vC06PD8OHh0c+f7x/Xc/rHe+e96ne6H3yDv2fvBOvTMv7m3HvWOe1/1f+/2f+r/7cz3bpV+3zstZ7+P/8CrWyU6g=</latexit> <latexit sha1_base64="S+C2pUnFb9NeQcjrf+w7RtILE54=">AJH3iclVXdbts2FazJfW8v2a73A27IjcNJmVDtjQIkCxDcOAIlsGLG2H0BVoibKJkpRAUmkMQW/RF9jT7G7YbZ9lNzskZceylS0RYJs+PN/3nR8ecVxwps1w+O7Oxnvb27d7X3Q/Cjz/59N72Z891XqEniU5z9XLMdGUM0nPDOcviwUJWLM6Yvx6+/t/osLqjTL5W9mVtCRIBPJMpYQA6Z4e/Ntfxefahbisaje1A+R/Z3VA3SMsC4F5kwo+NKHEf1q+pkP6qtAzZTakiNcJLmxkGyesEwi4X9qsRBP/EoL97GnpWLFiKvJsTyBRJKkwvi7ArhEFdtUJw5j0QZRJhQcw0Ibz6fa47qK8h2rNMt06yWPJeTyh8dhANwmdoH/04QMA9JcaHV9eWk6iJIJfxwoR5PkHzIqDlKvR3u9znoc6aUG25DhzLrQtyxbV3RdalaplvUYFjHYtw2Qw2vMQXV9HjE2Bdr7rfcbXv8F+JokNlFQcoqIV1DjEncsLpVxort3Awq7W/rgVkbUuLUw5h0APvCOzj+3mvmqCtmg1VJveYL8iLh4W3D0P/Pj8fNi+X8ugWu6D81f730D8yeA08yES6pO9HpNmBes2GRqbP6EF1Pi6zKflqsqsfCVdGKuA1pMxRIxIA0hVSLprWCS1Uns5xlqcLe7MgbsJ07KFNfyF7fS9sG6wFbwPldI3JDTgfTFCNmuScNVwcvPjezvBw6B60voiaxU7QPKfx9obCaZ6UgkqTcKL1eTQszKgiyrCE07qPS0LkrwmE3oOS0lAfFS5+6NGu2BJUZYr+EiDnHUZURGh9UyMwdMeNb26Z41de+elyb4dVUwWpaEy8UJZyZHJkb2MUMoUTQyfwYIkikGsKJkSuBUMXFnQ+SWZKeUX1LQTScSo0plTb4U0FnUbfOkT7WNFJX2T5EIQmT6ocEYE47OUZqTkxl432XzdVa+H6QUrdFO6BSWnBucwMEwSbkfNzVvb7MbJD1Uf/0ChQYqeQNS/FQRkyuIxL+Na2jYBN+3L6v6vzyZXHjCsp1W5QKAZGxd8oLKqvYvCJ5riscTlZdFK+A1vAsUCEgGbfD+tA3zHnBKo9Uzub54fnQYPTo8+vXrnafNe1F3wRfBmEQR8EzwNfgpOg7Mg2fxn6/7Wg6393h+9P3t/9f72rht3GsznQevpvfsXiMUzJw=</latexit> <latexit sha1_base64="mezX0sqYlDlGzh2M6vW+sE82j1s=">AKNXicrVXbtGEGWcNorUW9w+5mVbwzBZx67oBkiRQIDTGwoEbl2gzgVehViRS2mT3SWxu3QsEPyh/kC/pQ9K/raX+heKIWUqKRGS0DSaHZmzsyZ4ewkp0Sq4fD3a1vX3n3Ru9mf/De+x98+NGt7Y8fy6wQMT6LM5qJpxMkMSUcnymiKH6aC4zYhOInk5fmPMnF1hIkvFf1DzHY4amnKQkRkqrou0bvw524akPpyw8lV1B5jfeRWAEYCyYJASRpSMSjYKq+flyX5YGQOoZlihCsA4yZR1SatlhHnEzFfJDkL9jwWD3VPfRYWMJMCZWYBUoLiE+DL3u1IqrKVglXvaVDCAWRIzWJEy2cL3KDaEGjPRLpykXnDer0g/9FBGPiPwD74PgA69gwpl15VmZhITBm6jJYqSLMpWJAmiwMdrvMF6nO61QNXQc2ypUJeR1r73WwLlQT+QoMjGTE/KZa62AjXlSG90901HXW3YnlvsN+JYsOlFU/7aW5MY+pIhPKf5CQmEF62aw9texdLA2WhcWxJTqRA+cITGP7a+bz0SdBqdM8AO4Vsfm23MFb3h/nvyDL/bMFbOi+bv5672s3NwEUp8pvoFrQzZj6fYGCTGfK1I9oPkOl8Xb0mSRWRYNiD3gpkIGZMS1A7dEckvkJs9cZMnCzYTpcH1LDuQKkUbOtW6vLt7ZrXQ9WAc4agNYu/8IoueDMCxBXZ3V+o3Bc6sxwVQhM4TVQrYtjF7YNUP1mk+WKjMjzTn7jyNXnQOVTDW2TYT06/zKF/xXwzdcIGEW3doaHQ/uAdSGshR2vfk6j7S0BkywuGOYqpkjK83CYq3GJhCIxdUAFhLnKH6JpvhcixzploxLe6tWYFdrEpBmQn+4Alb9CgRk3LOJtrSvIBy9cwou87OC5V+NS4JzwuFeyA0oIClQFzRYOECBwrOtcCigXRuYJ4hnQ3lL7INU8NmBmF1i1C4nZuJSpRW+lNGFV2/nSFTqAnP8Ks4YQz5vIQpYoTOE5yigipzCacLuYuvO8kFyWVN3TIkxQpmeo0QjqhZQHYLtdV2ybhVM4DfYt0gU901j/lWCVCZ2Ju6Mq3bAp/NSs8OpNloQvLbXYLqu0CehiDC9ZjnlZubVJM4nhZCqyIm8lvOZvE9UBUKrb4Ox281Z6CkNV2dyXh8dBh+eXj0892d46/reb3p3fY+83wv9O5x94P3ql35sW97d693nHvYf+3/h/9P/t/OdOta7XPJ17r6f/9D9AplSI=</latexit> <latexit sha1_base64="KhLME4V4Sic9fp1LltTVpr0QY=">AKNXicrVXdbuNEFPZ2YbMJf9vlkpuBqpNtyUuSItAkcqvkFaFItHdRZ2sNbHyezO2NbMpNvI8gvxAjwLF9whbnkFzsw4WTtxCxVYSnJy5nznO38+Myk4U3o4/O3W1u3Xr/Tu9sfvPHmW2+/c2/7/mOVz2VMz+Kc5/LphCjKWUbPNOcPi0kJWLC6ZPJi6/M+ZMLKhXLs5/0oqBjQaYZS1lMNKi7Tu/DHbxqWI+nojyZfUAmd9FaARwmouMGeCaRWVYhRWz8qT/bAyBljPqCYVwnGSawtJq5WHRSTMVykOQvgngsHuqe+8YsES5MwsQSpJXGJ6WfhdIQRV2QrBqveAlGUIC6JnMeHlz0veoLrC0Z7xdOMki4b1ZkL+o4Mw8B+hfRtgMD3jGgXlUZn0ROBbmMVirM8ylaFgE1qzDY7TJfhrqoQzXlOrBeblyQV72XjnrYjWeb1CBkYqE31SDjf8RWX42Ql43ay6O7G17Bfi6KDZR0HKiFMfYxJ9mU048UlawMO1v8kFztpsXVyYcg6BHjhDZgy/bzXzWemzoNVSp/kcuVfExtuCo394fx+WSz3zyZwRfeh+Zu9r2FuAjhNtd9gtaRXc8L7giWbznQwIQXM+LKsnxZmkUtoiGwx5kJkGBVJQBILN1zFY9a0FZDS1knixk8X9vogbuJp5KB1eyF5Z7fW9WCT4KhNYO3+IwnMBxNUoTo7q/Ubg+dWY0K5JmYIq6VsWxg9t2uGw5pPViozI805+Y8jZ53DlUwhm1y1WE9Ov8yhP+V8PprBI2iezvDw6F90KYQ1sKOVz+n0faWxEkezwXNdMyJUufhsNDjkjNYk6rAZ4rWpD4BZnScxAzAi0Zl/ZWrdAuaBKU5hI+mUZW20SURCi1EBOwNC+gWj8zyq6z87lOPx2XLCvmaxI0rnHOkcmSsaJUzSWPMFCSWDGJF8YxANzRc5FCnBs2M8guq24nEYlyq1LK3QpqIqg2+dIkOsKQZfRnQpAs+bDEKRGMLxKakjnX5hJOl3JXvR4kF6xQdelWLjnVOIc1wjLCzQKyW6itkvGrZoB/pCgyQ9gah/KgkOpcQibujKmjYFL9vVnh1nSXLVpYgtMqbQCQjKlLXtCsrNza5LmieDKV+bxoBbyBt4GCA5JCG5w9bcOcBUxpuD6Tm8Ljo8Pw48OjHz/ZOf6ynte73nveB57vhd5D79j7zjv1zry4t9172DvufdH/tf97/4/+n85061aNedrPf2/gYLQpVa</latexit> Learning in CRFs ■ Likelihood can be computed efficiently using forward algorithm . Define: m m X X X Y α m ( y m ) = exp s n ( y n , y n − 1 ) = exp s n ( y n , y n − 1 ) n =1 n =1 y 1: m − 1 y 1: m − 1 ■ As in decoding / Viterbi, can be decomposed into recursive substructure: m − 1 X X Y = (exp s m ( y m , y m − 1 )) exp s n ( y n , y n − 1 ) n =1 y m − 1 y 1: m − 2 Viterbi is a special case of the max-product algorithm , X = (exp s m ( y m , y m − 1 )) × α m − 1 ( y m − 1 ) forward is a special case of the sum-product algorithm . y m − 1 sum instead of max 9

  41. <latexit sha1_base64="1jW+4tvq9pLDCdP9LJYNMcCYaQ=">AKNXicrVb9s2Fazra69W7M9oVbERamtRKB3RYSC7YkCRLQOWtkPoCrRE2WxJSiCpNIagP7Q/sN+yh70Ne91fGC+yK9lylgATYPv48JzvfOeiw0lOiVTD4R+3t56+53bvTv9wbvf/Bh3e3P3oqs0LE+CzOaCaeT5DElHB8poi+HkuMGITip9NXn1jzp9dYCFJxn9R8xyPGZpykpIYKa2Ktm/NtiFp5L4cMLK19V9YH7nVQBGAMqCQUoYUTIq2SisXpQn+2FlDKCaYUqAOMkU9YlrZYI84iZr5IdhPofCwa7p75DhYwkwJnZAKlAcQnxZe53UQiqskXBqvd0UMIBZEjNYkTLXxdxg2oD0J5BunGSecN6PSH/yUEY+E/APvg+ABp7hpSjV1UGE4kpQ5fRUgVpNgWLIoBmFQa7XeYLqvOaqinXgUW5cUHeYO29AeuKapBvUIGRjJjfVGsdbOBFZfjliUZdr7o7sbXvsF9h0RFl1U976VoYx9SxKcUP5BQWMG6mVj767E0WDtaVyIKdVED5whMY/tpr5ovRJ0Gqp0zwG7hWxfFvu4D/eH+e/KJb7ZxPY0H3d/PXe125uAihOld+IaoNujqnfFyjIdKZM/ojmM+TqsnhbmlVktomiD3gJkMGZMS1A7eF5MumdbrmIksWfgany/dqEuT6SCPnWbdXJ+/MVroerDM9auNbu01srxlEzwdhWI6Oav1G4PnVmOCqUJmCKuFbFsYvbRrhuo1nyxVZkac/bdeRq97ByqYKy3yabDenSuSeF/DXj1NQJG0d2d4eHQPmBdCGthx6uf02h7S8AkiwuGuYopkvI8HOZqXCKhSExNYCFxDmKX6EpPtciR7ol49LeqhXY1ZoEpJnQH6A1TY9SsSknLOJtjQvoFw9M8qus/NCpV+MS8LzQmEeu0BpQYHKgLmiQUIEjhWdawHFgmiuIJ4h3Q2lL3Jdp0aYGaYXWLUTidm4lKmN3qI0YVXb+dIlOoACc/w6zhDPmshClihM4TnKCKnMJpwu5q173kwuSy7p0S0iKFcz0GiEcUbOA7BZq+2ScatmAL/FukECn2jWP+VYIJUJzcTdUZVu2BR+YlZ4dZUl4UtLbTKi0BnYypS5ZjXlZubdJMYjiZiqzIW4TX/C1RDYBS3QZnj9tuzkJPabg6k+vC06PD8OHh0c+f7x/Xc/rHe+e96ne6H3yDv2fvBOvTMv7m3HvWOe1/1f+/2f+r/7cz3bpV+3zstZ7+P/8CrWyU6g=</latexit> <latexit sha1_base64="S+C2pUnFb9NeQcjrf+w7RtILE54=">AJH3iclVXdbts2FazJfW8v2a73A27IjcNJmVDtjQIkCxDcOAIlsGLG2H0BVoibKJkpRAUmkMQW/RF9jT7G7YbZ9lNzskZceylS0RYJs+PN/3nR8ecVxwps1w+O7Oxnvb27d7X3Q/Cjz/59N72Z891XqEniU5z9XLMdGUM0nPDOcviwUJWLM6Yvx6+/t/osLqjTL5W9mVtCRIBPJMpYQA6Z4e/Ntfxefahbisaje1A+R/Z3VA3SMsC4F5kwo+NKHEf1q+pkP6qtAzZTakiNcJLmxkGyesEwi4X9qsRBP/EoL97GnpWLFiKvJsTyBRJKkwvi7ArhEFdtUJw5j0QZRJhQcw0Ibz6fa47qK8h2rNMt06yWPJeTyh8dhANwmdoH/04QMA9JcaHV9eWk6iJIJfxwoR5PkHzIqDlKvR3u9znoc6aUG25DhzLrQtyxbV3RdalaplvUYFjHYtw2Qw2vMQXV9HjE2Bdr7rfcbXv8F+JokNlFQcoqIV1DjEncsLpVxort3Awq7W/rgVkbUuLUw5h0APvCOzj+3mvmqCtmg1VJveYL8iLh4W3D0P/Pj8fNi+X8ugWu6D81f730D8yeA08yES6pO9HpNmBes2GRqbP6EF1Pi6zKflqsqsfCVdGKuA1pMxRIxIA0hVSLprWCS1Uns5xlqcLe7MgbsJ07KFNfyF7fS9sG6wFbwPldI3JDTgfTFCNmuScNVwcvPjezvBw6B60voiaxU7QPKfx9obCaZ6UgkqTcKL1eTQszKgiyrCE07qPS0LkrwmE3oOS0lAfFS5+6NGu2BJUZYr+EiDnHUZURGh9UyMwdMeNb26Z41de+elyb4dVUwWpaEy8UJZyZHJkb2MUMoUTQyfwYIkikGsKJkSuBUMXFnQ+SWZKeUX1LQTScSo0plTb4U0FnUbfOkT7WNFJX2T5EIQmT6ocEYE47OUZqTkxl432XzdVa+H6QUrdFO6BSWnBucwMEwSbkfNzVvb7MbJD1Uf/0ChQYqeQNS/FQRkyuIxL+Na2jYBN+3L6v6vzyZXHjCsp1W5QKAZGxd8oLKqvYvCJ5riscTlZdFK+A1vAsUCEgGbfD+tA3zHnBKo9Uzub54fnQYPTo8+vXrnafNe1F3wRfBmEQR8EzwNfgpOg7Mg2fxn6/7Wg6393h+9P3t/9f72rht3GsznQevpvfsXiMUzJw=</latexit> <latexit sha1_base64="mezX0sqYlDlGzh2M6vW+sE82j1s=">AKNXicrVXbtGEGWcNorUW9w+5mVbwzBZx67oBkiRQIDTGwoEbl2gzgVehViRS2mT3SWxu3QsEPyh/kC/pQ9K/raX+heKIWUqKRGS0DSaHZmzsyZ4ewkp0Sq4fD3a1vX3n3Ru9mf/De+x98+NGt7Y8fy6wQMT6LM5qJpxMkMSUcnymiKH6aC4zYhOInk5fmPMnF1hIkvFf1DzHY4amnKQkRkqrou0bvw524akPpyw8lV1B5jfeRWAEYCyYJASRpSMSjYKq+flyX5YGQOoZlihCsA4yZR1SatlhHnEzFfJDkL9jwWD3VPfRYWMJMCZWYBUoLiE+DL3u1IqrKVglXvaVDCAWRIzWJEy2cL3KDaEGjPRLpykXnDer0g/9FBGPiPwD74PgA69gwpl15VmZhITBm6jJYqSLMpWJAmiwMdrvMF6nO61QNXQc2ypUJeR1r73WwLlQT+QoMjGTE/KZa62AjXlSG90901HXW3YnlvsN+JYsOlFU/7aW5MY+pIhPKf5CQmEF62aw9texdLA2WhcWxJTqRA+cITGP7a+bz0SdBqdM8AO4Vsfm23MFb3h/nvyDL/bMFbOi+bv5672s3NwEUp8pvoFrQzZj6fYGCTGfK1I9oPkOl8Xb0mSRWRYNiD3gpkIGZMS1A7dEckvkJs9cZMnCzYTpcH1LDuQKkUbOtW6vLt7ZrXQ9WAc4agNYu/8IoueDMCxBXZ3V+o3Bc6sxwVQhM4TVQrYtjF7YNUP1mk+WKjMjzTn7jyNXnQOVTDW2TYT06/zKF/xXwzdcIGEW3doaHQ/uAdSGshR2vfk6j7S0BkywuGOYqpkjK83CYq3GJhCIxdUAFhLnKH6JpvhcixzploxLe6tWYFdrEpBmQn+4Alb9CgRk3LOJtrSvIBy9cwou87OC5V+NS4JzwuFeyA0oIClQFzRYOECBwrOtcCigXRuYJ4hnQ3lL7INU8NmBmF1i1C4nZuJSpRW+lNGFV2/nSFTqAnP8Ks4YQz5vIQpYoTOE5yigipzCacLuYuvO8kFyWVN3TIkxQpmeo0QjqhZQHYLtdV2ybhVM4DfYt0gU901j/lWCVCZ2Ju6Mq3bAp/NSs8OpNloQvLbXYLqu0CehiDC9ZjnlZubVJM4nhZCqyIm8lvOZvE9UBUKrb4Ox281Z6CkNV2dyXh8dBh+eXj0892d46/reb3p3fY+83wv9O5x94P3ql35sW97d693nHvYf+3/h/9P/t/OdOta7XPJ17r6f/9D9AplSI=</latexit> <latexit sha1_base64="KhLME4V4Sic9fp1LltTVpr0QY=">AKNXicrVXdbuNEFPZ2YbMJf9vlkpuBqpNtyUuSItAkcqvkFaFItHdRZ2sNbHyezO2NbMpNvI8gvxAjwLF9whbnkFzsw4WTtxCxVYSnJy5nznO38+Myk4U3o4/O3W1u3Xr/Tu9sfvPHmW2+/c2/7/mOVz2VMz+Kc5/LphCjKWUbPNOcPi0kJWLC6ZPJi6/M+ZMLKhXLs5/0oqBjQaYZS1lMNKi7Tu/DHbxqWI+nojyZfUAmd9FaARwmouMGeCaRWVYhRWz8qT/bAyBljPqCYVwnGSawtJq5WHRSTMVykOQvgngsHuqe+8YsES5MwsQSpJXGJ6WfhdIQRV2QrBqveAlGUIC6JnMeHlz0veoLrC0Z7xdOMki4b1ZkL+o4Mw8B+hfRtgMD3jGgXlUZn0ROBbmMVirM8ylaFgE1qzDY7TJfhrqoQzXlOrBeblyQV72XjnrYjWeb1CBkYqE31SDjf8RWX42Ql43ay6O7G17Bfi6KDZR0HKiFMfYxJ9mU048UlawMO1v8kFztpsXVyYcg6BHjhDZgy/bzXzWemzoNVSp/kcuVfExtuCo394fx+WSz3zyZwRfeh+Zu9r2FuAjhNtd9gtaRXc8L7giWbznQwIQXM+LKsnxZmkUtoiGwx5kJkGBVJQBILN1zFY9a0FZDS1knixk8X9vogbuJp5KB1eyF5Z7fW9WCT4KhNYO3+IwnMBxNUoTo7q/Ubg+dWY0K5JmYIq6VsWxg9t2uGw5pPViozI805+Y8jZ53DlUwhm1y1WE9Ov8yhP+V8PprBI2iezvDw6F90KYQ1sKOVz+n0faWxEkezwXNdMyJUufhsNDjkjNYk6rAZ4rWpD4BZnScxAzAi0Zl/ZWrdAuaBKU5hI+mUZW20SURCi1EBOwNC+gWj8zyq6z87lOPx2XLCvmaxI0rnHOkcmSsaJUzSWPMFCSWDGJF8YxANzRc5FCnBs2M8guq24nEYlyq1LK3QpqIqg2+dIkOsKQZfRnQpAs+bDEKRGMLxKakjnX5hJOl3JXvR4kF6xQdelWLjnVOIc1wjLCzQKyW6itkvGrZoB/pCgyQ9gah/KgkOpcQibujKmjYFL9vVnh1nSXLVpYgtMqbQCQjKlLXtCsrNza5LmieDKV+bxoBbyBt4GCA5JCG5w9bcOcBUxpuD6Tm8Ljo8Pw48OjHz/ZOf6ynte73nveB57vhd5D79j7zjv1zry4t9172DvufdH/tf97/4/+n85061aNedrPf2/gYLQpVa</latexit> Learning in CRFs ■ Likelihood can be computed efficiently using forward algorithm . Define: m m X X X Y α m ( y m ) = exp s n ( y n , y n − 1 ) = exp s n ( y n , y n − 1 ) n =1 n =1 y 1: m − 1 y 1: m − 1 ■ As in decoding / Viterbi, can be decomposed into recursive substructure: m − 1 X X Y = (exp s m ( y m , y m − 1 )) exp s n ( y n , y n − 1 ) n =1 y m − 1 y 1: m − 2 Viterbi is a special case of the max-product algorithm , X = (exp s m ( y m , y m − 1 )) × α m − 1 ( y m − 1 ) forward is a special case of the sum-product algorithm . y m − 1 sum instead of max M s m ( k, k 0 ) ⊗ v m � 1 ( k 0 ) . v m ( k ) = k 0 2 Y 9

  42. Learning in CRFs 10

  43. <latexit sha1_base64="gBH4cwI4J7/lJfldZ9/VtEYFLGQ=">AKFHicrVfbxtFEL+mkNTmT5si9YUHFqIoZ9KkvoBEVRSp4p+QqkCQSFuUdU/ruz1729290+46jXW6D8AX4NPwhnjlnW/BR2B29+z47AskgpNsj2dn5jfzm9m5YcGZNv3+nzfWbr7x5vrGrU73rbfef2nc27T3U+UQk9SXKeq+dDoilnkp4YZjh9XihKxJDTZ8NX9rzZ2dUaZbLH820oANBRpJlLCEGVPHm+s/dbXysWYiHonxd3Uf2d1r10CHCeiIwZ4IZHZfiMKpelEe7UWUNsBlTQyqEkzQ3ziWr5hGmsbBfpdiL4J/odbePQx8VC5Yib+YAMkWSEtPzImxLoVeVjRScegdAmURYEDNOC9/muH2qksC7dhI1y6yWLBeLSh8shf1widoF3TQxB7TIxPr6psTKJGgpzHcxXm+QjNSECLHS328xnqU7rVC1dey7KtQm5iLVzEawN1Ua+BgOHOhbhohp0eCFeXEaPjiDqKuv+xHfYr+URQvKsh94ARfWOMScyBGnDzRWTnBuFmt3FQuCNdHasDlHBLd84bMGn7XaOaLMmS9Rku95nPkr4jLt+GO/uX+eP8ZWf6fK+CS7kPzV3tfu/kJ4DQz4QKqA70cE+4LVmw0NrZ+wosx8bzMbsFi+yRcCxaEHcgbYUC6ViCg3REynTWl0LlaczPxunzfdqSVwl0qF3rfsL1Xu7pb3VgEOmgDO7j+CwIAwQTWq3PacGHy/G5MKTfETmE1k10P45duz3DY8+lcZYdkcdC+Ps3il61T1RvAOrnsJ6d7pUy+B/xuvGdrf5+3z1oVYhqYSuon+N4c03hNE8mgkqTcKL1adQvzKAkyrCE06qLJ5oWJHlFRvQUREmA8EHpXpoV2gZNirJcwUca5LSLHiURWk/FECzt/dLZ1bZdnY6MdnDQclkMTFUJh4om3BkcmTfwChliaGT0EgiWKQK0rGBMg28J6Gzi/AjCk/o6ZSCIGpc4ceiOloaiazue+0C5WVNLXS4EkenHJc6IYHya0oxMuLHv2Gwmt/F1Pz1jha6pm4fk1OActgSThNv94pZMU+12iN8kXfwVhQYpegRZf19QRUyuIBP/CqgYSP8od3Q1T9ZMjm3BLFZVukSgGIsL3lBZVn5rchzTfFwpPJ0Uh4xd8lCgFIBm3w9rTp5i1gSqPlmVwVnh7sR5/sH/zw6dbjL+p5vRW8H3wUhEUfBY8Dr4NjoOTIFn/a+PexgcbqPNL59fOb53fvenajdrnvaDxdP74G06/io=</latexit> Learning in CRFs ■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference between observed and expected feature counts: N �` X E [ f j ( w ( i ) , y )] − f j ( w ( i ) , y ( i ) ) = �✓ j i =1 10

  44. <latexit sha1_base64="gBH4cwI4J7/lJfldZ9/VtEYFLGQ=">AKFHicrVfbxtFEL+mkNTmT5si9YUHFqIoZ9KkvoBEVRSp4p+QqkCQSFuUdU/ruz1729290+46jXW6D8AX4NPwhnjlnW/BR2B29+z47AskgpNsj2dn5jfzm9m5YcGZNv3+nzfWbr7x5vrGrU73rbfef2nc27T3U+UQk9SXKeq+dDoilnkp4YZjh9XihKxJDTZ8NX9rzZ2dUaZbLH820oANBRpJlLCEGVPHm+s/dbXysWYiHonxd3Uf2d1r10CHCeiIwZ4IZHZfiMKpelEe7UWUNsBlTQyqEkzQ3ziWr5hGmsbBfpdiL4J/odbePQx8VC5Yib+YAMkWSEtPzImxLoVeVjRScegdAmURYEDNOC9/muH2qksC7dhI1y6yWLBeLSh8shf1widoF3TQxB7TIxPr6psTKJGgpzHcxXm+QjNSECLHS328xnqU7rVC1dey7KtQm5iLVzEawN1Ua+BgOHOhbhohp0eCFeXEaPjiDqKuv+xHfYr+URQvKsh94ARfWOMScyBGnDzRWTnBuFmt3FQuCNdHasDlHBLd84bMGn7XaOaLMmS9Rku95nPkr4jLt+GO/uX+eP8ZWf6fK+CS7kPzV3tfu/kJ4DQz4QKqA70cE+4LVmw0NrZ+wosx8bzMbsFi+yRcCxaEHcgbYUC6ViCg3REynTWl0LlaczPxunzfdqSVwl0qF3rfsL1Xu7pb3VgEOmgDO7j+CwIAwQTWq3PacGHy/G5MKTfETmE1k10P45duz3DY8+lcZYdkcdC+Ps3il61T1RvAOrnsJ6d7pUy+B/xuvGdrf5+3z1oVYhqYSuon+N4c03hNE8mgkqTcKL1adQvzKAkyrCE06qLJ5oWJHlFRvQUREmA8EHpXpoV2gZNirJcwUca5LSLHiURWk/FECzt/dLZ1bZdnY6MdnDQclkMTFUJh4om3BkcmTfwChliaGT0EgiWKQK0rGBMg28J6Gzi/AjCk/o6ZSCIGpc4ceiOloaiazue+0C5WVNLXS4EkenHJc6IYHya0oxMuLHv2Gwmt/F1Pz1jha6pm4fk1OActgSThNv94pZMU+12iN8kXfwVhQYpegRZf19QRUyuIBP/CqgYSP8od3Q1T9ZMjm3BLFZVukSgGIsL3lBZVn5rchzTfFwpPJ0Uh4xd8lCgFIBm3w9rTp5i1gSqPlmVwVnh7sR5/sH/zw6dbjL+p5vRW8H3wUhEUfBY8Dr4NjoOTIFn/a+PexgcbqPNL59fOb53fvenajdrnvaDxdP74G06/io=</latexit> Learning in CRFs ■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference between observed and expected feature counts: count of feature j for token sequence w (i) , N tag sequence y (i) �` X E [ f j ( w ( i ) , y )] − f j ( w ( i ) , y ( i ) ) = �✓ j i =1 10

  45. <latexit sha1_base64="gBH4cwI4J7/lJfldZ9/VtEYFLGQ=">AKFHicrVfbxtFEL+mkNTmT5si9YUHFqIoZ9KkvoBEVRSp4p+QqkCQSFuUdU/ruz1729290+46jXW6D8AX4NPwhnjlnW/BR2B29+z47AskgpNsj2dn5jfzm9m5YcGZNv3+nzfWbr7x5vrGrU73rbfef2nc27T3U+UQk9SXKeq+dDoilnkp4YZjh9XihKxJDTZ8NX9rzZ2dUaZbLH820oANBRpJlLCEGVPHm+s/dbXysWYiHonxd3Uf2d1r10CHCeiIwZ4IZHZfiMKpelEe7UWUNsBlTQyqEkzQ3ziWr5hGmsbBfpdiL4J/odbePQx8VC5Yib+YAMkWSEtPzImxLoVeVjRScegdAmURYEDNOC9/muH2qksC7dhI1y6yWLBeLSh8shf1widoF3TQxB7TIxPr6psTKJGgpzHcxXm+QjNSECLHS328xnqU7rVC1dey7KtQm5iLVzEawN1Ua+BgOHOhbhohp0eCFeXEaPjiDqKuv+xHfYr+URQvKsh94ARfWOMScyBGnDzRWTnBuFmt3FQuCNdHasDlHBLd84bMGn7XaOaLMmS9Rku95nPkr4jLt+GO/uX+eP8ZWf6fK+CS7kPzV3tfu/kJ4DQz4QKqA70cE+4LVmw0NrZ+wosx8bzMbsFi+yRcCxaEHcgbYUC6ViCg3REynTWl0LlaczPxunzfdqSVwl0qF3rfsL1Xu7pb3VgEOmgDO7j+CwIAwQTWq3PacGHy/G5MKTfETmE1k10P45duz3DY8+lcZYdkcdC+Ps3il61T1RvAOrnsJ6d7pUy+B/xuvGdrf5+3z1oVYhqYSuon+N4c03hNE8mgkqTcKL1adQvzKAkyrCE06qLJ5oWJHlFRvQUREmA8EHpXpoV2gZNirJcwUca5LSLHiURWk/FECzt/dLZ1bZdnY6MdnDQclkMTFUJh4om3BkcmTfwChliaGT0EgiWKQK0rGBMg28J6Gzi/AjCk/o6ZSCIGpc4ceiOloaiazue+0C5WVNLXS4EkenHJc6IYHya0oxMuLHv2Gwmt/F1Pz1jha6pm4fk1OActgSThNv94pZMU+12iN8kXfwVhQYpegRZf19QRUyuIBP/CqgYSP8od3Q1T9ZMjm3BLFZVukSgGIsL3lBZVn5rchzTfFwpPJ0Uh4xd8lCgFIBm3w9rTp5i1gSqPlmVwVnh7sR5/sH/zw6dbjL+p5vRW8H3wUhEUfBY8Dr4NjoOTIFn/a+PexgcbqPNL59fOb53fvenajdrnvaDxdP74G06/io=</latexit> Learning in CRFs ■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference between observed and expected feature counts: count of feature j for token sequence w (i) , N tag sequence y (i) �` X E [ f j ( w ( i ) , y )] − f j ( w ( i ) , y ( i ) ) = �✓ j i =1 ■ Gradients can be computed by automatic differentiation! 10

  46. <latexit sha1_base64="gBH4cwI4J7/lJfldZ9/VtEYFLGQ=">AKFHicrVfbxtFEL+mkNTmT5si9YUHFqIoZ9KkvoBEVRSp4p+QqkCQSFuUdU/ruz1729290+46jXW6D8AX4NPwhnjlnW/BR2B29+z47AskgpNsj2dn5jfzm9m5YcGZNv3+nzfWbr7x5vrGrU73rbfef2nc27T3U+UQk9SXKeq+dDoilnkp4YZjh9XihKxJDTZ8NX9rzZ2dUaZbLH820oANBRpJlLCEGVPHm+s/dbXysWYiHonxd3Uf2d1r10CHCeiIwZ4IZHZfiMKpelEe7UWUNsBlTQyqEkzQ3ziWr5hGmsbBfpdiL4J/odbePQx8VC5Yib+YAMkWSEtPzImxLoVeVjRScegdAmURYEDNOC9/muH2qksC7dhI1y6yWLBeLSh8shf1widoF3TQxB7TIxPr6psTKJGgpzHcxXm+QjNSECLHS328xnqU7rVC1dey7KtQm5iLVzEawN1Ua+BgOHOhbhohp0eCFeXEaPjiDqKuv+xHfYr+URQvKsh94ARfWOMScyBGnDzRWTnBuFmt3FQuCNdHasDlHBLd84bMGn7XaOaLMmS9Rku95nPkr4jLt+GO/uX+eP8ZWf6fK+CS7kPzV3tfu/kJ4DQz4QKqA70cE+4LVmw0NrZ+wosx8bzMbsFi+yRcCxaEHcgbYUC6ViCg3REynTWl0LlaczPxunzfdqSVwl0qF3rfsL1Xu7pb3VgEOmgDO7j+CwIAwQTWq3PacGHy/G5MKTfETmE1k10P45duz3DY8+lcZYdkcdC+Ps3il61T1RvAOrnsJ6d7pUy+B/xuvGdrf5+3z1oVYhqYSuon+N4c03hNE8mgkqTcKL1adQvzKAkyrCE06qLJ5oWJHlFRvQUREmA8EHpXpoV2gZNirJcwUca5LSLHiURWk/FECzt/dLZ1bZdnY6MdnDQclkMTFUJh4om3BkcmTfwChliaGT0EgiWKQK0rGBMg28J6Gzi/AjCk/o6ZSCIGpc4ceiOloaiazue+0C5WVNLXS4EkenHJc6IYHya0oxMuLHv2Gwmt/F1Pz1jha6pm4fk1OActgSThNv94pZMU+12iN8kXfwVhQYpegRZf19QRUyuIBP/CqgYSP8od3Q1T9ZMjm3BLFZVukSgGIsL3lBZVn5rchzTfFwpPJ0Uh4xd8lCgFIBm3w9rTp5i1gSqPlmVwVnh7sR5/sH/zw6dbjL+p5vRW8H3wUhEUfBY8Dr4NjoOTIFn/a+PexgcbqPNL59fOb53fvenajdrnvaDxdP74G06/io=</latexit> Learning in CRFs ■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference between observed and expected feature counts: count of feature j for token sequence w (i) , N tag sequence y (i) �` X E [ f j ( w ( i ) , y )] − f j ( w ( i ) , y ( i ) ) = �✓ j i =1 ■ Gradients can be computed by automatic differentiation! ■ In the Olden Days, would use the forward-backward algorithm to compute expected counts. 10

  47. <latexit sha1_base64="gBH4cwI4J7/lJfldZ9/VtEYFLGQ=">AKFHicrVfbxtFEL+mkNTmT5si9YUHFqIoZ9KkvoBEVRSp4p+QqkCQSFuUdU/ruz1729290+46jXW6D8AX4NPwhnjlnW/BR2B29+z47AskgpNsj2dn5jfzm9m5YcGZNv3+nzfWbr7x5vrGrU73rbfef2nc27T3U+UQk9SXKeq+dDoilnkp4YZjh9XihKxJDTZ8NX9rzZ2dUaZbLH820oANBRpJlLCEGVPHm+s/dbXysWYiHonxd3Uf2d1r10CHCeiIwZ4IZHZfiMKpelEe7UWUNsBlTQyqEkzQ3ziWr5hGmsbBfpdiL4J/odbePQx8VC5Yib+YAMkWSEtPzImxLoVeVjRScegdAmURYEDNOC9/muH2qksC7dhI1y6yWLBeLSh8shf1widoF3TQxB7TIxPr6psTKJGgpzHcxXm+QjNSECLHS328xnqU7rVC1dey7KtQm5iLVzEawN1Ua+BgOHOhbhohp0eCFeXEaPjiDqKuv+xHfYr+URQvKsh94ARfWOMScyBGnDzRWTnBuFmt3FQuCNdHasDlHBLd84bMGn7XaOaLMmS9Rku95nPkr4jLt+GO/uX+eP8ZWf6fK+CS7kPzV3tfu/kJ4DQz4QKqA70cE+4LVmw0NrZ+wosx8bzMbsFi+yRcCxaEHcgbYUC6ViCg3REynTWl0LlaczPxunzfdqSVwl0qF3rfsL1Xu7pb3VgEOmgDO7j+CwIAwQTWq3PacGHy/G5MKTfETmE1k10P45duz3DY8+lcZYdkcdC+Ps3il61T1RvAOrnsJ6d7pUy+B/xuvGdrf5+3z1oVYhqYSuon+N4c03hNE8mgkqTcKL1adQvzKAkyrCE06qLJ5oWJHlFRvQUREmA8EHpXpoV2gZNirJcwUca5LSLHiURWk/FECzt/dLZ1bZdnY6MdnDQclkMTFUJh4om3BkcmTfwChliaGT0EgiWKQK0rGBMg28J6Gzi/AjCk/o6ZSCIGpc4ceiOloaiazue+0C5WVNLXS4EkenHJc6IYHya0oxMuLHv2Gwmt/F1Pz1jha6pm4fk1OActgSThNv94pZMU+12iN8kXfwVhQYpegRZf19QRUyuIBP/CqgYSP8od3Q1T9ZMjm3BLFZVukSgGIsL3lBZVn5rchzTfFwpPJ0Uh4xd8lCgFIBm3w9rTp5i1gSqPlmVwVnh7sR5/sH/zw6dbjL+p5vRW8H3wUhEUfBY8Dr4NjoOTIFn/a+PexgcbqPNL59fOb53fvenajdrnvaDxdP74G06/io=</latexit> Learning in CRFs ■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference between observed and expected feature counts: count of feature j for token sequence w (i) , N tag sequence y (i) �` X E [ f j ( w ( i ) , y )] − f j ( w ( i ) , y ( i ) ) = �✓ j i =1 ■ Gradients can be computed by automatic differentiation! ■ In the Olden Days, would use the forward-backward algorithm to compute expected counts. Y m � 1 = k 0 Y m = k α m � 1 ( k 0 ) exp s m ( k, k 0 ) β m ( k ) 10

  48. <latexit sha1_base64="gBH4cwI4J7/lJfldZ9/VtEYFLGQ=">AKFHicrVfbxtFEL+mkNTmT5si9YUHFqIoZ9KkvoBEVRSp4p+QqkCQSFuUdU/ruz1729290+46jXW6D8AX4NPwhnjlnW/BR2B29+z47AskgpNsj2dn5jfzm9m5YcGZNv3+nzfWbr7x5vrGrU73rbfef2nc27T3U+UQk9SXKeq+dDoilnkp4YZjh9XihKxJDTZ8NX9rzZ2dUaZbLH820oANBRpJlLCEGVPHm+s/dbXysWYiHonxd3Uf2d1r10CHCeiIwZ4IZHZfiMKpelEe7UWUNsBlTQyqEkzQ3ziWr5hGmsbBfpdiL4J/odbePQx8VC5Yib+YAMkWSEtPzImxLoVeVjRScegdAmURYEDNOC9/muH2qksC7dhI1y6yWLBeLSh8shf1widoF3TQxB7TIxPr6psTKJGgpzHcxXm+QjNSECLHS328xnqU7rVC1dey7KtQm5iLVzEawN1Ua+BgOHOhbhohp0eCFeXEaPjiDqKuv+xHfYr+URQvKsh94ARfWOMScyBGnDzRWTnBuFmt3FQuCNdHasDlHBLd84bMGn7XaOaLMmS9Rku95nPkr4jLt+GO/uX+eP8ZWf6fK+CS7kPzV3tfu/kJ4DQz4QKqA70cE+4LVmw0NrZ+wosx8bzMbsFi+yRcCxaEHcgbYUC6ViCg3REynTWl0LlaczPxunzfdqSVwl0qF3rfsL1Xu7pb3VgEOmgDO7j+CwIAwQTWq3PacGHy/G5MKTfETmE1k10P45duz3DY8+lcZYdkcdC+Ps3il61T1RvAOrnsJ6d7pUy+B/xuvGdrf5+3z1oVYhqYSuon+N4c03hNE8mgkqTcKL1adQvzKAkyrCE06qLJ5oWJHlFRvQUREmA8EHpXpoV2gZNirJcwUca5LSLHiURWk/FECzt/dLZ1bZdnY6MdnDQclkMTFUJh4om3BkcmTfwChliaGT0EgiWKQK0rGBMg28J6Gzi/AjCk/o6ZSCIGpc4ceiOloaiazue+0C5WVNLXS4EkenHJc6IYHya0oxMuLHv2Gwmt/F1Pz1jha6pm4fk1OActgSThNv94pZMU+12iN8kXfwVhQYpegRZf19QRUyuIBP/CqgYSP8od3Q1T9ZMjm3BLFZVukSgGIsL3lBZVn5rchzTfFwpPJ0Uh4xd8lCgFIBm3w9rTp5i1gSqPlmVwVnh7sR5/sH/zw6dbjL+p5vRW8H3wUhEUfBY8Dr4NjoOTIFn/a+PexgcbqPNL59fOb53fvenajdrnvaDxdP74G06/io=</latexit> Learning in CRFs ■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference between observed and expected feature counts: count of feature j for token sequence w (i) , N tag sequence y (i) �` X E [ f j ( w ( i ) , y )] − f j ( w ( i ) , y ( i ) ) = �✓ j i =1 ■ Gradients can be computed by automatic differentiation! ■ In the Olden Days, would use the forward-backward algorithm to compute expected counts. Y m � 1 = k 0 Y m = k forward score: α m � 1 ( k 0 ) exp s m ( k, k 0 ) β m ( k ) sum over all prefixes 10

  49. <latexit sha1_base64="gBH4cwI4J7/lJfldZ9/VtEYFLGQ=">AKFHicrVfbxtFEL+mkNTmT5si9YUHFqIoZ9KkvoBEVRSp4p+QqkCQSFuUdU/ruz1729290+46jXW6D8AX4NPwhnjlnW/BR2B29+z47AskgpNsj2dn5jfzm9m5YcGZNv3+nzfWbr7x5vrGrU73rbfef2nc27T3U+UQk9SXKeq+dDoilnkp4YZjh9XihKxJDTZ8NX9rzZ2dUaZbLH820oANBRpJlLCEGVPHm+s/dbXysWYiHonxd3Uf2d1r10CHCeiIwZ4IZHZfiMKpelEe7UWUNsBlTQyqEkzQ3ziWr5hGmsbBfpdiL4J/odbePQx8VC5Yib+YAMkWSEtPzImxLoVeVjRScegdAmURYEDNOC9/muH2qksC7dhI1y6yWLBeLSh8shf1widoF3TQxB7TIxPr6psTKJGgpzHcxXm+QjNSECLHS328xnqU7rVC1dey7KtQm5iLVzEawN1Ua+BgOHOhbhohp0eCFeXEaPjiDqKuv+xHfYr+URQvKsh94ARfWOMScyBGnDzRWTnBuFmt3FQuCNdHasDlHBLd84bMGn7XaOaLMmS9Rku95nPkr4jLt+GO/uX+eP8ZWf6fK+CS7kPzV3tfu/kJ4DQz4QKqA70cE+4LVmw0NrZ+wosx8bzMbsFi+yRcCxaEHcgbYUC6ViCg3REynTWl0LlaczPxunzfdqSVwl0qF3rfsL1Xu7pb3VgEOmgDO7j+CwIAwQTWq3PacGHy/G5MKTfETmE1k10P45duz3DY8+lcZYdkcdC+Ps3il61T1RvAOrnsJ6d7pUy+B/xuvGdrf5+3z1oVYhqYSuon+N4c03hNE8mgkqTcKL1adQvzKAkyrCE06qLJ5oWJHlFRvQUREmA8EHpXpoV2gZNirJcwUca5LSLHiURWk/FECzt/dLZ1bZdnY6MdnDQclkMTFUJh4om3BkcmTfwChliaGT0EgiWKQK0rGBMg28J6Gzi/AjCk/o6ZSCIGpc4ceiOloaiazue+0C5WVNLXS4EkenHJc6IYHya0oxMuLHv2Gwmt/F1Pz1jha6pm4fk1OActgSThNv94pZMU+12iN8kXfwVhQYpegRZf19QRUyuIBP/CqgYSP8od3Q1T9ZMjm3BLFZVukSgGIsL3lBZVn5rchzTfFwpPJ0Uh4xd8lCgFIBm3w9rTp5i1gSqPlmVwVnh7sR5/sH/zw6dbjL+p5vRW8H3wUhEUfBY8Dr4NjoOTIFn/a+PexgcbqPNL59fOb53fvenajdrnvaDxdP74G06/io=</latexit> Learning in CRFs ■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference between observed and expected feature counts: count of feature j for token sequence w (i) , N tag sequence y (i) �` X E [ f j ( w ( i ) , y )] − f j ( w ( i ) , y ( i ) ) = �✓ j i =1 ■ Gradients can be computed by automatic differentiation! ■ In the Olden Days, would use the forward-backward algorithm to compute expected counts. Y m � 1 = k 0 Y m = k backward score: forward score: sum over all suffixes α m � 1 ( k 0 ) exp s m ( k, k 0 ) β m ( k ) sum over all prefixes 10

  50. <latexit sha1_base64="gBH4cwI4J7/lJfldZ9/VtEYFLGQ=">AKFHicrVfbxtFEL+mkNTmT5si9YUHFqIoZ9KkvoBEVRSp4p+QqkCQSFuUdU/ruz1729290+46jXW6D8AX4NPwhnjlnW/BR2B29+z47AskgpNsj2dn5jfzm9m5YcGZNv3+nzfWbr7x5vrGrU73rbfef2nc27T3U+UQk9SXKeq+dDoilnkp4YZjh9XihKxJDTZ8NX9rzZ2dUaZbLH820oANBRpJlLCEGVPHm+s/dbXysWYiHonxd3Uf2d1r10CHCeiIwZ4IZHZfiMKpelEe7UWUNsBlTQyqEkzQ3ziWr5hGmsbBfpdiL4J/odbePQx8VC5Yib+YAMkWSEtPzImxLoVeVjRScegdAmURYEDNOC9/muH2qksC7dhI1y6yWLBeLSh8shf1widoF3TQxB7TIxPr6psTKJGgpzHcxXm+QjNSECLHS328xnqU7rVC1dey7KtQm5iLVzEawN1Ua+BgOHOhbhohp0eCFeXEaPjiDqKuv+xHfYr+URQvKsh94ARfWOMScyBGnDzRWTnBuFmt3FQuCNdHasDlHBLd84bMGn7XaOaLMmS9Rku95nPkr4jLt+GO/uX+eP8ZWf6fK+CS7kPzV3tfu/kJ4DQz4QKqA70cE+4LVmw0NrZ+wosx8bzMbsFi+yRcCxaEHcgbYUC6ViCg3REynTWl0LlaczPxunzfdqSVwl0qF3rfsL1Xu7pb3VgEOmgDO7j+CwIAwQTWq3PacGHy/G5MKTfETmE1k10P45duz3DY8+lcZYdkcdC+Ps3il61T1RvAOrnsJ6d7pUy+B/xuvGdrf5+3z1oVYhqYSuon+N4c03hNE8mgkqTcKL1adQvzKAkyrCE06qLJ5oWJHlFRvQUREmA8EHpXpoV2gZNirJcwUca5LSLHiURWk/FECzt/dLZ1bZdnY6MdnDQclkMTFUJh4om3BkcmTfwChliaGT0EgiWKQK0rGBMg28J6Gzi/AjCk/o6ZSCIGpc4ceiOloaiazue+0C5WVNLXS4EkenHJc6IYHya0oxMuLHv2Gwmt/F1Pz1jha6pm4fk1OActgSThNv94pZMU+12iN8kXfwVhQYpegRZf19QRUyuIBP/CqgYSP8od3Q1T9ZMjm3BLFZVukSgGIsL3lBZVn5rchzTfFwpPJ0Uh4xd8lCgFIBm3w9rTp5i1gSqPlmVwVnh7sR5/sH/zw6dbjL+p5vRW8H3wUhEUfBY8Dr4NjoOTIFn/a+PexgcbqPNL59fOb53fvenajdrnvaDxdP74G06/io=</latexit> Learning in CRFs ■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference between observed and expected feature counts: count of feature j for token sequence w (i) , N tag sequence y (i) �` X E [ f j ( w ( i ) , y )] − f j ( w ( i ) , y ( i ) ) = �✓ j i =1 ■ Gradients can be computed by automatic differentiation! ■ In the Olden Days, would use the forward-backward algorithm to compute expected counts. Y m � 1 = k 0 Y m = k transition score backward score: forward score: sum over all suffixes α m � 1 ( k 0 ) exp s m ( k, k 0 ) β m ( k ) sum over all prefixes 10

  51. Better features for sequence labeling? ■ Until now: hand-engineered features: 11

  52. Better features for sequence labeling? ■ Until now: hand-engineered features: w i contains a particular prefix (from all prefixes of length  4) w i contains a particular suffix (from all suffixes of length  4) f i ±{ 0 , 1 , 2 , 3 } , ( m i − 2 ,i − 1 ), ( m i − 1 ,i ), ( m i − 1 ,i +1 ), w i contains a number ( m i,i +1 ), ( m i +1 ,i +2 ), ( m i − 2 ,i − 1 ,i ), ( m i − 1 ,i,i +1 ), Lexical w i contains an upper-case letter ( m i,i +1 ,i +2 ), ( m i − 2 ,i − 1 ,i +1 ), ( m i − 1 ,i +1 ,i +2 ) w i contains a hyphen p i − { 3 , 2 , 1 } , a i + { 0 , 1 , 2 , 3 } , ( p i − 2 ,i − 1 ), ( a i +1 ,i +2 ), ( p i − 1 , a i +1 ), ( p i − 2 , p i − 1 , a i ), ( p i − 2 , p i − 1 , a i +1 ), POS w i is all upper case ( p i − 1 , a i , a i +1 ), ( p i − 1 , a i +1 , a i +2 ) w i ’s word shape Affix c :1 , c :2 , c :3 , c n : , c n − 1: , c n − 2: , c n − 3: w i ’s short word shape initial uppercase, all uppercase/lowercase, w i is upper case and has a digit and a dash (like CFC-12 ) Binary contains 1/2+ capital(s) not at the beginning, w i is upper case and followed within 3 words by Co., Inc., etc. contains a (period/number/hyphen) 11

  53. <latexit sha1_base64="y2cxWKwz3S3xVgA5ofaWNPquJR0=">AKNXicrVXdbts2FbTra69v6a73A23Ii8NJmVDejQwUD2iwFtgxY2g6hK9ASZbMlKYGk0xiCXmgvsGfZxe6G3e4VdkjKjhUra4JNgO3jw3O+7/zpcFxwps1g8PutjdtvHmnc7fbe+vtd959797m/Sc6n6mEniQ5z9WzMdGUM0lPDOcPisUJWLM6dPxy6/t+dMzqjTL5c9mXtCRIBPJMpYQA6p486vW18rFmIx6J8VT1A9nde9dEQYT0TmDPBjI5LMYyq5+XRblRZA2ym1JAK4STNjXPJqiXCPBb2qxR7EfwT/d72cehRsWAp8maOIFMkKTE9L8K2EPpV2QjBqXeAlEmEBTHThPDylwVv7oCaMci3TjJYsV6PaHw8V7UDx+jXfRdHwH2lBgfXlVZTKImgpzHSxXm+QtioBWq9DbjNfhDqvQ7Xl2nMoNy7IBdbOBVgbq0W+QWGOhbhqhp0eAUvLqNHR4C6XnV/4mrfYn8pihaWy37gBbWwxiHmRE4/URj5QTnZrl217kArMnWxoUp5xDonjdk1vCHRjOflyHrN1rqNV8g/4q4eBvu6DXvj/dfFMv/cwlc0X1o/nrvazc/AZxmJlxhdaRXc8L7ghWbTI3Nn/BiSnxdFm/LRXZI+GqaEncgbQZCqRjCQ7SFVIum9bqWqg8XfhZnDbf6wVxHaShd637C9l7u0t768THDQJnN1/JIEBYJqVGfntOHK5PndmFJuiJ3CaiG7HsYv3J7hsOfTpcoOyeqgfXuaxS9ap6o/gnVy1WE9O9cM4X8kfM01gobxva3B/sA9aF2IamErqJ/jeHND4TRPZoJKk3Ci9Wk0KMyoJMqwhNOqh2eaFiR5Sb0FERJoCOj0t2qFdoGTYqyXMFHGuS0qx4lEVrPxRgs7QuoL59ZdvZ6cxkn49KJouZoTLxRNmMI5Mje0WjlCmaGD4HgSKQawomRJohoGLHPqyQjOl/IyaZiKJGJU6c+yNkMaiajqf+0R7WFJXyW5ESmH5c4I4LxeUozMuPGXsLZQm6r14P0jBW6Lt0SklODc1gjTBJuF5DbQk21WzJ+1fTwNxQapOgRP1jQRUxuYJI/B1VQcMm+EO7wqt/s2RyaQliM63SBQDJ2LrkBZVl5dcmzXF4nKZ0Uj4DV/FygAkAza4O1p081bwJRGl2dyXhysB9un/w02dbh1/V83o3+CD4KAiDKHgYHAbfB8fBSZB0NjsPO4edL7u/df/o/tn9y5tu3Kp93g8aT/fvfwDACJWS</latexit> Better features for sequence labeling? ■ Until now: hand-engineered features: w i contains a particular prefix (from all prefixes of length  4) w i contains a particular suffix (from all suffixes of length  4) f i ±{ 0 , 1 , 2 , 3 } , ( m i − 2 ,i − 1 ), ( m i − 1 ,i ), ( m i − 1 ,i +1 ), w i contains a number ( m i,i +1 ), ( m i +1 ,i +2 ), ( m i − 2 ,i − 1 ,i ), ( m i − 1 ,i,i +1 ), Lexical w i contains an upper-case letter ( m i,i +1 ,i +2 ), ( m i − 2 ,i − 1 ,i +1 ), ( m i − 1 ,i +1 ,i +2 ) w i contains a hyphen p i − { 3 , 2 , 1 } , a i + { 0 , 1 , 2 , 3 } , ( p i − 2 ,i − 1 ), ( a i +1 ,i +2 ), ( p i − 1 , a i +1 ), ( p i − 2 , p i − 1 , a i ), ( p i − 2 , p i − 1 , a i +1 ), POS w i is all upper case ( p i − 1 , a i , a i +1 ), ( p i − 1 , a i +1 , a i +2 ) w i ’s word shape Affix c :1 , c :2 , c :3 , c n : , c n − 1: , c n − 2: , c n − 3: w i ’s short word shape initial uppercase, all uppercase/lowercase, w i is upper case and has a digit and a dash (like CFC-12 ) Binary contains 1/2+ capital(s) not at the beginning, w i is upper case and followed within 3 words by Co., Inc., etc. contains a (period/number/hyphen) f ( w , y m , y m − 1 , m ) = 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 w m = back w m+1 = ache w m-1 = will w m-1 = my y m-1 = MD Janet will back the bill . 11

  54. <latexit sha1_base64="y2cxWKwz3S3xVgA5ofaWNPquJR0=">AKNXicrVXdbts2FbTra69v6a73A23Ii8NJmVDejQwUD2iwFtgxY2g6hK9ASZbMlKYGk0xiCXmgvsGfZxe6G3e4VdkjKjhUra4JNgO3jw3O+7/zpcFxwps1g8PutjdtvHmnc7fbe+vtd959797m/Sc6n6mEniQ5z9WzMdGUM0lPDOcPisUJWLM6dPxy6/t+dMzqjTL5c9mXtCRIBPJMpYQA6p486vW18rFmIx6J8VT1A9nde9dEQYT0TmDPBjI5LMYyq5+XRblRZA2ym1JAK4STNjXPJqiXCPBb2qxR7EfwT/d72cehRsWAp8maOIFMkKTE9L8K2EPpV2QjBqXeAlEmEBTHThPDylwVv7oCaMci3TjJYsV6PaHw8V7UDx+jXfRdHwH2lBgfXlVZTKImgpzHSxXm+QtioBWq9DbjNfhDqvQ7Xl2nMoNy7IBdbOBVgbq0W+QWGOhbhqhp0eAUvLqNHR4C6XnV/4mrfYn8pihaWy37gBbWwxiHmRE4/URj5QTnZrl217kArMnWxoUp5xDonjdk1vCHRjOflyHrN1rqNV8g/4q4eBvu6DXvj/dfFMv/cwlc0X1o/nrvazc/AZxmJlxhdaRXc8L7ghWbTI3Nn/BiSnxdFm/LRXZI+GqaEncgbQZCqRjCQ7SFVIum9bqWqg8XfhZnDbf6wVxHaShd637C9l7u0t768THDQJnN1/JIEBYJqVGfntOHK5PndmFJuiJ3CaiG7HsYv3J7hsOfTpcoOyeqgfXuaxS9ap6o/gnVy1WE9O9cM4X8kfM01gobxva3B/sA9aF2IamErqJ/jeHND4TRPZoJKk3Ci9Wk0KMyoJMqwhNOqh2eaFiR5Sb0FERJoCOj0t2qFdoGTYqyXMFHGuS0qx4lEVrPxRgs7QuoL59ZdvZ6cxkn49KJouZoTLxRNmMI5Mje0WjlCmaGD4HgSKQawomRJohoGLHPqyQjOl/IyaZiKJGJU6c+yNkMaiajqf+0R7WFJXyW5ESmH5c4I4LxeUozMuPGXsLZQm6r14P0jBW6Lt0SklODc1gjTBJuF5DbQk21WzJ+1fTwNxQapOgRP1jQRUxuYJI/B1VQcMm+EO7wqt/s2RyaQliM63SBQDJ2LrkBZVl5dcmzXF4nKZ0Uj4DV/FygAkAza4O1p081bwJRGl2dyXhysB9un/w02dbh1/V83o3+CD4KAiDKHgYHAbfB8fBSZB0NjsPO4edL7u/df/o/tn9y5tu3Kp93g8aT/fvfwDACJWS</latexit> Better features for sequence labeling? ■ Until now: hand-engineered features: pros: cons: f ( w , y m , y m − 1 , m ) = 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 w m = back w m+1 = ache w m-1 = will w m-1 = my y m-1 = MD Janet will back the bill . 12

  55. <latexit sha1_base64="y2cxWKwz3S3xVgA5ofaWNPquJR0=">AKNXicrVXdbts2FbTra69v6a73A23Ii8NJmVDejQwUD2iwFtgxY2g6hK9ASZbMlKYGk0xiCXmgvsGfZxe6G3e4VdkjKjhUra4JNgO3jw3O+7/zpcFxwps1g8PutjdtvHmnc7fbe+vtd959797m/Sc6n6mEniQ5z9WzMdGUM0lPDOcPisUJWLM6dPxy6/t+dMzqjTL5c9mXtCRIBPJMpYQA6p486vW18rFmIx6J8VT1A9nde9dEQYT0TmDPBjI5LMYyq5+XRblRZA2ym1JAK4STNjXPJqiXCPBb2qxR7EfwT/d72cehRsWAp8maOIFMkKTE9L8K2EPpV2QjBqXeAlEmEBTHThPDylwVv7oCaMci3TjJYsV6PaHw8V7UDx+jXfRdHwH2lBgfXlVZTKImgpzHSxXm+QtioBWq9DbjNfhDqvQ7Xl2nMoNy7IBdbOBVgbq0W+QWGOhbhqhp0eAUvLqNHR4C6XnV/4mrfYn8pihaWy37gBbWwxiHmRE4/URj5QTnZrl217kArMnWxoUp5xDonjdk1vCHRjOflyHrN1rqNV8g/4q4eBvu6DXvj/dfFMv/cwlc0X1o/nrvazc/AZxmJlxhdaRXc8L7ghWbTI3Nn/BiSnxdFm/LRXZI+GqaEncgbQZCqRjCQ7SFVIum9bqWqg8XfhZnDbf6wVxHaShd637C9l7u0t768THDQJnN1/JIEBYJqVGfntOHK5PndmFJuiJ3CaiG7HsYv3J7hsOfTpcoOyeqgfXuaxS9ap6o/gnVy1WE9O9cM4X8kfM01gobxva3B/sA9aF2IamErqJ/jeHND4TRPZoJKk3Ci9Wk0KMyoJMqwhNOqh2eaFiR5Sb0FERJoCOj0t2qFdoGTYqyXMFHGuS0qx4lEVrPxRgs7QuoL59ZdvZ6cxkn49KJouZoTLxRNmMI5Mje0WjlCmaGD4HgSKQawomRJohoGLHPqyQjOl/IyaZiKJGJU6c+yNkMaiajqf+0R7WFJXyW5ESmH5c4I4LxeUozMuPGXsLZQm6r14P0jBW6Lt0SklODc1gjTBJuF5DbQk21WzJ+1fTwNxQapOgRP1jQRUxuYJI/B1VQcMm+EO7wqt/s2RyaQliM63SBQDJ2LrkBZVl5dcmzXF4nKZ0Uj4DV/FygAkAza4O1p081bwJRGl2dyXhysB9un/w02dbh1/V83o3+CD4KAiDKHgYHAbfB8fBSZB0NjsPO4edL7u/df/o/tn9y5tu3Kp93g8aT/fvfwDACJWS</latexit> Better features for sequence labeling? ■ Until now: hand-engineered features: pros: cons: ■ interpretable, explainable ■ can generalize well ■ fast training and inference ■ channel domain knowledge f ( w , y m , y m − 1 , m ) = 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 w m = back w m+1 = ache w m-1 = will w m-1 = my y m-1 = MD Janet will back the bill . 12

  56. <latexit sha1_base64="y2cxWKwz3S3xVgA5ofaWNPquJR0=">AKNXicrVXdbts2FbTra69v6a73A23Ii8NJmVDejQwUD2iwFtgxY2g6hK9ASZbMlKYGk0xiCXmgvsGfZxe6G3e4VdkjKjhUra4JNgO3jw3O+7/zpcFxwps1g8PutjdtvHmnc7fbe+vtd959797m/Sc6n6mEniQ5z9WzMdGUM0lPDOcPisUJWLM6dPxy6/t+dMzqjTL5c9mXtCRIBPJMpYQA6p486vW18rFmIx6J8VT1A9nde9dEQYT0TmDPBjI5LMYyq5+XRblRZA2ym1JAK4STNjXPJqiXCPBb2qxR7EfwT/d72cehRsWAp8maOIFMkKTE9L8K2EPpV2QjBqXeAlEmEBTHThPDylwVv7oCaMci3TjJYsV6PaHw8V7UDx+jXfRdHwH2lBgfXlVZTKImgpzHSxXm+QtioBWq9DbjNfhDqvQ7Xl2nMoNy7IBdbOBVgbq0W+QWGOhbhqhp0eAUvLqNHR4C6XnV/4mrfYn8pihaWy37gBbWwxiHmRE4/URj5QTnZrl217kArMnWxoUp5xDonjdk1vCHRjOflyHrN1rqNV8g/4q4eBvu6DXvj/dfFMv/cwlc0X1o/nrvazc/AZxmJlxhdaRXc8L7ghWbTI3Nn/BiSnxdFm/LRXZI+GqaEncgbQZCqRjCQ7SFVIum9bqWqg8XfhZnDbf6wVxHaShd637C9l7u0t768THDQJnN1/JIEBYJqVGfntOHK5PndmFJuiJ3CaiG7HsYv3J7hsOfTpcoOyeqgfXuaxS9ap6o/gnVy1WE9O9cM4X8kfM01gobxva3B/sA9aF2IamErqJ/jeHND4TRPZoJKk3Ci9Wk0KMyoJMqwhNOqh2eaFiR5Sb0FERJoCOj0t2qFdoGTYqyXMFHGuS0qx4lEVrPxRgs7QuoL59ZdvZ6cxkn49KJouZoTLxRNmMI5Mje0WjlCmaGD4HgSKQawomRJohoGLHPqyQjOl/IyaZiKJGJU6c+yNkMaiajqf+0R7WFJXyW5ESmH5c4I4LxeUozMuPGXsLZQm6r14P0jBW6Lt0SklODc1gjTBJuF5DbQk21WzJ+1fTwNxQapOgRP1jQRUxuYJI/B1VQcMm+EO7wqt/s2RyaQliM63SBQDJ2LrkBZVl5dcmzXF4nKZ0Uj4DV/FygAkAza4O1p081bwJRGl2dyXhysB9un/w02dbh1/V83o3+CD4KAiDKHgYHAbfB8fBSZB0NjsPO4edL7u/df/o/tn9y5tu3Kp93g8aT/fvfwDACJWS</latexit> Better features for sequence labeling? ■ Until now: hand-engineered features: pros: cons: ■ interpretable, explainable ■ can be sparse/high variance ■ can generalize well ■ lack of shared representations ■ fast training and inference ■ task-specific ■ channel domain knowledge ■ worse performance f ( w , y m , y m − 1 , m ) = 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 w m = back w m+1 = ache w m-1 = will w m-1 = my y m-1 = MD Janet will back the bill . 12

  57. <latexit sha1_base64="y2cxWKwz3S3xVgA5ofaWNPquJR0=">AKNXicrVXdbts2FbTra69v6a73A23Ii8NJmVDejQwUD2iwFtgxY2g6hK9ASZbMlKYGk0xiCXmgvsGfZxe6G3e4VdkjKjhUra4JNgO3jw3O+7/zpcFxwps1g8PutjdtvHmnc7fbe+vtd959797m/Sc6n6mEniQ5z9WzMdGUM0lPDOcPisUJWLM6dPxy6/t+dMzqjTL5c9mXtCRIBPJMpYQA6p486vW18rFmIx6J8VT1A9nde9dEQYT0TmDPBjI5LMYyq5+XRblRZA2ym1JAK4STNjXPJqiXCPBb2qxR7EfwT/d72cehRsWAp8maOIFMkKTE9L8K2EPpV2QjBqXeAlEmEBTHThPDylwVv7oCaMci3TjJYsV6PaHw8V7UDx+jXfRdHwH2lBgfXlVZTKImgpzHSxXm+QtioBWq9DbjNfhDqvQ7Xl2nMoNy7IBdbOBVgbq0W+QWGOhbhqhp0eAUvLqNHR4C6XnV/4mrfYn8pihaWy37gBbWwxiHmRE4/URj5QTnZrl217kArMnWxoUp5xDonjdk1vCHRjOflyHrN1rqNV8g/4q4eBvu6DXvj/dfFMv/cwlc0X1o/nrvazc/AZxmJlxhdaRXc8L7ghWbTI3Nn/BiSnxdFm/LRXZI+GqaEncgbQZCqRjCQ7SFVIum9bqWqg8XfhZnDbf6wVxHaShd637C9l7u0t768THDQJnN1/JIEBYJqVGfntOHK5PndmFJuiJ3CaiG7HsYv3J7hsOfTpcoOyeqgfXuaxS9ap6o/gnVy1WE9O9cM4X8kfM01gobxva3B/sA9aF2IamErqJ/jeHND4TRPZoJKk3Ci9Wk0KMyoJMqwhNOqh2eaFiR5Sb0FERJoCOj0t2qFdoGTYqyXMFHGuS0qx4lEVrPxRgs7QuoL59ZdvZ6cxkn49KJouZoTLxRNmMI5Mje0WjlCmaGD4HgSKQawomRJohoGLHPqyQjOl/IyaZiKJGJU6c+yNkMaiajqf+0R7WFJXyW5ESmH5c4I4LxeUozMuPGXsLZQm6r14P0jBW6Lt0SklODc1gjTBJuF5DbQk21WzJ+1fTwNxQapOgRP1jQRUxuYJI/B1VQcMm+EO7wqt/s2RyaQliM63SBQDJ2LrkBZVl5dcmzXF4nKZ0Uj4DV/FygAkAza4O1p081bwJRGl2dyXhysB9un/w02dbh1/V83o3+CD4KAiDKHgYHAbfB8fBSZB0NjsPO4edL7u/df/o/tn9y5tu3Kp93g8aT/fvfwDACJWS</latexit> Neural sequence labeling ■ Parameterize f with a (deep) neural network. f ( w , y m , y m − 1 , m ) = . . . . . . 13

  58. <latexit sha1_base64="y2cxWKwz3S3xVgA5ofaWNPquJR0=">AKNXicrVXdbts2FbTra69v6a73A23Ii8NJmVDejQwUD2iwFtgxY2g6hK9ASZbMlKYGk0xiCXmgvsGfZxe6G3e4VdkjKjhUra4JNgO3jw3O+7/zpcFxwps1g8PutjdtvHmnc7fbe+vtd959797m/Sc6n6mEniQ5z9WzMdGUM0lPDOcPisUJWLM6dPxy6/t+dMzqjTL5c9mXtCRIBPJMpYQA6p486vW18rFmIx6J8VT1A9nde9dEQYT0TmDPBjI5LMYyq5+XRblRZA2ym1JAK4STNjXPJqiXCPBb2qxR7EfwT/d72cehRsWAp8maOIFMkKTE9L8K2EPpV2QjBqXeAlEmEBTHThPDylwVv7oCaMci3TjJYsV6PaHw8V7UDx+jXfRdHwH2lBgfXlVZTKImgpzHSxXm+QtioBWq9DbjNfhDqvQ7Xl2nMoNy7IBdbOBVgbq0W+QWGOhbhqhp0eAUvLqNHR4C6XnV/4mrfYn8pihaWy37gBbWwxiHmRE4/URj5QTnZrl217kArMnWxoUp5xDonjdk1vCHRjOflyHrN1rqNV8g/4q4eBvu6DXvj/dfFMv/cwlc0X1o/nrvazc/AZxmJlxhdaRXc8L7ghWbTI3Nn/BiSnxdFm/LRXZI+GqaEncgbQZCqRjCQ7SFVIum9bqWqg8XfhZnDbf6wVxHaShd637C9l7u0t768THDQJnN1/JIEBYJqVGfntOHK5PndmFJuiJ3CaiG7HsYv3J7hsOfTpcoOyeqgfXuaxS9ap6o/gnVy1WE9O9cM4X8kfM01gobxva3B/sA9aF2IamErqJ/jeHND4TRPZoJKk3Ci9Wk0KMyoJMqwhNOqh2eaFiR5Sb0FERJoCOj0t2qFdoGTYqyXMFHGuS0qx4lEVrPxRgs7QuoL59ZdvZ6cxkn49KJouZoTLxRNmMI5Mje0WjlCmaGD4HgSKQawomRJohoGLHPqyQjOl/IyaZiKJGJU6c+yNkMaiajqf+0R7WFJXyW5ESmH5c4I4LxeUozMuPGXsLZQm6r14P0jBW6Lt0SklODc1gjTBJuF5DbQk21WzJ+1fTwNxQapOgRP1jQRUxuYJI/B1VQcMm+EO7wqt/s2RyaQliM63SBQDJ2LrkBZVl5dcmzXF4nKZ0Uj4DV/FygAkAza4O1p081bwJRGl2dyXhysB9un/w02dbh1/V83o3+CD4KAiDKHgYHAbfB8fBSZB0NjsPO4edL7u/df/o/tn9y5tu3Kp93g8aT/fvfwDACJWS</latexit> Neural sequence labeling ■ Parameterize f with a (deep) neural network. cons: pros: f ( w , y m , y m − 1 , m ) = . . . . . . 13

  59. <latexit sha1_base64="y2cxWKwz3S3xVgA5ofaWNPquJR0=">AKNXicrVXdbts2FbTra69v6a73A23Ii8NJmVDejQwUD2iwFtgxY2g6hK9ASZbMlKYGk0xiCXmgvsGfZxe6G3e4VdkjKjhUra4JNgO3jw3O+7/zpcFxwps1g8PutjdtvHmnc7fbe+vtd959797m/Sc6n6mEniQ5z9WzMdGUM0lPDOcPisUJWLM6dPxy6/t+dMzqjTL5c9mXtCRIBPJMpYQA6p486vW18rFmIx6J8VT1A9nde9dEQYT0TmDPBjI5LMYyq5+XRblRZA2ym1JAK4STNjXPJqiXCPBb2qxR7EfwT/d72cehRsWAp8maOIFMkKTE9L8K2EPpV2QjBqXeAlEmEBTHThPDylwVv7oCaMci3TjJYsV6PaHw8V7UDx+jXfRdHwH2lBgfXlVZTKImgpzHSxXm+QtioBWq9DbjNfhDqvQ7Xl2nMoNy7IBdbOBVgbq0W+QWGOhbhqhp0eAUvLqNHR4C6XnV/4mrfYn8pihaWy37gBbWwxiHmRE4/URj5QTnZrl217kArMnWxoUp5xDonjdk1vCHRjOflyHrN1rqNV8g/4q4eBvu6DXvj/dfFMv/cwlc0X1o/nrvazc/AZxmJlxhdaRXc8L7ghWbTI3Nn/BiSnxdFm/LRXZI+GqaEncgbQZCqRjCQ7SFVIum9bqWqg8XfhZnDbf6wVxHaShd637C9l7u0t768THDQJnN1/JIEBYJqVGfntOHK5PndmFJuiJ3CaiG7HsYv3J7hsOfTpcoOyeqgfXuaxS9ap6o/gnVy1WE9O9cM4X8kfM01gobxva3B/sA9aF2IamErqJ/jeHND4TRPZoJKk3Ci9Wk0KMyoJMqwhNOqh2eaFiR5Sb0FERJoCOj0t2qFdoGTYqyXMFHGuS0qx4lEVrPxRgs7QuoL59ZdvZ6cxkn49KJouZoTLxRNmMI5Mje0WjlCmaGD4HgSKQawomRJohoGLHPqyQjOl/IyaZiKJGJU6c+yNkMaiajqf+0R7WFJXyW5ESmH5c4I4LxeUozMuPGXsLZQm6r14P0jBW6Lt0SklODc1gjTBJuF5DbQk21WzJ+1fTwNxQapOgRP1jQRUxuYJI/B1VQcMm+EO7wqt/s2RyaQliM63SBQDJ2LrkBZVl5dcmzXF4nKZ0Uj4DV/FygAkAza4O1p081bwJRGl2dyXhysB9un/w02dbh1/V83o3+CD4KAiDKHgYHAbfB8fBSZB0NjsPO4edL7u/df/o/tn9y5tu3Kp93g8aT/fvfwDACJWS</latexit> Neural sequence labeling ■ Parameterize f with a (deep) neural network. cons: pros: ■ shared representations ■ channel external knowledge (e.g. word embeddings) ■ high accuracy f ( w , y m , y m − 1 , m ) = . . . . . . 13

  60. <latexit sha1_base64="y2cxWKwz3S3xVgA5ofaWNPquJR0=">AKNXicrVXdbts2FbTra69v6a73A23Ii8NJmVDejQwUD2iwFtgxY2g6hK9ASZbMlKYGk0xiCXmgvsGfZxe6G3e4VdkjKjhUra4JNgO3jw3O+7/zpcFxwps1g8PutjdtvHmnc7fbe+vtd959797m/Sc6n6mEniQ5z9WzMdGUM0lPDOcPisUJWLM6dPxy6/t+dMzqjTL5c9mXtCRIBPJMpYQA6p486vW18rFmIx6J8VT1A9nde9dEQYT0TmDPBjI5LMYyq5+XRblRZA2ym1JAK4STNjXPJqiXCPBb2qxR7EfwT/d72cehRsWAp8maOIFMkKTE9L8K2EPpV2QjBqXeAlEmEBTHThPDylwVv7oCaMci3TjJYsV6PaHw8V7UDx+jXfRdHwH2lBgfXlVZTKImgpzHSxXm+QtioBWq9DbjNfhDqvQ7Xl2nMoNy7IBdbOBVgbq0W+QWGOhbhqhp0eAUvLqNHR4C6XnV/4mrfYn8pihaWy37gBbWwxiHmRE4/URj5QTnZrl217kArMnWxoUp5xDonjdk1vCHRjOflyHrN1rqNV8g/4q4eBvu6DXvj/dfFMv/cwlc0X1o/nrvazc/AZxmJlxhdaRXc8L7ghWbTI3Nn/BiSnxdFm/LRXZI+GqaEncgbQZCqRjCQ7SFVIum9bqWqg8XfhZnDbf6wVxHaShd637C9l7u0t768THDQJnN1/JIEBYJqVGfntOHK5PndmFJuiJ3CaiG7HsYv3J7hsOfTpcoOyeqgfXuaxS9ap6o/gnVy1WE9O9cM4X8kfM01gobxva3B/sA9aF2IamErqJ/jeHND4TRPZoJKk3Ci9Wk0KMyoJMqwhNOqh2eaFiR5Sb0FERJoCOj0t2qFdoGTYqyXMFHGuS0qx4lEVrPxRgs7QuoL59ZdvZ6cxkn49KJouZoTLxRNmMI5Mje0WjlCmaGD4HgSKQawomRJohoGLHPqyQjOl/IyaZiKJGJU6c+yNkMaiajqf+0R7WFJXyW5ESmH5c4I4LxeUozMuPGXsLZQm6r14P0jBW6Lt0SklODc1gjTBJuF5DbQk21WzJ+1fTwNxQapOgRP1jQRUxuYJI/B1VQcMm+EO7wqt/s2RyaQliM63SBQDJ2LrkBZVl5dcmzXF4nKZ0Uj4DV/FygAkAza4O1p081bwJRGl2dyXhysB9un/w02dbh1/V83o3+CD4KAiDKHgYHAbfB8fBSZB0NjsPO4edL7u/df/o/tn9y5tu3Kp93g8aT/fvfwDACJWS</latexit> Neural sequence labeling ■ Parameterize f with a (deep) neural network. cons: pros: ■ hard to interpret feature ■ shared representations meaning, explain predictions ■ channel external knowledge (e.g. ■ optimization/hyperparameters word embeddings) ■ prone to overfitting ■ high accuracy ■ compute-heavy f ( w , y m , y m − 1 , m ) = . . . . . . 13

  61. Neural sequence labeling 14

  62. Neural sequence labeling word embeddings will </s> Janet <s> back the bill 14

  63. Neural sequence labeling neural network word embeddings will </s> Janet <s> back the bill 14

  64. Neural sequence labeling per-token features neural network word embeddings will </s> Janet <s> back the bill 14

  65. Neural sequence labeling VB? DT? NNP MD NN? per-token features neural network word embeddings will </s> Janet <s> back the bill 14

  66. Neural sequence labeling Bidirectional RNNs word embeddings will </s> Janet <s> back the bill 15

  67. Neural sequence labeling Bidirectional RNNs forward RNN word embeddings will </s> Janet <s> back the bill 15

  68. Neural sequence labeling Bidirectional RNNs forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 15

  69. Neural sequence labeling Bidirectional RNNs per-token features concatenate forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 15

  70. <latexit sha1_base64="/4jVRd01VWx7s0GMy/AWIkSfBo=">AD43icfVJb9MwFHabAaPcOnjkJVBV2hBUbTcJIQ0AQ+8Ia0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u5HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2B7FQLZ1RiBZnuzj817M3ara6nW65wk3QW4EWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ5BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ=</latexit> <latexit sha1_base64="/4jVRd01VWx7s0GMy/AWIkSfBo=">AD43icfVJb9MwFHabAaPcOnjkJVBV2hBUbTcJIQ0AQ+8Ia0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u5HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2B7FQLZ1RiBZnuzj817M3ara6nW65wk3QW4EWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ5BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ=</latexit> Neural sequence labeling Bidirectional RNNs D J N B T P H D M D N R V W A P softmax( ) per-token features concatenate forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 15

  71. Neural sequence labeling Bidirectional RNNs VB? NNP MD per-token features forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 16

  72. Neural sequence labeling Bidirectional RNNs t i-1 VB? NNP MD per-token features forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 16

  73. Neural sequence labeling Bidirectional RNNs t i-1 VB? DT? NNP MD NN? per-token features forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 17

  74. Neural sequence labeling Bidirectional RNNs VB? DT? NNP MD NN? per-token features forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 17

  75. Neural sequence labeling Bidirectional RNNs VB? DT? NNP MD NN? per-token features forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 18

  76. Neural sequence labeling Bidirectional RNNs VB? DT? NNP MD NN? per-token features forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 18

  77. Neural sequence labeling Bidirectional RNN-CRFs VB? DT? NNP MD NN? per-token features forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 19

  78. Neural sequence labeling VB? DT? NNP MD NN? per-token features neural network word embeddings will </s> Janet <s> back the bill 20

  79. Convolutional neural networks 21

  80. Convolutional neural networks ■ In NLP , CNNs merge information across contiguous, fixed-width spans of tokens. 21

  81. Convolutional neural networks ■ In NLP , CNNs merge information across contiguous, fixed-width spans of tokens. ■ Unlike computer vision, in NLP we use 1D CNNs. 21

  82. Convolutional neural networks ■ In NLP , CNNs merge information across contiguous, fixed-width spans of tokens. ■ Unlike computer vision, in NLP we use 1D CNNs. ■ For sentence/document classification: pooling function over representations. wait for the video and do n't rent it n x k representation of Convolutional layer with Fully connected layer Max-over-time sentence with static and multiple filter widths and with dropout and pooling non-static channels feature maps softmax output Figure from: Yoon Kim. Convolutional Neural Networks for Sentence Classification. EMNLP 2014. 21

  83. Convolutional neural networks ■ In NLP , CNNs merge information across contiguous, fixed-width spans of tokens. ■ Unlike computer vision, in NLP we use 1D CNNs. ■ For sentence/document classification: pooling function over representations. ■ For example: sum, average. Most common: max pooling (over time). wait for the video and do n't rent it n x k representation of Convolutional layer with Fully connected layer Max-over-time sentence with static and multiple filter widths and with dropout and pooling non-static channels feature maps softmax output Figure from: Yoon Kim. Convolutional Neural Networks for Sentence Classification. EMNLP 2014. 21

  84. Convolutional neural networks ■ In NLP , CNNs merge information across contiguous, fixed-width spans of tokens. dims: [ d word ] will Janet <s> back the bill 22

  85. Convolutional neural networks ■ In NLP , CNNs merge information across contiguous, fixed-width spans of tokens. dims: [ d word ] will Janet <s> back the bill kernel size = 3 22

  86. Convolutional neural networks ■ In NLP , CNNs merge information across contiguous, fixed-width spans of tokens. dims: [ kd word ] [ d word ] will Janet <s> back the bill kernel size = 3 22

  87. <latexit sha1_base64="oC6lR82ch2B7kQhLCX1aCX53M=">ADtXicfVLbtNAEN3GXIq5pfDIiyGKVBEcVqpSAgpAh54QRSpaStlTbRej5NV92LtrkOC5d/ha3gFib9h7SobipG8vp4Lp4zZyfODO23/+z0/Ju3Lx1e/eOf/fe/QcP23uPTo3KNYURVzp85gY4EzCyDL4TzTQETM4Sy+eF/Fz+agDVPyxC4ziASZSpYySqxzTdrDLo5FsSiDt8F4MSnYq8PyTVCDgw0YbEBYRj4+mYElX4v9BbYq+P68nLQ7/V6/tmAbhGvQWs7nuy1IpwomguQlnJizDjsZzYqiLaMcih9nBvICL0gUxg7KIkAExX1qGXQdZ4kSJV2j7RB7b1cURBhzFLELlMQOzNXY5Xzutg4t+nrqGAy1IumqU5jxwU1a6BQnTQC1fOkCoZo5rQGdE2qdun73cpsZ8DnY5iBURIVJ6+4NSrFw3xokfKNKCKTFwVOiWB8mUBKcm7LApt0g6+T5mUyZ5lZq7RYyeRjDhYrzaZMEs4htbg6m73mlcnz7+AO4uNHxyBD9noIlV2jEheirIonR3M8VPcQX/l8nkv0wHm2MVNQE3TCWBykAWZQ0pVwZwPNUqzxqEt+prou4HJHWKr/KhWbKcAsZXl2/bXA6IUHvcGXw87w3Xo1d9ET9AztoxAdoSH6iI7RCFH0A/1Ev9Bv78iLvMRLV6mtnXNY9QwT/0FBYlBGA=</latexit> Convolutional neural networks ■ In NLP , CNNs merge information across contiguous, fixed-width spans of tokens. dims: [ d z ] # filters [ kd word x d z ] Θ ( x → z ) [ kd word ] [ d word ] will Janet <s> back the bill kernel size = 3 22

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Greedy Algorithms Chapter 16 1 CPTR 430 Algorithms Greedy Algorithms Greedy Algorithms For

Greedy Algorithms Chapter 16 1 CPTR 430 Algorithms Greedy Algorithms Greedy Algorithms For

Greedy Algorithms Chapter 16 1 CPTR 430 Algorithms Greedy Algorithms Greedy Algorithms For some optimization problems, dynamic programming algorithms are overkill A greedy algorithm always makes a choice that looks best at the moment

1.09k views • 65 slides
Evolutionary Algorithms CS 478 - Evolutionary Algorithms 1 Evolutionary Computation/Algorithms

Evolutionary Algorithms CS 478 - Evolutionary Algorithms 1 Evolutionary Computation/Algorithms

Evolutionary Algorithms CS 478 - Evolutionary Algorithms 1 Evolutionary Computation/Algorithms Genetic Algorithms l Simulate natural evolution of structures via selection and reproduction, based on performance (fitness) l Type of

524 views • 30 slides
Graph Algorithms Chapter 22 1 CPTR 430 Algorithms Graph Algorithms Why Study Graph Algorithms?

Graph Algorithms Chapter 22 1 CPTR 430 Algorithms Graph Algorithms Why Study Graph Algorithms?

Graph Algorithms Chapter 22 1 CPTR 430 Algorithms Graph Algorithms Why Study Graph Algorithms? Mathematical graphs seem to be relatively specialized and abstract Why spend so much time and effort on algorithms for such an obscure area?

744 views • 47 slides
- - packing p a - packing algo- packing cking rithms algo- a l g o - theorems rithms

- - packing p a - packing algo- packing cking rithms algo- a l g o - theorems rithms

algorithmic composition dForm + IM WS2010 Inst. f. Arch. a. Media packing algorithms packing algorithms packing algorithms p a c k i n g packing algorithms packing algorithms algorithms packing packing algorithm 1 algorithms - -

39 views • 3 slides
Algorithms Chapter 3 Chapter Summary Algorithms n Example Algorithms n Algorithmic Paradigms

Algorithms Chapter 3 Chapter Summary Algorithms n Example Algorithms n Algorithmic Paradigms

Algorithms Chapter 3 Chapter Summary Algorithms n Example Algorithms n Algorithmic Paradigms Growth of Functions n Big- O and other Notation Complexity of Algorithms Algorithms Section 3.1 Section Summary Properties of Algorithms

847 views • 83 slides
Algorithms Theory Algorithms Theory 10 10 Greedy Algorithms G d Al ith Dr. Alexander

Algorithms Theory Algorithms Theory 10 10 Greedy Algorithms G d Al ith Dr. Alexander

Algorithms Theory Algorithms Theory 10 10 Greedy Algorithms G d Al ith Dr. Alexander Souza Winter term 11/12 Greedy algorithms 1 1. Introductory remarks Introductory remarks 2. Basic examples: The coin-changing problem

504 views • 19 slides
Randomized Algorithms Randomized Algorithms Two Types of Randomized Algorithms Two Types of

Randomized Algorithms Randomized Algorithms Two Types of Randomized Algorithms Two Types of

Randomized Algorithms Randomized Algorithms Two Types of Randomized Algorithms Two Types of Randomized Algorithms and and Some Complexity Classes Some Complexity Classes Speaker: Chuang-Chieh Lin Advisor: Professor Maw-Shang Chang National

910 views • 52 slides
The Role of Algorithms in Computing Chapter 1 1 CPTR 430 Algorithms The Role of Algorithms in

The Role of Algorithms in Computing Chapter 1 1 CPTR 430 Algorithms The Role of Algorithms in

The Role of Algorithms in Computing Chapter 1 1 CPTR 430 Algorithms The Role of Algorithms in Computing What is an Algorithm? Any well-defined computational procedure that takes some value (or set of values)its input and produces some

321 views • 13 slides
Algorithms X. Zhang Fordham Univ. 1 Real World applications of algorithms Algorithms for

Algorithms X. Zhang Fordham Univ. 1 Real World applications of algorithms Algorithms for

Algorithms X. Zhang Fordham Univ. 1 Real World applications of algorithms Algorithms for solving specific, complex, real world problems: Google's success is largely due to its PageRank algorithm, which determines importance&quot;

1.03k views • 46 slides
Mathematical Analysis of Genetic Algorithms Genetic Algorithms are not appropriate for

Mathematical Analysis of Genetic Algorithms Genetic Algorithms are not appropriate for

Mathematical Analysis of Genetic Algorithms Genetic Algorithms are not appropriate for certain problems where finding the exact global optimum is required. Genetic Algorithms are non-deterministic. However, there are theories about

271 views • 5 slides
Genetic Algorithms Welcome to this introduction to Genetic Algorithms. The objective of this paper

Genetic Algorithms Welcome to this introduction to Genetic Algorithms. The objective of this paper

Jesse Anderson CSCI 446, Artificial Intelligence Michelle Van Dyne 11-16-2016 Genetic Algorithms Welcome to this introduction to Genetic Algorithms. The objective of this paper is to inform you, the reader, on Genetic Algorithms and give

564 views • 9 slides
Efficient Algorithms P and NP So far, we have developed algorithms for finding shortest

Efficient Algorithms P and NP So far, we have developed algorithms for finding shortest

Efficient Algorithms P and NP So far, we have developed algorithms for finding shortest paths in graphs, CISC5835, Algorithms for Big Data minimum spanning trees in graphs, CIS, Fordham Univ. matchings in bipartite graphs,

514 views • 8 slides
Algorithms in Nature Nature inspired algorithms http://www.cs.cmu.edu/~02317/ Ziv Bar-Joseph

Algorithms in Nature Nature inspired algorithms http://www.cs.cmu.edu/~02317/ Ziv Bar-Joseph

Algorithms in Nature Nature inspired algorithms http://www.cs.cmu.edu/~02317/ Ziv Bar-Joseph Matt Ruffalo zivbj@cs.cmu.edu mruffalo@andrew.cmu.edu GHC 7715 GHC 8006 Topics Introduction (1 Week) Classic algorithms (4 weeks)

1.68k views • 28 slides
Algorithms for Big Data CISC5835 Fordham Univ. Instructor: X. Zhang Lecture 1 Outline

Algorithms for Big Data CISC5835 Fordham Univ. Instructor: X. Zhang Lecture 1 Outline

Algorithms for Big Data CISC5835 Fordham Univ. Instructor: X. Zhang Lecture 1 Outline What is algorithm: word origin, first algorithms, algorithms of todays world Sequential algorithms, Parallel algorithms, approximation

828 views • 62 slides
Algorithms for Big Data CISC5835 Fordham Univ. Instructor: X. Zhang Lecture 1 Outline

Algorithms for Big Data CISC5835 Fordham Univ. Instructor: X. Zhang Lecture 1 Outline

Algorithms for Big Data CISC5835 Fordham Univ. Instructor: X. Zhang Lecture 1 Outline What is algorithm: word origin, first algorithms, algorithms of todays world Sequential algorithms, Parallel algorithms, approximation

332 views • 21 slides
Algorithms and Architecture 1 Introduction to Algorithms Alexandre David 1 Outline Notion

Algorithms and Architecture 1 Introduction to Algorithms Alexandre David 1 Outline Notion

Algorithms and Architecture 1 Introduction to Algorithms Alexandre David 1 Outline Notion of algorithms, GCD example. Algorithmic problem solving. Problem types. Sorting example. Numerical example. Analyzing algorithms.

333 views • 12 slides
GTrans Service Change Community Meeting September 7, 2019 GTrans Administration Building Dana

GTrans Service Change Community Meeting September 7, 2019 GTrans Administration Building Dana

GTrans Service Change Community Meeting September 7, 2019 GTrans Administration Building Dana Pynn Transit Administrative Officer Agenda 1. Welcome and Introductions 2. Overview of proposed service changes 3. Questions and answers section 4.

971 views • 15 slides
Building Java Programs Chapter 6 Lecture 6-2: Line-Based File Input reading: 6.3 - 6.5 2 Hours

Building Java Programs Chapter 6 Lecture 6-2: Line-Based File Input reading: 6.3 - 6.5 2 Hours

Building Java Programs Chapter 6 Lecture 6-2: Line-Based File Input reading: 6.3 - 6.5 2 Hours question Given a file hours.txt with the following contents: 123 Cody 12.5 8.2 7.6 4.0 456 Rachel 4.2 11.6 6.3 2.5 12.0 789 Riley 16.0 12.0 8.0

361 views • 19 slides
Podman, Buildah and Skopeo coreutils for containers who am i Aleksandra Fedorova

Podman, Buildah and Skopeo coreutils for containers who am i Aleksandra Fedorova

Podman, Buildah and Skopeo coreutils for containers who am i Aleksandra Fedorova DevOps/CI/Build Engineer CI Engineer at Red Hat Fedora Linux Ambassador https://medium.com/quantum-integration Fedora User Group NRW @

521 views • 24 slides
Efficient Validation of FOL ID Cyclic Induction Reasoning VeriDis + MATRYOSHKA Workshop, Amsterdam

Efficient Validation of FOL ID Cyclic Induction Reasoning VeriDis + MATRYOSHKA Workshop, Amsterdam

Efficient Validation of FOL ID Cyclic Induction Reasoning VeriDis + MATRYOSHKA Workshop, Amsterdam June 12, 2019 Sorin Stratulat INRIA, Universit de Lorraine Motivation soundness checking of cyclic pre-proofs in FOL with inductive

675 views • 43 slides
Evolution : Comparing Biology and Culture Andy Wedel Department of Linguistics University of

Evolution : Comparing Biology and Culture Andy Wedel Department of Linguistics University of

Evolution : Comparing Biology and Culture Andy Wedel Department of Linguistics University of Arizona Evolution as a population- based mechanism of change Requirements for evolutionary change in the frequency of a trait: Population of

426 views • 40 slides
Wide-coverage Translation in GF Krasimir Angelov University of Gothenburg, Digital Grammars AB

Wide-coverage Translation in GF Krasimir Angelov University of Gothenburg, Digital Grammars AB

Wide-coverage Translation in GF Krasimir Angelov University of Gothenburg, Digital Grammars AB August 18, 2017 Introduction 1 Free Language 2 Resource Grammar Resource Lexicon Demo Uncontrolled Language 3 Demo Summary 4 Get me out of

927 views • 67 slides
Inflectional Morphology for Slavonic Languages in DATR Velis islava ava St Stoykov ykova

Inflectional Morphology for Slavonic Languages in DATR Velis islava ava St Stoykov ykova

Representing Nominal Inflectional Morphology for Slavonic Languages in DATR Velis islava ava St Stoykov ykova Institute of Bulgarian Language BAS Bulgarian Academy of Sciences vili1@bas.bg Introduction Presenting inflectional

471 views • 31 slides
Projective Dependency Parsing with Perceptron Xavier Carreras , Mihai Surdeanu, and Llus

Projective Dependency Parsing with Perceptron Xavier Carreras , Mihai Surdeanu, and Llus

Projective Dependency Parsing with Perceptron Xavier Carreras , Mihai Surdeanu, and Llus Mrquez Technical University of Catalonia {carreras,surdeanu,lluism}@lsi.upc.edu 8th June 2006 Outline Introduction Parsing and Learning Parsing

356 views • 22 slides

More recommend