Adversarially Robust Generalization Requires More Data Ludwig - - PowerPoint PPT Presentation

▶
adversarially robust generalization requires more data
SMART_READER_LITE
LIVE PREVIEW

Adversarially Robust Generalization Requires More Data Ludwig - - PowerPoint PPT Presentation

Dec 06, 2023 2.36k likes •2.54k views

Adversarially Robust Generalization Requires More Data Ludwig Schmidt Shibani Santurkar Dimitris Tsipras Poster #31 Kunal Talwar Aleksander M dry Adversarial Examples [Szegedy, Zaremba, Sutskever, Bruna, Erhan, Goodfellow,

slide-1
SLIDE 1

Adversarially Robust Generalization Requires More Data

Ludwig Schmidt Shibani Santurkar Dimitris Tsipras Kunal Talwar Aleksander Mądry

Poster #31

slide-2
SLIDE 2

Adversarial Examples

[Szegedy, Zaremba, Sutskever, Bruna, Erhan, Goodfellow, Fergus, 2013] [Biggio, Corona, Maiorca, Nelson, Srndic, Laskov, Giacinto, Roli, 2013]

slide-3
SLIDE 3

Adversarial Examples

What makes adversarial examples a hard problem? This paper: perspective on sample complexity

[Szegedy, Zaremba, Sutskever, Bruna, Erhan, Goodfellow, Fergus, 2013] [Biggio, Corona, Maiorca, Nelson, Srndic, Laskov, Giacinto, Roli, 2013]

slide-4
SLIDE 4

Standard vs Robust Generalization

“Standard” Generalization:

E

x,y∼D [ loss(f(x), y) ]

<latexit sha1_base64="TCezhzHnvwDoFMFoQ4EK/L9JVgU=">ADOHicdVJdaxNBFJ2sXzV+NVHXwaDkEosiQj6IpRapULVCqYt7CxhdnI3GTq7s8zcLQnD/jHfPBH+OZb8c32Fzi7CaVt6oWFs+eO/fOuRPnSlrs9X41ghs3b92+s3K3e/+g4erbVH+1YXRsBAaKXNYcwtKJnBACUqOMwN8DRWcBAfvavyB8dgrNTZN5zlEKV8nMlECo6eGrZClnKc6NyxXW7GUP/FsXtflkM37dIZVamtKYFV267LClTkGBIWZcyhCk6pa0tO0lnu716zVv5HiC0bDV7m306qDLoL8AbKIveFa4ycbaVGkKFQ3Nqw38sxctygFArKJis5Fwc8TGEHmY8BRu52oWSPvPMiCba+C9DWrMXKxPrZ2lsVdW17FXcxV5XS4sMHkTOZnlBUIm5o2SQlHUtLKUjqQBgWrmARdG+lmpmHDBXrjL3WpBrM5CH8TC5hymVM2KSUfkSupPhQTV7FW7oD6hj8WfwzFPBit3J0rulW8i2tRufiZflAoeF1zbn6QoP/q+eiyF3TvGw2Tb4tRj45C36koPhqM1zx/zDSWVW+jWNWbdCfvH9q2teBvsvN/oef3V3txaPIEV8oQ8JR3SJ6/Jtkhe2RABPlBTshfchp8D34HJ8GfuTRoLGoek0sRnP0DAYoHOQ=</latexit><latexit sha1_base64="TCezhzHnvwDoFMFoQ4EK/L9JVgU=">ADOHicdVJdaxNBFJ2sXzV+NVHXwaDkEosiQj6IpRapULVCqYt7CxhdnI3GTq7s8zcLQnD/jHfPBH+OZb8c32Fzi7CaVt6oWFs+eO/fOuRPnSlrs9X41ghs3b92+s3K3e/+g4erbVH+1YXRsBAaKXNYcwtKJnBACUqOMwN8DRWcBAfvavyB8dgrNTZN5zlEKV8nMlECo6eGrZClnKc6NyxXW7GUP/FsXtflkM37dIZVamtKYFV267LClTkGBIWZcyhCk6pa0tO0lnu716zVv5HiC0bDV7m306qDLoL8AbKIveFa4ycbaVGkKFQ3Nqw38sxctygFArKJis5Fwc8TGEHmY8BRu52oWSPvPMiCba+C9DWrMXKxPrZ2lsVdW17FXcxV5XS4sMHkTOZnlBUIm5o2SQlHUtLKUjqQBgWrmARdG+lmpmHDBXrjL3WpBrM5CH8TC5hymVM2KSUfkSupPhQTV7FW7oD6hj8WfwzFPBit3J0rulW8i2tRufiZflAoeF1zbn6QoP/q+eiyF3TvGw2Tb4tRj45C36koPhqM1zx/zDSWVW+jWNWbdCfvH9q2teBvsvN/oef3V3txaPIEV8oQ8JR3SJ6/Jtkhe2RABPlBTshfchp8D34HJ8GfuTRoLGoek0sRnP0DAYoHOQ=</latexit><latexit sha1_base64="TCezhzHnvwDoFMFoQ4EK/L9JVgU=">ADOHicdVJdaxNBFJ2sXzV+NVHXwaDkEosiQj6IpRapULVCqYt7CxhdnI3GTq7s8zcLQnD/jHfPBH+OZb8c32Fzi7CaVt6oWFs+eO/fOuRPnSlrs9X41ghs3b92+s3K3e/+g4erbVH+1YXRsBAaKXNYcwtKJnBACUqOMwN8DRWcBAfvavyB8dgrNTZN5zlEKV8nMlECo6eGrZClnKc6NyxXW7GUP/FsXtflkM37dIZVamtKYFV267LClTkGBIWZcyhCk6pa0tO0lnu716zVv5HiC0bDV7m306qDLoL8AbKIveFa4ycbaVGkKFQ3Nqw38sxctygFArKJis5Fwc8TGEHmY8BRu52oWSPvPMiCba+C9DWrMXKxPrZ2lsVdW17FXcxV5XS4sMHkTOZnlBUIm5o2SQlHUtLKUjqQBgWrmARdG+lmpmHDBXrjL3WpBrM5CH8TC5hymVM2KSUfkSupPhQTV7FW7oD6hj8WfwzFPBit3J0rulW8i2tRufiZflAoeF1zbn6QoP/q+eiyF3TvGw2Tb4tRj45C36koPhqM1zx/zDSWVW+jWNWbdCfvH9q2teBvsvN/oef3V3txaPIEV8oQ8JR3SJ6/Jtkhe2RABPlBTshfchp8D34HJ8GfuTRoLGoek0sRnP0DAYoHOQ=</latexit><latexit sha1_base64="TCezhzHnvwDoFMFoQ4EK/L9JVgU=">ADOHicdVJdaxNBFJ2sXzV+NVHXwaDkEosiQj6IpRapULVCqYt7CxhdnI3GTq7s8zcLQnD/jHfPBH+OZb8c32Fzi7CaVt6oWFs+eO/fOuRPnSlrs9X41ghs3b92+s3K3e/+g4erbVH+1YXRsBAaKXNYcwtKJnBACUqOMwN8DRWcBAfvavyB8dgrNTZN5zlEKV8nMlECo6eGrZClnKc6NyxXW7GUP/FsXtflkM37dIZVamtKYFV267LClTkGBIWZcyhCk6pa0tO0lnu716zVv5HiC0bDV7m306qDLoL8AbKIveFa4ycbaVGkKFQ3Nqw38sxctygFArKJis5Fwc8TGEHmY8BRu52oWSPvPMiCba+C9DWrMXKxPrZ2lsVdW17FXcxV5XS4sMHkTOZnlBUIm5o2SQlHUtLKUjqQBgWrmARdG+lmpmHDBXrjL3WpBrM5CH8TC5hymVM2KSUfkSupPhQTV7FW7oD6hj8WfwzFPBit3J0rulW8i2tRufiZflAoeF1zbn6QoP/q+eiyF3TvGw2Tb4tRj45C36koPhqM1zx/zDSWVW+jWNWbdCfvH9q2teBvsvN/oef3V3txaPIEV8oQ8JR3SJ6/Jtkhe2RABPlBTshfchp8D34HJ8GfuTRoLGoek0sRnP0DAYoHOQ=</latexit>
slide-5
SLIDE 5

Standard vs Robust Generalization

Adversarially robust generalization:

Perturbation set: small -perturbations, rotations, translations, …

E

x,y⇠D

 max

x02P (x)loss(f(x0), y)

  • <latexit sha1_base64="ezivjb9Zu0IHh3owel+6Sq/LS5I=">ADbnicdZJtaxNBEMf3Gh9qfEoVfCPFxaBJSiyJCPpGKLVKhaoRTFvIHmFvM0mW7j2wO1cSlvsQfig/hN/Cd7519xJKm9SBg/O/GZnbmajTEmDnc7vYKNy4+at25t3qnfv3X/wsLb16NikuRbQF6lK9WnEDSiZQB8lKjNPA4UnASnX3w8ZNz0EamyQ+cZxDGfJLIsRQcnWtY+0lZzHGaZpYdcT2B8hRF9mNRDO2sPafMyHjBCK7sQVFQpmCMA8ralCHMsOzBahgV1mEzl9WgTCa015y1PO0Zq1Jjiua4uZoxaxStNp23ytu0nEwxHNbqnd1OaXRdJeiTpbWG24Fv9goFXkMCQrFjRl0OxmGlmuUQkFRZbmBjIszPoGBkwmPwYS27KgL5xnRMepdl+CtPRezrA8NmYeR470QzCrMe+8LjbIcfwutDLJcoRELAqNc0UxpX4RdCQ1CFRzJ7jQ0vVKxZRrLtCt60oV35jJQLg/MYAxl4n3DKqU0s/IlRSfOfe3tNDUOfg7uJfIYdXR36iC6bt8f1UjS7gdbyvUPMy54K+VOD/9AIK7TXFi2qVHYBbi4YvbkTfMtAcU71jmXtsUwKt6YJa3vlFt9dXfO6OH6923X6+5v63v7yCWySp+Q5aZIueUv2yCHpkT4R5G+wHbwMGht/Kk8q25VnC3QjWOY8Jles0vwHPbMXnQ=</latexit><latexit sha1_base64="ezivjb9Zu0IHh3owel+6Sq/LS5I=">ADbnicdZJtaxNBEMf3Gh9qfEoVfCPFxaBJSiyJCPpGKLVKhaoRTFvIHmFvM0mW7j2wO1cSlvsQfig/hN/Cd7519xJKm9SBg/O/GZnbmajTEmDnc7vYKNy4+at25t3qnfv3X/wsLb16NikuRbQF6lK9WnEDSiZQB8lKjNPA4UnASnX3w8ZNz0EamyQ+cZxDGfJLIsRQcnWtY+0lZzHGaZpYdcT2B8hRF9mNRDO2sPafMyHjBCK7sQVFQpmCMA8ralCHMsOzBahgV1mEzl9WgTCa015y1PO0Zq1Jjiua4uZoxaxStNp23ytu0nEwxHNbqnd1OaXRdJeiTpbWG24Fv9goFXkMCQrFjRl0OxmGlmuUQkFRZbmBjIszPoGBkwmPwYS27KgL5xnRMepdl+CtPRezrA8NmYeR470QzCrMe+8LjbIcfwutDLJcoRELAqNc0UxpX4RdCQ1CFRzJ7jQ0vVKxZRrLtCt60oV35jJQLg/MYAxl4n3DKqU0s/IlRSfOfe3tNDUOfg7uJfIYdXR36iC6bt8f1UjS7gdbyvUPMy54K+VOD/9AIK7TXFi2qVHYBbi4YvbkTfMtAcU71jmXtsUwKt6YJa3vlFt9dXfO6OH6923X6+5v63v7yCWySp+Q5aZIueUv2yCHpkT4R5G+wHbwMGht/Kk8q25VnC3QjWOY8Jles0vwHPbMXnQ=</latexit><latexit sha1_base64="ezivjb9Zu0IHh3owel+6Sq/LS5I=">ADbnicdZJtaxNBEMf3Gh9qfEoVfCPFxaBJSiyJCPpGKLVKhaoRTFvIHmFvM0mW7j2wO1cSlvsQfig/hN/Cd7519xJKm9SBg/O/GZnbmajTEmDnc7vYKNy4+at25t3qnfv3X/wsLb16NikuRbQF6lK9WnEDSiZQB8lKjNPA4UnASnX3w8ZNz0EamyQ+cZxDGfJLIsRQcnWtY+0lZzHGaZpYdcT2B8hRF9mNRDO2sPafMyHjBCK7sQVFQpmCMA8ralCHMsOzBahgV1mEzl9WgTCa015y1PO0Zq1Jjiua4uZoxaxStNp23ytu0nEwxHNbqnd1OaXRdJeiTpbWG24Fv9goFXkMCQrFjRl0OxmGlmuUQkFRZbmBjIszPoGBkwmPwYS27KgL5xnRMepdl+CtPRezrA8NmYeR470QzCrMe+8LjbIcfwutDLJcoRELAqNc0UxpX4RdCQ1CFRzJ7jQ0vVKxZRrLtCt60oV35jJQLg/MYAxl4n3DKqU0s/IlRSfOfe3tNDUOfg7uJfIYdXR36iC6bt8f1UjS7gdbyvUPMy54K+VOD/9AIK7TXFi2qVHYBbi4YvbkTfMtAcU71jmXtsUwKt6YJa3vlFt9dXfO6OH6923X6+5v63v7yCWySp+Q5aZIueUv2yCHpkT4R5G+wHbwMGht/Kk8q25VnC3QjWOY8Jles0vwHPbMXnQ=</latexit><latexit sha1_base64="ezivjb9Zu0IHh3owel+6Sq/LS5I=">ADbnicdZJtaxNBEMf3Gh9qfEoVfCPFxaBJSiyJCPpGKLVKhaoRTFvIHmFvM0mW7j2wO1cSlvsQfig/hN/Cd7519xJKm9SBg/O/GZnbmajTEmDnc7vYKNy4+at25t3qnfv3X/wsLb16NikuRbQF6lK9WnEDSiZQB8lKjNPA4UnASnX3w8ZNz0EamyQ+cZxDGfJLIsRQcnWtY+0lZzHGaZpYdcT2B8hRF9mNRDO2sPafMyHjBCK7sQVFQpmCMA8ralCHMsOzBahgV1mEzl9WgTCa015y1PO0Zq1Jjiua4uZoxaxStNp23ytu0nEwxHNbqnd1OaXRdJeiTpbWG24Fv9goFXkMCQrFjRl0OxmGlmuUQkFRZbmBjIszPoGBkwmPwYS27KgL5xnRMepdl+CtPRezrA8NmYeR470QzCrMe+8LjbIcfwutDLJcoRELAqNc0UxpX4RdCQ1CFRzJ7jQ0vVKxZRrLtCt60oV35jJQLg/MYAxl4n3DKqU0s/IlRSfOfe3tNDUOfg7uJfIYdXR36iC6bt8f1UjS7gdbyvUPMy54K+VOD/9AIK7TXFi2qVHYBbi4YvbkTfMtAcU71jmXtsUwKt6YJa3vlFt9dXfO6OH6923X6+5v63v7yCWySp+Q5aZIueUv2yCHpkT4R5G+wHbwMGht/Kk8q25VnC3QjWOY8Jles0vwHPbMXnQ=</latexit>

`∞

“Standard” Generalization:

E

x,y∼D [ loss(f(x), y) ]

<latexit sha1_base64="TCezhzHnvwDoFMFoQ4EK/L9JVgU=">ADOHicdVJdaxNBFJ2sXzV+NVHXwaDkEosiQj6IpRapULVCqYt7CxhdnI3GTq7s8zcLQnD/jHfPBH+OZb8c32Fzi7CaVt6oWFs+eO/fOuRPnSlrs9X41ghs3b92+s3K3e/+g4erbVH+1YXRsBAaKXNYcwtKJnBACUqOMwN8DRWcBAfvavyB8dgrNTZN5zlEKV8nMlECo6eGrZClnKc6NyxXW7GUP/FsXtflkM37dIZVamtKYFV267LClTkGBIWZcyhCk6pa0tO0lnu716zVv5HiC0bDV7m306qDLoL8AbKIveFa4ycbaVGkKFQ3Nqw38sxctygFArKJis5Fwc8TGEHmY8BRu52oWSPvPMiCba+C9DWrMXKxPrZ2lsVdW17FXcxV5XS4sMHkTOZnlBUIm5o2SQlHUtLKUjqQBgWrmARdG+lmpmHDBXrjL3WpBrM5CH8TC5hymVM2KSUfkSupPhQTV7FW7oD6hj8WfwzFPBit3J0rulW8i2tRufiZflAoeF1zbn6QoP/q+eiyF3TvGw2Tb4tRj45C36koPhqM1zx/zDSWVW+jWNWbdCfvH9q2teBvsvN/oef3V3txaPIEV8oQ8JR3SJ6/Jtkhe2RABPlBTshfchp8D34HJ8GfuTRoLGoek0sRnP0DAYoHOQ=</latexit><latexit sha1_base64="TCezhzHnvwDoFMFoQ4EK/L9JVgU=">ADOHicdVJdaxNBFJ2sXzV+NVHXwaDkEosiQj6IpRapULVCqYt7CxhdnI3GTq7s8zcLQnD/jHfPBH+OZb8c32Fzi7CaVt6oWFs+eO/fOuRPnSlrs9X41ghs3b92+s3K3e/+g4erbVH+1YXRsBAaKXNYcwtKJnBACUqOMwN8DRWcBAfvavyB8dgrNTZN5zlEKV8nMlECo6eGrZClnKc6NyxXW7GUP/FsXtflkM37dIZVamtKYFV267LClTkGBIWZcyhCk6pa0tO0lnu716zVv5HiC0bDV7m306qDLoL8AbKIveFa4ycbaVGkKFQ3Nqw38sxctygFArKJis5Fwc8TGEHmY8BRu52oWSPvPMiCba+C9DWrMXKxPrZ2lsVdW17FXcxV5XS4sMHkTOZnlBUIm5o2SQlHUtLKUjqQBgWrmARdG+lmpmHDBXrjL3WpBrM5CH8TC5hymVM2KSUfkSupPhQTV7FW7oD6hj8WfwzFPBit3J0rulW8i2tRufiZflAoeF1zbn6QoP/q+eiyF3TvGw2Tb4tRj45C36koPhqM1zx/zDSWVW+jWNWbdCfvH9q2teBvsvN/oef3V3txaPIEV8oQ8JR3SJ6/Jtkhe2RABPlBTshfchp8D34HJ8GfuTRoLGoek0sRnP0DAYoHOQ=</latexit><latexit sha1_base64="TCezhzHnvwDoFMFoQ4EK/L9JVgU=">ADOHicdVJdaxNBFJ2sXzV+NVHXwaDkEosiQj6IpRapULVCqYt7CxhdnI3GTq7s8zcLQnD/jHfPBH+OZb8c32Fzi7CaVt6oWFs+eO/fOuRPnSlrs9X41ghs3b92+s3K3e/+g4erbVH+1YXRsBAaKXNYcwtKJnBACUqOMwN8DRWcBAfvavyB8dgrNTZN5zlEKV8nMlECo6eGrZClnKc6NyxXW7GUP/FsXtflkM37dIZVamtKYFV267LClTkGBIWZcyhCk6pa0tO0lnu716zVv5HiC0bDV7m306qDLoL8AbKIveFa4ycbaVGkKFQ3Nqw38sxctygFArKJis5Fwc8TGEHmY8BRu52oWSPvPMiCba+C9DWrMXKxPrZ2lsVdW17FXcxV5XS4sMHkTOZnlBUIm5o2SQlHUtLKUjqQBgWrmARdG+lmpmHDBXrjL3WpBrM5CH8TC5hymVM2KSUfkSupPhQTV7FW7oD6hj8WfwzFPBit3J0rulW8i2tRufiZflAoeF1zbn6QoP/q+eiyF3TvGw2Tb4tRj45C36koPhqM1zx/zDSWVW+jWNWbdCfvH9q2teBvsvN/oef3V3txaPIEV8oQ8JR3SJ6/Jtkhe2RABPlBTshfchp8D34HJ8GfuTRoLGoek0sRnP0DAYoHOQ=</latexit><latexit sha1_base64="TCezhzHnvwDoFMFoQ4EK/L9JVgU=">ADOHicdVJdaxNBFJ2sXzV+NVHXwaDkEosiQj6IpRapULVCqYt7CxhdnI3GTq7s8zcLQnD/jHfPBH+OZb8c32Fzi7CaVt6oWFs+eO/fOuRPnSlrs9X41ghs3b92+s3K3e/+g4erbVH+1YXRsBAaKXNYcwtKJnBACUqOMwN8DRWcBAfvavyB8dgrNTZN5zlEKV8nMlECo6eGrZClnKc6NyxXW7GUP/FsXtflkM37dIZVamtKYFV267LClTkGBIWZcyhCk6pa0tO0lnu716zVv5HiC0bDV7m306qDLoL8AbKIveFa4ycbaVGkKFQ3Nqw38sxctygFArKJis5Fwc8TGEHmY8BRu52oWSPvPMiCba+C9DWrMXKxPrZ2lsVdW17FXcxV5XS4sMHkTOZnlBUIm5o2SQlHUtLKUjqQBgWrmARdG+lmpmHDBXrjL3WpBrM5CH8TC5hymVM2KSUfkSupPhQTV7FW7oD6hj8WfwzFPBit3J0rulW8i2tRufiZflAoeF1zbn6QoP/q+eiyF3TvGw2Tb4tRj45C36koPhqM1zx/zDSWVW+jWNWbdCfvH9q2teBvsvN/oef3V3txaPIEV8oQ8JR3SJ6/Jtkhe2RABPlBTshfchp8D34HJ8GfuTRoLGoek0sRnP0DAYoHOQ=</latexit>
slide-6
SLIDE 6

Standard vs Robust Generalization

Adversarially robust generalization:

Perturbation set: small -perturbations, rotations, translations, …

E

x,y⇠D

 max

x02P (x)loss(f(x0), y)

  • <latexit sha1_base64="ezivjb9Zu0IHh3owel+6Sq/LS5I=">ADbnicdZJtaxNBEMf3Gh9qfEoVfCPFxaBJSiyJCPpGKLVKhaoRTFvIHmFvM0mW7j2wO1cSlvsQfig/hN/Cd7519xJKm9SBg/O/GZnbmajTEmDnc7vYKNy4+at25t3qnfv3X/wsLb16NikuRbQF6lK9WnEDSiZQB8lKjNPA4UnASnX3w8ZNz0EamyQ+cZxDGfJLIsRQcnWtY+0lZzHGaZpYdcT2B8hRF9mNRDO2sPafMyHjBCK7sQVFQpmCMA8ralCHMsOzBahgV1mEzl9WgTCa015y1PO0Zq1Jjiua4uZoxaxStNp23ytu0nEwxHNbqnd1OaXRdJeiTpbWG24Fv9goFXkMCQrFjRl0OxmGlmuUQkFRZbmBjIszPoGBkwmPwYS27KgL5xnRMepdl+CtPRezrA8NmYeR470QzCrMe+8LjbIcfwutDLJcoRELAqNc0UxpX4RdCQ1CFRzJ7jQ0vVKxZRrLtCt60oV35jJQLg/MYAxl4n3DKqU0s/IlRSfOfe3tNDUOfg7uJfIYdXR36iC6bt8f1UjS7gdbyvUPMy54K+VOD/9AIK7TXFi2qVHYBbi4YvbkTfMtAcU71jmXtsUwKt6YJa3vlFt9dXfO6OH6923X6+5v63v7yCWySp+Q5aZIueUv2yCHpkT4R5G+wHbwMGht/Kk8q25VnC3QjWOY8Jles0vwHPbMXnQ=</latexit><latexit sha1_base64="ezivjb9Zu0IHh3owel+6Sq/LS5I=">ADbnicdZJtaxNBEMf3Gh9qfEoVfCPFxaBJSiyJCPpGKLVKhaoRTFvIHmFvM0mW7j2wO1cSlvsQfig/hN/Cd7519xJKm9SBg/O/GZnbmajTEmDnc7vYKNy4+at25t3qnfv3X/wsLb16NikuRbQF6lK9WnEDSiZQB8lKjNPA4UnASnX3w8ZNz0EamyQ+cZxDGfJLIsRQcnWtY+0lZzHGaZpYdcT2B8hRF9mNRDO2sPafMyHjBCK7sQVFQpmCMA8ralCHMsOzBahgV1mEzl9WgTCa015y1PO0Zq1Jjiua4uZoxaxStNp23ytu0nEwxHNbqnd1OaXRdJeiTpbWG24Fv9goFXkMCQrFjRl0OxmGlmuUQkFRZbmBjIszPoGBkwmPwYS27KgL5xnRMepdl+CtPRezrA8NmYeR470QzCrMe+8LjbIcfwutDLJcoRELAqNc0UxpX4RdCQ1CFRzJ7jQ0vVKxZRrLtCt60oV35jJQLg/MYAxl4n3DKqU0s/IlRSfOfe3tNDUOfg7uJfIYdXR36iC6bt8f1UjS7gdbyvUPMy54K+VOD/9AIK7TXFi2qVHYBbi4YvbkTfMtAcU71jmXtsUwKt6YJa3vlFt9dXfO6OH6923X6+5v63v7yCWySp+Q5aZIueUv2yCHpkT4R5G+wHbwMGht/Kk8q25VnC3QjWOY8Jles0vwHPbMXnQ=</latexit><latexit sha1_base64="ezivjb9Zu0IHh3owel+6Sq/LS5I=">ADbnicdZJtaxNBEMf3Gh9qfEoVfCPFxaBJSiyJCPpGKLVKhaoRTFvIHmFvM0mW7j2wO1cSlvsQfig/hN/Cd7519xJKm9SBg/O/GZnbmajTEmDnc7vYKNy4+at25t3qnfv3X/wsLb16NikuRbQF6lK9WnEDSiZQB8lKjNPA4UnASnX3w8ZNz0EamyQ+cZxDGfJLIsRQcnWtY+0lZzHGaZpYdcT2B8hRF9mNRDO2sPafMyHjBCK7sQVFQpmCMA8ralCHMsOzBahgV1mEzl9WgTCa015y1PO0Zq1Jjiua4uZoxaxStNp23ytu0nEwxHNbqnd1OaXRdJeiTpbWG24Fv9goFXkMCQrFjRl0OxmGlmuUQkFRZbmBjIszPoGBkwmPwYS27KgL5xnRMepdl+CtPRezrA8NmYeR470QzCrMe+8LjbIcfwutDLJcoRELAqNc0UxpX4RdCQ1CFRzJ7jQ0vVKxZRrLtCt60oV35jJQLg/MYAxl4n3DKqU0s/IlRSfOfe3tNDUOfg7uJfIYdXR36iC6bt8f1UjS7gdbyvUPMy54K+VOD/9AIK7TXFi2qVHYBbi4YvbkTfMtAcU71jmXtsUwKt6YJa3vlFt9dXfO6OH6923X6+5v63v7yCWySp+Q5aZIueUv2yCHpkT4R5G+wHbwMGht/Kk8q25VnC3QjWOY8Jles0vwHPbMXnQ=</latexit><latexit sha1_base64="ezivjb9Zu0IHh3owel+6Sq/LS5I=">ADbnicdZJtaxNBEMf3Gh9qfEoVfCPFxaBJSiyJCPpGKLVKhaoRTFvIHmFvM0mW7j2wO1cSlvsQfig/hN/Cd7519xJKm9SBg/O/GZnbmajTEmDnc7vYKNy4+at25t3qnfv3X/wsLb16NikuRbQF6lK9WnEDSiZQB8lKjNPA4UnASnX3w8ZNz0EamyQ+cZxDGfJLIsRQcnWtY+0lZzHGaZpYdcT2B8hRF9mNRDO2sPafMyHjBCK7sQVFQpmCMA8ralCHMsOzBahgV1mEzl9WgTCa015y1PO0Zq1Jjiua4uZoxaxStNp23ytu0nEwxHNbqnd1OaXRdJeiTpbWG24Fv9goFXkMCQrFjRl0OxmGlmuUQkFRZbmBjIszPoGBkwmPwYS27KgL5xnRMepdl+CtPRezrA8NmYeR470QzCrMe+8LjbIcfwutDLJcoRELAqNc0UxpX4RdCQ1CFRzJ7jQ0vVKxZRrLtCt60oV35jJQLg/MYAxl4n3DKqU0s/IlRSfOfe3tNDUOfg7uJfIYdXR36iC6bt8f1UjS7gdbyvUPMy54K+VOD/9AIK7TXFi2qVHYBbi4YvbkTfMtAcU71jmXtsUwKt6YJa3vlFt9dXfO6OH6923X6+5v63v7yCWySp+Q5aZIueUv2yCHpkT4R5G+wHbwMGht/Kk8q25VnC3QjWOY8Jles0vwHPbMXnQ=</latexit>

`∞

“Standard” Generalization:

E

x,y∼D [ loss(f(x), y) ]

<latexit sha1_base64="TCezhzHnvwDoFMFoQ4EK/L9JVgU=">ADOHicdVJdaxNBFJ2sXzV+NVHXwaDkEosiQj6IpRapULVCqYt7CxhdnI3GTq7s8zcLQnD/jHfPBH+OZb8c32Fzi7CaVt6oWFs+eO/fOuRPnSlrs9X41ghs3b92+s3K3e/+g4erbVH+1YXRsBAaKXNYcwtKJnBACUqOMwN8DRWcBAfvavyB8dgrNTZN5zlEKV8nMlECo6eGrZClnKc6NyxXW7GUP/FsXtflkM37dIZVamtKYFV267LClTkGBIWZcyhCk6pa0tO0lnu716zVv5HiC0bDV7m306qDLoL8AbKIveFa4ycbaVGkKFQ3Nqw38sxctygFArKJis5Fwc8TGEHmY8BRu52oWSPvPMiCba+C9DWrMXKxPrZ2lsVdW17FXcxV5XS4sMHkTOZnlBUIm5o2SQlHUtLKUjqQBgWrmARdG+lmpmHDBXrjL3WpBrM5CH8TC5hymVM2KSUfkSupPhQTV7FW7oD6hj8WfwzFPBit3J0rulW8i2tRufiZflAoeF1zbn6QoP/q+eiyF3TvGw2Tb4tRj45C36koPhqM1zx/zDSWVW+jWNWbdCfvH9q2teBvsvN/oef3V3txaPIEV8oQ8JR3SJ6/Jtkhe2RABPlBTshfchp8D34HJ8GfuTRoLGoek0sRnP0DAYoHOQ=</latexit><latexit sha1_base64="TCezhzHnvwDoFMFoQ4EK/L9JVgU=">ADOHicdVJdaxNBFJ2sXzV+NVHXwaDkEosiQj6IpRapULVCqYt7CxhdnI3GTq7s8zcLQnD/jHfPBH+OZb8c32Fzi7CaVt6oWFs+eO/fOuRPnSlrs9X41ghs3b92+s3K3e/+g4erbVH+1YXRsBAaKXNYcwtKJnBACUqOMwN8DRWcBAfvavyB8dgrNTZN5zlEKV8nMlECo6eGrZClnKc6NyxXW7GUP/FsXtflkM37dIZVamtKYFV267LClTkGBIWZcyhCk6pa0tO0lnu716zVv5HiC0bDV7m306qDLoL8AbKIveFa4ycbaVGkKFQ3Nqw38sxctygFArKJis5Fwc8TGEHmY8BRu52oWSPvPMiCba+C9DWrMXKxPrZ2lsVdW17FXcxV5XS4sMHkTOZnlBUIm5o2SQlHUtLKUjqQBgWrmARdG+lmpmHDBXrjL3WpBrM5CH8TC5hymVM2KSUfkSupPhQTV7FW7oD6hj8WfwzFPBit3J0rulW8i2tRufiZflAoeF1zbn6QoP/q+eiyF3TvGw2Tb4tRj45C36koPhqM1zx/zDSWVW+jWNWbdCfvH9q2teBvsvN/oef3V3txaPIEV8oQ8JR3SJ6/Jtkhe2RABPlBTshfchp8D34HJ8GfuTRoLGoek0sRnP0DAYoHOQ=</latexit><latexit sha1_base64="TCezhzHnvwDoFMFoQ4EK/L9JVgU=">ADOHicdVJdaxNBFJ2sXzV+NVHXwaDkEosiQj6IpRapULVCqYt7CxhdnI3GTq7s8zcLQnD/jHfPBH+OZb8c32Fzi7CaVt6oWFs+eO/fOuRPnSlrs9X41ghs3b92+s3K3e/+g4erbVH+1YXRsBAaKXNYcwtKJnBACUqOMwN8DRWcBAfvavyB8dgrNTZN5zlEKV8nMlECo6eGrZClnKc6NyxXW7GUP/FsXtflkM37dIZVamtKYFV267LClTkGBIWZcyhCk6pa0tO0lnu716zVv5HiC0bDV7m306qDLoL8AbKIveFa4ycbaVGkKFQ3Nqw38sxctygFArKJis5Fwc8TGEHmY8BRu52oWSPvPMiCba+C9DWrMXKxPrZ2lsVdW17FXcxV5XS4sMHkTOZnlBUIm5o2SQlHUtLKUjqQBgWrmARdG+lmpmHDBXrjL3WpBrM5CH8TC5hymVM2KSUfkSupPhQTV7FW7oD6hj8WfwzFPBit3J0rulW8i2tRufiZflAoeF1zbn6QoP/q+eiyF3TvGw2Tb4tRj45C36koPhqM1zx/zDSWVW+jWNWbdCfvH9q2teBvsvN/oef3V3txaPIEV8oQ8JR3SJ6/Jtkhe2RABPlBTshfchp8D34HJ8GfuTRoLGoek0sRnP0DAYoHOQ=</latexit><latexit sha1_base64="TCezhzHnvwDoFMFoQ4EK/L9JVgU=">ADOHicdVJdaxNBFJ2sXzV+NVHXwaDkEosiQj6IpRapULVCqYt7CxhdnI3GTq7s8zcLQnD/jHfPBH+OZb8c32Fzi7CaVt6oWFs+eO/fOuRPnSlrs9X41ghs3b92+s3K3e/+g4erbVH+1YXRsBAaKXNYcwtKJnBACUqOMwN8DRWcBAfvavyB8dgrNTZN5zlEKV8nMlECo6eGrZClnKc6NyxXW7GUP/FsXtflkM37dIZVamtKYFV267LClTkGBIWZcyhCk6pa0tO0lnu716zVv5HiC0bDV7m306qDLoL8AbKIveFa4ycbaVGkKFQ3Nqw38sxctygFArKJis5Fwc8TGEHmY8BRu52oWSPvPMiCba+C9DWrMXKxPrZ2lsVdW17FXcxV5XS4sMHkTOZnlBUIm5o2SQlHUtLKUjqQBgWrmARdG+lmpmHDBXrjL3WpBrM5CH8TC5hymVM2KSUfkSupPhQTV7FW7oD6hj8WfwzFPBit3J0rulW8i2tRufiZflAoeF1zbn6QoP/q+eiyF3TvGw2Tb4tRj45C36koPhqM1zx/zDSWVW+jWNWbdCfvH9q2teBvsvN/oef3V3txaPIEV8oQ8JR3SJ6/Jtkhe2RABPlBTshfchp8D34HJ8GfuTRoLGoek0sRnP0DAYoHOQ=</latexit>

How do these two notions of generalization compare?

slide-7
SLIDE 7

State Of The Art in -Robustness

Robust optimization as in [Madry, Makelov, Schmidt, Tsipras, Vladu, 2017]:

`∞

slide-8
SLIDE 8

State Of The Art in -Robustness

Robust optimization as in [Madry, Makelov, Schmidt, Tsipras, Vladu, 2017]:

`∞

slide-9
SLIDE 9

State Of The Art in -Robustness

Optimization succeeds in both cases, but the model overfits on CIFAR-10. Robust optimization as in [Madry, Makelov, Schmidt, Tsipras, Vladu, 2017]:

`∞

slide-10
SLIDE 10

Robust Generalization

Main question: Does robust generalization require more data?

slide-11
SLIDE 11

Robust Generalization

Main question: Does robust generalization require more data? Theorem (informal): There is a natural distribution over points in Rd with the following property: Learning an -robust classifier for this distribution requires times more samples than learning a non-robust classifier.

`∞

√ d

<latexit sha1_base64="dYSrsEgEkD0OTfdt+p+zx36Ev/c=">AC9nicdVJda9RAFJ2NXzV+bfVJfBlcBJFaEhHaF6FUkQp+VHDbwiYsN5O72aGTSZy5KV1C8J/4VnxT/4Q/wn/jJLuUtlsvDBzOPXfOnXsnKZW0FAR/e96Vq9eu31i56d+6fefuvf7q/T1bVEbgUBSqMAcJWFRS45AkKTwoDUKeKNxPDl+3+f0jNFYW+gvNSoxzyLScSAHkqH/YUR4TN09tcG0qSP71VCdNs24PwjWgy74MgXYMAWsTte7f2J0kJUOWoSCqwdhUFJcQ2GpFDY+FlsQRxCBmOHNSQo43rzrvhTxyT8klh3NHEO/ZsRQ25tbM8cocaGov5lrystyoslmXEtdVoRazI0mleJU8HYgPJUGBamZAyCMdL1yMQUDgtzYzrm0jdkShXuJRcpB6pYZ+ZzdwRKirdt5284juojtDdBR+xwufvZTaluWatlW8XKj0VL8uHigx0NafqMwb/V89FcX2JeP70Rt0azH4wY3oU4kGqDP6ghMlkvduDVl0VqL3OLDi2teBnsv1kOHP78cbG0vsAKe8Qes6csZBtsi+2wXTZkgn1jJ+wn+Ude9+9E+/HXOr1FjUP2Lnwfv8DvI3uZQ=</latexit><latexit sha1_base64="dYSrsEgEkD0OTfdt+p+zx36Ev/c=">AC9nicdVJda9RAFJ2NXzV+bfVJfBlcBJFaEhHaF6FUkQp+VHDbwiYsN5O72aGTSZy5KV1C8J/4VnxT/4Q/wn/jJLuUtlsvDBzOPXfOnXsnKZW0FAR/e96Vq9eu31i56d+6fefuvf7q/T1bVEbgUBSqMAcJWFRS45AkKTwoDUKeKNxPDl+3+f0jNFYW+gvNSoxzyLScSAHkqH/YUR4TN09tcG0qSP71VCdNs24PwjWgy74MgXYMAWsTte7f2J0kJUOWoSCqwdhUFJcQ2GpFDY+FlsQRxCBmOHNSQo43rzrvhTxyT8klh3NHEO/ZsRQ25tbM8cocaGov5lrystyoslmXEtdVoRazI0mleJU8HYgPJUGBamZAyCMdL1yMQUDgtzYzrm0jdkShXuJRcpB6pYZ+ZzdwRKirdt5284juojtDdBR+xwufvZTaluWatlW8XKj0VL8uHigx0NafqMwb/V89FcX2JeP70Rt0azH4wY3oU4kGqDP6ghMlkvduDVl0VqL3OLDi2teBnsv1kOHP78cbG0vsAKe8Qes6csZBtsi+2wXTZkgn1jJ+wn+Ude9+9E+/HXOr1FjUP2Lnwfv8DvI3uZQ=</latexit><latexit sha1_base64="dYSrsEgEkD0OTfdt+p+zx36Ev/c=">AC9nicdVJda9RAFJ2NXzV+bfVJfBlcBJFaEhHaF6FUkQp+VHDbwiYsN5O72aGTSZy5KV1C8J/4VnxT/4Q/wn/jJLuUtlsvDBzOPXfOnXsnKZW0FAR/e96Vq9eu31i56d+6fefuvf7q/T1bVEbgUBSqMAcJWFRS45AkKTwoDUKeKNxPDl+3+f0jNFYW+gvNSoxzyLScSAHkqH/YUR4TN09tcG0qSP71VCdNs24PwjWgy74MgXYMAWsTte7f2J0kJUOWoSCqwdhUFJcQ2GpFDY+FlsQRxCBmOHNSQo43rzrvhTxyT8klh3NHEO/ZsRQ25tbM8cocaGov5lrystyoslmXEtdVoRazI0mleJU8HYgPJUGBamZAyCMdL1yMQUDgtzYzrm0jdkShXuJRcpB6pYZ+ZzdwRKirdt5284juojtDdBR+xwufvZTaluWatlW8XKj0VL8uHigx0NafqMwb/V89FcX2JeP70Rt0azH4wY3oU4kGqDP6ghMlkvduDVl0VqL3OLDi2teBnsv1kOHP78cbG0vsAKe8Qes6csZBtsi+2wXTZkgn1jJ+wn+Ude9+9E+/HXOr1FjUP2Lnwfv8DvI3uZQ=</latexit><latexit sha1_base64="dYSrsEgEkD0OTfdt+p+zx36Ev/c=">AC9nicdVJda9RAFJ2NXzV+bfVJfBlcBJFaEhHaF6FUkQp+VHDbwiYsN5O72aGTSZy5KV1C8J/4VnxT/4Q/wn/jJLuUtlsvDBzOPXfOnXsnKZW0FAR/e96Vq9eu31i56d+6fefuvf7q/T1bVEbgUBSqMAcJWFRS45AkKTwoDUKeKNxPDl+3+f0jNFYW+gvNSoxzyLScSAHkqH/YUR4TN09tcG0qSP71VCdNs24PwjWgy74MgXYMAWsTte7f2J0kJUOWoSCqwdhUFJcQ2GpFDY+FlsQRxCBmOHNSQo43rzrvhTxyT8klh3NHEO/ZsRQ25tbM8cocaGov5lrystyoslmXEtdVoRazI0mleJU8HYgPJUGBamZAyCMdL1yMQUDgtzYzrm0jdkShXuJRcpB6pYZ+ZzdwRKirdt5284juojtDdBR+xwufvZTaluWatlW8XKj0VL8uHigx0NafqMwb/V89FcX2JeP70Rt0azH4wY3oU4kGqDP6ghMlkvduDVl0VqL3OLDi2teBnsv1kOHP78cbG0vsAKe8Qes6csZBtsi+2wXTZkgn1jJ+wn+Ude9+9E+/HXOr1FjUP2Lnwfv8DvI3uZQ=</latexit>
slide-12
SLIDE 12

Further results

  • An alternative data model for MNIST
  • Experiments on MNIST, CIFAR-10, SVHN

Conclusions

slide-13
SLIDE 13

Poster #31

Further results

  • An alternative data model for MNIST
  • Experiments on MNIST, CIFAR-10, SVHN

Conclusions

Main takeaways

  • Sample complexity can be an obstacle for adv. robustness
  • Need to improve priors encoded in models?
  • Many phenomena not yet understood theoretically

gradient-science.org