self supervised learning in computer vision
play

Self-supervised learning in computer vision Ishan Misra Facebook AI - PowerPoint PPT Presentation

Mar 14, 2024 •1.01k likes •2.09k views

Self-supervised learning in computer vision Ishan Misra Facebook AI Research With slides from Andrew Zisserman, Carl Doersch Success story of supervision: Pre-training Features from networks pre-trained on ImageNet can be used for a variety

  1. <latexit sha1_base64="zG0+86ACKs0ZivmNXL/57ldlvG0=">AB83icbVDLSsNAFL2pr1pfVZduBlvBVUmr+NgV3eiugn1AE8pkOmHTiZhZiKU0N9w40IRt/6MO/GSRpErQcGDufcyz1zvIgzpW370yosLa+srhXSxubW9s75d29jgpjSWibhDyUPQ8rypmgbc0p71IUhx4nHa9yXqdx+oVCwU93oaUTfAI8F8RrA2klN1AqzHnp/czqDcsWu2RnQIqnpAI5WoPyhzMSRxQoQnHSvXrdqTdBEvNCKezkhMrGmEywSPaN1TgCo3yTLP0JFRhsgPpXlCo0z9uZHgQKlp4JnJNKL6Xif14/1v6FmzARxZoKMj/kxzpEKUFoCGTlGg+NQTyUxWRMZYqJNTaWshMsUZ9fXiSdRq1+Uju9a1SaV3kdRTiAQziGOpxDE26gBW0gEMEjPMOLFVtP1qv1Nh8tWPnOPvyC9f4FiwCRfw=</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> <latexit sha1_base64="zG0+86ACKs0ZivmNXL/57ldlvG0=">AB83icbVDLSsNAFL2pr1pfVZduBlvBVUmr+NgV3eiugn1AE8pkOmHTiZhZiKU0N9w40IRt/6MO/GSRpErQcGDufcyz1zvIgzpW370yosLa+srhXSxubW9s75d29jgpjSWibhDyUPQ8rypmgbc0p71IUhx4nHa9yXqdx+oVCwU93oaUTfAI8F8RrA2klN1AqzHnp/czqDcsWu2RnQIqnpAI5WoPyhzMSRxQoQnHSvXrdqTdBEvNCKezkhMrGmEywSPaN1TgCo3yTLP0JFRhsgPpXlCo0z9uZHgQKlp4JnJNKL6Xif14/1v6FmzARxZoKMj/kxzpEKUFoCGTlGg+NQTyUxWRMZYqJNTaWshMsUZ9fXiSdRq1+Uju9a1SaV3kdRTiAQziGOpxDE26gBW0gEMEjPMOLFVtP1qv1Nh8tWPnOPvyC9f4FiwCRfw=</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> <latexit sha1_base64="zG0+86ACKs0ZivmNXL/57ldlvG0=">AB83icbVDLSsNAFL2pr1pfVZduBlvBVUmr+NgV3eiugn1AE8pkOmHTiZhZiKU0N9w40IRt/6MO/GSRpErQcGDufcyz1zvIgzpW370yosLa+srhXSxubW9s75d29jgpjSWibhDyUPQ8rypmgbc0p71IUhx4nHa9yXqdx+oVCwU93oaUTfAI8F8RrA2klN1AqzHnp/czqDcsWu2RnQIqnpAI5WoPyhzMSRxQoQnHSvXrdqTdBEvNCKezkhMrGmEywSPaN1TgCo3yTLP0JFRhsgPpXlCo0z9uZHgQKlp4JnJNKL6Xif14/1v6FmzARxZoKMj/kxzpEKUFoCGTlGg+NQTyUxWRMZYqJNTaWshMsUZ9fXiSdRq1+Uju9a1SaV3kdRTiAQziGOpxDE26gBW0gEMEjPMOLFVtP1qv1Nh8tWPnOPvyC9f4FiwCRfw=</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> <latexit sha1_base64="zG0+86ACKs0ZivmNXL/57ldlvG0=">AB83icbVDLSsNAFL2pr1pfVZduBlvBVUmr+NgV3eiugn1AE8pkOmHTiZhZiKU0N9w40IRt/6MO/GSRpErQcGDufcyz1zvIgzpW370yosLa+srhXSxubW9s75d29jgpjSWibhDyUPQ8rypmgbc0p71IUhx4nHa9yXqdx+oVCwU93oaUTfAI8F8RrA2klN1AqzHnp/czqDcsWu2RnQIqnpAI5WoPyhzMSRxQoQnHSvXrdqTdBEvNCKezkhMrGmEywSPaN1TgCo3yTLP0JFRhsgPpXlCo0z9uZHgQKlp4JnJNKL6Xif14/1v6FmzARxZoKMj/kxzpEKUFoCGTlGg+NQTyUxWRMZYqJNTaWshMsUZ9fXiSdRq1+Uju9a1SaV3kdRTiAQziGOpxDE26gBW0gEMEjPMOLFVtP1qv1Nh8tWPnOPvyC9f4FiwCRfw=</latexit> Pretext tasks Generative Contrastive/Clustering Related AutoEncoder, Unrelated VAE, GAN, BiGAN Pretext Image Standar Pretext Invariant Transform Representation Learning I t I I ConvNet ConvNet ConvNet Transform t I t Representation Representation Pr Encourage to be similar Predict more information 46

  2. Scaling self-supervised learning Jigsaw puzzles (Noorozi & Favaro, 2016) Goyal et al., 2019, Scaling and benchmarking self-supervised visual representation learning 47

  3. Evaluating the representation Extract "fixed" features ConvNet 48

  4. Evaluating the representation • Train a Linear SVM on fixed feature representations • Use the VOC07 image classification task 49

  5. Increasing amount of information predicted Linear classifier on VOC07 mAP = mean Average Precision 50 (Higher is better)

  6. Surface Normal Estimation • Predict surface normals on NYU-v2 • Same optimization parameters for all methods (including supervised) • PSPNet Architecture Train last few layers only (res5 onwards) • Input Output 51 Image from the NYU dataset

  7. Surface Normal Estimation % correct within Median Error Initialization 11.25 0 (Lower better) (higher better) 17.1 36.1 ImageNet Supervised 13.1 44.6 Jigsaw Flickr 100M Outperforms ImageNet supervised 52

  8. What is missing from "pretext" tasks? Or in general "proxy" tasks

  9. Pretext tasks Rotation Jigsaw puzzles (Gidaris et al., 2018) (Noroozi et al., 2016) 54

  10. The hope of generalization • We really hope that the pre-training task and the transfer task are "aligned" Pre-training Transfer Tasks Self-supervised

  11. The hope of generalization • We really hope that the pre-training task and the transfer task are "aligned" #sun #nofilter #fun #tree #aruba Transfer Tasks Pre-training Weak or self-supervised Why should solving Jigsaw puzzles teach about "semantics"? Why should performing a non semantic task produce good features?

  12. The hope of generalization ... ? Linear classifiers on "fixed" features Jigsaw ConvNet Pre-train data ConvNet Transfer Pre-training Weak or self-supervised

  13. Higher layers do not generalize ... Linear classifier on VOC07 Jigsaw res5 conv1 mAP = mean Average Precision (Higher is better)

  14. Pretext-Invariant Representation Learning (PIRL) Ishan Misra, Laurens van der Maaten

  15. <latexit sha1_base64="Xy1u5b/DFE3jDC5HvZeUuaZkrIE=">ACBXicbVA7T8MwGHTKq5RXgBEGixaJqUoqJGCrYIGtSPQhNaFyXKe16jiR7SBVURYW/goLAwix8h/Y+Dc4aQZoOcnS6e57+byIUaks69soLS2vrK6V1ysbm1vbO+buXkeGscCkjUMWip6HJGUk7aipFeJAgKPEa63uQq87sPREga8js1jYgboBGnPsVIaWlgHtacAKmx5yc36X3i5AMTj8UkVWltYFatupUDLhK7IFVQoDUwv5xhiOAcIUZkrJvW5FyEyQUxYykFSeWJEJ4gkakrylHAZFuki9N4bFWhtAPhX5cwVz93ZGgQMp4OnK7GQ572Xif14/Vv65m1AexYpwPFvkxwyqEGaRwCEVBCs21QRhQfWtEI+RQFjp4Co6BHv+y4uk06jbp/WL20a1eVnEUQYH4AicABucgSa4Bi3QBhg8gmfwCt6MJ+PFeDc+ZqUlo+jZB39gfP4AL8+ZCQ=</latexit> <latexit sha1_base64="UMBKcX4627dw7ura0nNTPruZI8=">AB83icbVDLSsNAFL2pr1pfUZduBlvBVUmKoO6KbnRXwT6gCWUynbRDJ5MwMxFK6G+4caGIW3/GnX/jpM1CWw8MHM65l3vmBAlnSjvOt1VaW9/Y3CpvV3Z29/YP7MOjopTSWibxDyWvQArypmgbc0p71EUhwFnHaDyW3ud5+oVCwWj3qaUD/CI8FCRrA2klfzIqzHQZjdz2oDu+rUnTnQKnELUoUCrYH95Q1jkZUaMKxUn3XSbSfYakZ4XRW8VJFE0wmeET7hgocUeVn8wzdGaUIQpjaZ7QaK7+3shwpNQ0CsxkHlEte7n4n9dPdXjlZ0wkqaCLA6FKUc6RnkBaMgkJZpPDcFEMpMVkTGWmGhTU8WU4C5/eZV0GnX3on790Kg2b4o6ynACp3AOLlxCE+6gBW0gkMAzvMKblVov1rv1sRgtWcXOMfyB9fkDdcaRUw=</latexit> <latexit sha1_base64="MXdteYo7j3dYQsNtpapn5lT1fY=">AB9XicbVDLSgNBEOz1GeMr6tHLYCJ4CrtBUG9BL3qLYB6QbMLsZDYZMvtgplcJS/7DiwdFvPov3vwbZ5McNLFgoKjqpmvKi6XQaNvf1srq2vrGZm4rv72zu7dfODhs6ChRjNdZJCPV8qjmUoS8jgIlb8WK08CTvOmNbjK/+ciVFlH4gOYuwEdhMIXjKRuqVOQHo+endpIulXqFol+0pyDJx5qQIc9R6ha9OP2JwENkmrduwY3ZQqFEzySb6TaB5TNqID3jY0pAHXbjpNPSGnRukTP1LmhUim6u+NlAZajwPTGYh9aKXif957QT9SzcVYZwgD9nskJ9IghHJKiB9oThDOTaEMiVMVsKGVFGpqi8KcFZ/PIyaVTKzn56r5SrF7P68jBMZzAGThwAVW4hRrUgYGCZ3iFN+vJerHerY/Z6Io13zmCP7A+fwAHuJI5</latexit> <latexit sha1_base64="MXdteYo7j3dYQsNtpapn5lT1fY=">AB9XicbVDLSgNBEOz1GeMr6tHLYCJ4CrtBUG9BL3qLYB6QbMLsZDYZMvtgplcJS/7DiwdFvPov3vwbZ5McNLFgoKjqpmvKi6XQaNvf1srq2vrGZm4rv72zu7dfODhs6ChRjNdZJCPV8qjmUoS8jgIlb8WK08CTvOmNbjK/+ciVFlH4gOYuwEdhMIXjKRuqVOQHo+endpIulXqFol+0pyDJx5qQIc9R6ha9OP2JwENkmrduwY3ZQqFEzySb6TaB5TNqID3jY0pAHXbjpNPSGnRukTP1LmhUim6u+NlAZajwPTGYh9aKXif957QT9SzcVYZwgD9nskJ9IghHJKiB9oThDOTaEMiVMVsKGVFGpqi8KcFZ/PIyaVTKzn56r5SrF7P68jBMZzAGThwAVW4hRrUgYGCZ3iFN+vJerHerY/Z6Io13zmCP7A+fwAHuJI5</latexit> Less Contain information Predict property Semantic about transform t of transform t Features Pretext task ConvNet I t I t I Image I t Transform t 60

  16. <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> <latexit sha1_base64="zG0+86ACKs0ZivmNXL/57ldlvG0=">AB83icbVDLSsNAFL2pr1pfVZduBlvBVUmr+NgV3eiugn1AE8pkOmHTiZhZiKU0N9w40IRt/6MO/GSRpErQcGDufcyz1zvIgzpW370yosLa+srhXSxubW9s75d29jgpjSWibhDyUPQ8rypmgbc0p71IUhx4nHa9yXqdx+oVCwU93oaUTfAI8F8RrA2klN1AqzHnp/czqDcsWu2RnQIqnpAI5WoPyhzMSRxQoQnHSvXrdqTdBEvNCKezkhMrGmEywSPaN1TgCo3yTLP0JFRhsgPpXlCo0z9uZHgQKlp4JnJNKL6Xif14/1v6FmzARxZoKMj/kxzpEKUFoCGTlGg+NQTyUxWRMZYqJNTaWshMsUZ9fXiSdRq1+Uju9a1SaV3kdRTiAQziGOpxDE26gBW0gEMEjPMOLFVtP1qv1Nh8tWPnOPvyC9f4FiwCRfw=</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> <latexit sha1_base64="zG0+86ACKs0ZivmNXL/57ldlvG0=">AB83icbVDLSsNAFL2pr1pfVZduBlvBVUmr+NgV3eiugn1AE8pkOmHTiZhZiKU0N9w40IRt/6MO/GSRpErQcGDufcyz1zvIgzpW370yosLa+srhXSxubW9s75d29jgpjSWibhDyUPQ8rypmgbc0p71IUhx4nHa9yXqdx+oVCwU93oaUTfAI8F8RrA2klN1AqzHnp/czqDcsWu2RnQIqnpAI5WoPyhzMSRxQoQnHSvXrdqTdBEvNCKezkhMrGmEywSPaN1TgCo3yTLP0JFRhsgPpXlCo0z9uZHgQKlp4JnJNKL6Xif14/1v6FmzARxZoKMj/kxzpEKUFoCGTlGg+NQTyUxWRMZYqJNTaWshMsUZ9fXiSdRq1+Uju9a1SaV3kdRTiAQziGOpxDE26gBW0gEMEjPMOLFVtP1qv1Nh8tWPnOPvyC9f4FiwCRfw=</latexit> Underlying Principle for Pretext Tasks • Apply known image transform t Pretext Image Standard Pretext Transform Learning • Construct task to predict t from I t transformed Image ( I t ) • Final layer representations must I carry information about t ConvNet Transform t • Representations "covary" with t I t Representation Predict property of t 61

  17. How important has invariance been? • Hand-crafted features like SIFT and HOG SIFT - Scale Invariant Feature Transform • • Supervised systems are trained to be invariant to "data augmentation" 62

  18. <latexit sha1_base64="zG0+86ACKs0ZivmNXL/57ldlvG0=">AB83icbVDLSsNAFL2pr1pfVZduBlvBVUmr+NgV3eiugn1AE8pkOmHTiZhZiKU0N9w40IRt/6MO/GSRpErQcGDufcyz1zvIgzpW370yosLa+srhXSxubW9s75d29jgpjSWibhDyUPQ8rypmgbc0p71IUhx4nHa9yXqdx+oVCwU93oaUTfAI8F8RrA2klN1AqzHnp/czqDcsWu2RnQIqnpAI5WoPyhzMSRxQoQnHSvXrdqTdBEvNCKezkhMrGmEywSPaN1TgCo3yTLP0JFRhsgPpXlCo0z9uZHgQKlp4JnJNKL6Xif14/1v6FmzARxZoKMj/kxzpEKUFoCGTlGg+NQTyUxWRMZYqJNTaWshMsUZ9fXiSdRq1+Uju9a1SaV3kdRTiAQziGOpxDE26gBW0gEMEjPMOLFVtP1qv1Nh8tWPnOPvyC9f4FiwCRfw=</latexit> <latexit sha1_base64="zG0+86ACKs0ZivmNXL/57ldlvG0=">AB83icbVDLSsNAFL2pr1pfVZduBlvBVUmr+NgV3eiugn1AE8pkOmHTiZhZiKU0N9w40IRt/6MO/GSRpErQcGDufcyz1zvIgzpW370yosLa+srhXSxubW9s75d29jgpjSWibhDyUPQ8rypmgbc0p71IUhx4nHa9yXqdx+oVCwU93oaUTfAI8F8RrA2klN1AqzHnp/czqDcsWu2RnQIqnpAI5WoPyhzMSRxQoQnHSvXrdqTdBEvNCKezkhMrGmEywSPaN1TgCo3yTLP0JFRhsgPpXlCo0z9uZHgQKlp4JnJNKL6Xif14/1v6FmzARxZoKMj/kxzpEKUFoCGTlGg+NQTyUxWRMZYqJNTaWshMsUZ9fXiSdRq1+Uju9a1SaV3kdRTiAQziGOpxDE26gBW0gEMEjPMOLFVtP1qv1Nh8tWPnOPvyC9f4FiwCRfw=</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> Pretext-Invariant Representation Learning (PIRL) • Be invariant to t Pretext Image Standard Pretext Pretext Invariant Transform Learning Representation Learning I t I t I I ConvNet ConvNet ConvNet ConvNet Transform t I t Representation Representation Representation Predict property of t Encourage to be similar 63

  19. <latexit sha1_base64="zG0+86ACKs0ZivmNXL/57ldlvG0=">AB83icbVDLSsNAFL2pr1pfVZduBlvBVUmr+NgV3eiugn1AE8pkOmHTiZhZiKU0N9w40IRt/6MO/GSRpErQcGDufcyz1zvIgzpW370yosLa+srhXSxubW9s75d29jgpjSWibhDyUPQ8rypmgbc0p71IUhx4nHa9yXqdx+oVCwU93oaUTfAI8F8RrA2klN1AqzHnp/czqDcsWu2RnQIqnpAI5WoPyhzMSRxQoQnHSvXrdqTdBEvNCKezkhMrGmEywSPaN1TgCo3yTLP0JFRhsgPpXlCo0z9uZHgQKlp4JnJNKL6Xif14/1v6FmzARxZoKMj/kxzpEKUFoCGTlGg+NQTyUxWRMZYqJNTaWshMsUZ9fXiSdRq1+Uju9a1SaV3kdRTiAQziGOpxDE26gBW0gEMEjPMOLFVtP1qv1Nh8tWPnOPvyC9f4FiwCRfw=</latexit> <latexit sha1_base64="zG0+86ACKs0ZivmNXL/57ldlvG0=">AB83icbVDLSsNAFL2pr1pfVZduBlvBVUmr+NgV3eiugn1AE8pkOmHTiZhZiKU0N9w40IRt/6MO/GSRpErQcGDufcyz1zvIgzpW370yosLa+srhXSxubW9s75d29jgpjSWibhDyUPQ8rypmgbc0p71IUhx4nHa9yXqdx+oVCwU93oaUTfAI8F8RrA2klN1AqzHnp/czqDcsWu2RnQIqnpAI5WoPyhzMSRxQoQnHSvXrdqTdBEvNCKezkhMrGmEywSPaN1TgCo3yTLP0JFRhsgPpXlCo0z9uZHgQKlp4JnJNKL6Xif14/1v6FmzARxZoKMj/kxzpEKUFoCGTlGg+NQTyUxWRMZYqJNTaWshMsUZ9fXiSdRq1+Uju9a1SaV3kdRTiAQziGOpxDE26gBW0gEMEjPMOLFVtP1qv1Nh8tWPnOPvyC9f4FiwCRfw=</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> Pretext-Invariant Representation Learning (PIRL) • Be invariant to t Pretext Image Standard Pretext Pretext Invariant Transform Learning Representation Learning • Representation I t I t contains no I information about t I ConvNet ConvNet ConvNet ConvNet Transform t I t Representation Representation Representation Predict property of t Encourage to be similar 64

  20. <latexit sha1_base64="A/7PWM+VfPfU7Nqhfek7uRT9E=">ACOXicbVDLSgMxFM34rPVdekm2AoKUmYU0aXoRsFBfuAtpZMmqmhmcyQ3CmUIb/lxr9wJ7hxoYhbf8C0nYWPHgczjmX3Hv8WHANrvszMzOzS8s5pbyura+uFjc2ajhJFWZVGIlINn2gmuGRV4CBYI1aMhL5gdb9/MfLrA6Y0j+QtDGPWDklP8oBTAlbqFCql607aCgncqzClkQRFNPABM2ZvrPpBOjBZwvIrYw7wdOMuBWP2S51C0S27Y+D/xMtIEWodApPrW5Ek5BJoIJo3fTcGNopUcCpYCbfSjSLCe2THmtaKknIdDsdX27wrlW6OIiUfRLwWP05kZJQ62Ho2+RoUf3XG4nTvGYCwWk75TJOgEk6+ShIBIYIj2rEXa4YBTG0hFDF7a6Y3hNFKNiy87YE7+/J/0ntsOwdlY9vDotn51kdObSNdtAe8tAJOkOXqIKqiKIH9ILe0Lvz6Lw6H87nJDrjZDNb6Becr2/jF6/1</latexit> <latexit sha1_base64="zG0+86ACKs0ZivmNXL/57ldlvG0=">AB83icbVDLSsNAFL2pr1pfVZduBlvBVUmr+NgV3eiugn1AE8pkOmHTiZhZiKU0N9w40IRt/6MO/GSRpErQcGDufcyz1zvIgzpW370yosLa+srhXSxubW9s75d29jgpjSWibhDyUPQ8rypmgbc0p71IUhx4nHa9yXqdx+oVCwU93oaUTfAI8F8RrA2klN1AqzHnp/czqDcsWu2RnQIqnpAI5WoPyhzMSRxQoQnHSvXrdqTdBEvNCKezkhMrGmEywSPaN1TgCo3yTLP0JFRhsgPpXlCo0z9uZHgQKlp4JnJNKL6Xif14/1v6FmzARxZoKMj/kxzpEKUFoCGTlGg+NQTyUxWRMZYqJNTaWshMsUZ9fXiSdRq1+Uju9a1SaV3kdRTiAQziGOpxDE26gBW0gEMEjPMOLFVtP1qv1Nh8tWPnOPvyC9f4FiwCRfw=</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> <latexit sha1_base64="zG0+86ACKs0ZivmNXL/57ldlvG0=">AB83icbVDLSsNAFL2pr1pfVZduBlvBVUmr+NgV3eiugn1AE8pkOmHTiZhZiKU0N9w40IRt/6MO/GSRpErQcGDufcyz1zvIgzpW370yosLa+srhXSxubW9s75d29jgpjSWibhDyUPQ8rypmgbc0p71IUhx4nHa9yXqdx+oVCwU93oaUTfAI8F8RrA2klN1AqzHnp/czqDcsWu2RnQIqnpAI5WoPyhzMSRxQoQnHSvXrdqTdBEvNCKezkhMrGmEywSPaN1TgCo3yTLP0JFRhsgPpXlCo0z9uZHgQKlp4JnJNKL6Xif14/1v6FmzARxZoKMj/kxzpEKUFoCGTlGg+NQTyUxWRMZYqJNTaWshMsUZ9fXiSdRq1+Uju9a1SaV3kdRTiAQziGOpxDE26gBW0gEMEjPMOLFVtP1qv1Nh8tWPnOPvyC9f4FiwCRfw=</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> <latexit sha1_base64="zG0+86ACKs0ZivmNXL/57ldlvG0=">AB83icbVDLSsNAFL2pr1pfVZduBlvBVUmr+NgV3eiugn1AE8pkOmHTiZhZiKU0N9w40IRt/6MO/GSRpErQcGDufcyz1zvIgzpW370yosLa+srhXSxubW9s75d29jgpjSWibhDyUPQ8rypmgbc0p71IUhx4nHa9yXqdx+oVCwU93oaUTfAI8F8RrA2klN1AqzHnp/czqDcsWu2RnQIqnpAI5WoPyhzMSRxQoQnHSvXrdqTdBEvNCKezkhMrGmEywSPaN1TgCo3yTLP0JFRhsgPpXlCo0z9uZHgQKlp4JnJNKL6Xif14/1v6FmzARxZoKMj/kxzpEKUFoCGTlGg+NQTyUxWRMZYqJNTaWshMsUZ9fXiSdRq1+Uju9a1SaV3kdRTiAQziGOpxDE26gBW0gEMEjPMOLFVtP1qv1Nh8tWPnOPvyC9f4FiwCRfw=</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> <latexit sha1_base64="zG0+86ACKs0ZivmNXL/57ldlvG0=">AB83icbVDLSsNAFL2pr1pfVZduBlvBVUmr+NgV3eiugn1AE8pkOmHTiZhZiKU0N9w40IRt/6MO/GSRpErQcGDufcyz1zvIgzpW370yosLa+srhXSxubW9s75d29jgpjSWibhDyUPQ8rypmgbc0p71IUhx4nHa9yXqdx+oVCwU93oaUTfAI8F8RrA2klN1AqzHnp/czqDcsWu2RnQIqnpAI5WoPyhzMSRxQoQnHSvXrdqTdBEvNCKezkhMrGmEywSPaN1TgCo3yTLP0JFRhsgPpXlCo0z9uZHgQKlp4JnJNKL6Xif14/1v6FmzARxZoKMj/kxzpEKUFoCGTlGg+NQTyUxWRMZYqJNTaWshMsUZ9fXiSdRq1+Uju9a1SaV3kdRTiAQziGOpxDE26gBW0gEMEjPMOLFVtP1qv1Nh8tWPnOPvyC9f4FiwCRfw=</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> <latexit sha1_base64="b9Db75leVutsWGyjrhn+yRcISj8=">AB9XicbVDLSgMxFM34rPVdekm2AquykwVH7uiG91VsA9opyWTZtrQTGZI7ihl6H+4caGIW/FnX9jZlpErQcCh3Pu5Z4cLxJcg21/WguLS8srq7m1/PrG5tZ2YWe3ocNYUVanoQhVyOaCS5ZHTgI1oUI4EnWNMbXaV+854pzUN5B+OIuQEZSO5zSsBI3VInID0/ORm0oVSr1C0y3YGPE+cGSmiGWq9wkenH9I4YBKoIFq3HTsCNyEKOBVsku/EmkWEjsiAtQ2VJGDaTbLUE3xolD72Q2WeBJypPzcSEmg9DjwzmYbUf71U/M9rx+CfuwmXUQxM0ukhPxYQpxWgPtcMQpibAihipusmA6JIhRMUfmshIsUp9fnieNStk5Lp/cVorVy1kdObSPDtARctAZqJrVEN1RJFCj+gZvVgP1pP1ar1NRxes2c4e+gXr/Qsc8pJl</latexit> PIRL • Representations from I and I t Pretext Image Pretext Invariant Transform should be similar Representation Learning • t = Pretext Transforms I t I (Jigsaw/ Rotation, combinations etc.) I ConvNet ConvNet ConvNet Transform t • Use a contrastive loss to I t enforce similarity of features L contrastive ( v I , v I t ) Representation Representation Encourage to be similar Pr 65

  21. Contrastive Learning Groups of Related and Unrelated Images

  22. Contrastive Learning Groups of Shared network Image Features Related and Unrelated (Siamese Net) (Embeddings) Images

  23. Contrastive Learning Related and Shared Loss Function Image Unrelated network Embeddings from related images should be Features Images (Siamese closer than embeddings from unrelated images (Embeddings) Net) d( ) < d( ) d( ) < d( ) Hadsell et al., 2005, DrLim

  24. Contrastive Loss Function Loss Function Embeddings from related images should be closer than embeddings from unrelated images d( ) < d( ) d( ) < d( ) Positive Negative (Related) (Unrelated) Good negatives are very important in contrastive learning Hadsell et al., 2005, DrLim

  25. Contrastive learning -- what does it do? Negative samples Positive Sample Negative samples

  26. How does this relate to "pretext" tasks?

  27. <latexit sha1_base64="1XIy5Ptm3yjnfAU4ZvSDtu2g1M8=">ACKXicbVDLSsNAFJ34rPEVdekm2BbqpiQV0WXRje4q2Ae0MUymk3boZBJmJoUS8jtu/BU3Coq69UectAFr64GBM+fcy73eBElQlrWp7ayura+sVnY0rd3dvf2jYPDlghjnAThTkHQ8KTAnDTUkxZ2IYxh4FLe90Xmt8eYCxKyezmJsBPASM+QVAqyTXq5dKg0gugHp+cps+yNOSXi75c5IS5v7j1E1+PW6RtGqWlOYy8TOSRHkaLjGa68fojATCIKhejaViSdBHJEMWp3osFjiAawQHuKspgIWTC9NzbJS+qYfcvWYNKfqfEcCAyEmgacqsyXFopeJ/3ndWPqXTkJYFEvM0GyQH1NThmYWm9knHCNJ4pAxIna1URDyCGSKlxdhWAvnrxMWrWqfVY9v6sV61d5HAVwDE5ABdjgAtTBDWiAJkDgETyDN/CuPWkv2of2NStd0fKeI/AH2vcPpm6lHg=</latexit> <latexit sha1_base64="NZU1SuG24a2ZQjMLXDCxlDXMhQ=">ADYHicjVLPb9MwFHYTfpSwsRZucIloJ4aEqiR0g92mcYED0pDWbVJTIsdxWmvOD9lOobL8T3LjwIW/BDsJpaxD4kl+en7v8+fPfi8uKeHC8753LPvO3Xv3uw+chzu7j/Z6/cXvKgYwhNU0IJdxZBjSnI8EURQfFUyDLOY4sv4+p2pXy4x46TIz8WqxLMznOSEgSFTkX9znJ/GJYLEoVigQU8CDMoFnEqP6jP4uXQ0UWTQJDKj0pvw4rjEqJrOMfTZElKnsM85n8WitRGpDgVEuptxLyEjI4r7iSbB4r6Y2CV643OjJuHCjnBvyUFV/yNfTw2MC2nuaWYabqOFvoZmK5B/RarjFeo4hXZN6hu2wdhuUNeR2xhcN5b8ZfUNWu/FYOf/NGPUGWk1t7nbgt8EAtHYW9b6FSYGqDOcCUcj51PdKMZOQCYIoVs5mZ3TY9qVpi7uvM4mbFkyvXLh1dvOEhBnqyzWSCOT36yZ5G21aSXStzNJ8rISOEfNRWlFXVG4ZtrchDCMBF3pACJGtFYXLfRMIKFnsvmEY2NH6ydvBxfByH89Gn8KBien7Xd0wTPwHBwAH7wBJ+A9OAMTgDo/LNvasXatn3bX3rP7DdTqtGegL/MfvoLchwPOg=</latexit> <latexit sha1_base64="Tn5ijQgOnAy4mPiBG3gQMhvSFH4=">ACE3icbVDLSsNAFJ3UV62vqEs3waZQXZSkio9d0Y0uhAr2AU0tk+mkHTp5MHMjlNB/cOvuHGhiFs37vwbkzSIWg8MnDnXu69xw4k2AYn0pubn5hcSm/XFhZXVvfUDe3mtIPBaEN4nNftG0sKWcebQADTtuBoNi1OW3Zo/PEb91RIZnv3cA4oF0XDzmMIhlnrqfkm3giHrWTCkgMuWi2FoO9Hl5Bb29IKe/gnm0dVE76lFo2Kk0GaJmZEiylDvqR9W3yehSz0gHEvZMY0AuhEWwAink4IVShpgMsID2omph10qu1F60QrxUpfc3wRPw+0VP3ZEWFXyrFrx5XJjvKvl4j/eZ0QnJNuxLwgBOqR6SAn5Br4WhKQ1meCEuDjmGAiWLyrRoZYAJxjIU0hNMER98nz5JmtWIeVA6vq8XaWRZHu2gXVRGJjpGNXSB6qiBCLpHj+gZvSgPypPyqrxNS3NK1rONfkF5/wJm4J1f</latexit> <latexit sha1_base64="UJ5UsvgNfPvuSUTDITK4OtV6Cvg=">ACRHicdZDLSsNAFIYn9VbjLerSTbAp1E1JKqLohvdVbAXSGOYTCft0MmFmUmhDycGx/AnU/gxoUibsVJW7C2emDg5/vP4Zz5vZgSLkzWSmsrK6tbxQ31a3tnd09bf+gxaOEIdxEY1Yx4McUxLipiC4k7MAw8itve8Cr32yPMOInCOzGOsRPAfkh8gqCQyNXstGvdAMoBp6f3mT34sRQy4Y/hxbAKHPTHzN3jX8sw9VKZtWclL4srJkogVk1XO2p24tQEuBQIAo5ty0zFk4KmSCI4kztJhzHEA1hH9tShjDA3EknIWR6WZKe7kdMvlDoEzo/kcKA83Hgyc78Rr7o5fAvz06Ef+GkJIwTgUM0XeQnVBeRnieq9wjDSNCxFBAxIm/V0QAyiITMXZUhWItfXhatWtU6rZ7d1kr1y1kcRXAEjkEFWOAc1ME1aIAmQOABvIA38K48Kq/Kh/I5bS0os5lD8KuUr2+xRbBF</latexit> <latexit sha1_base64="1XIy5Ptm3yjnfAU4ZvSDtu2g1M8=">ACKXicbVDLSsNAFJ34rPEVdekm2BbqpiQV0WXRje4q2Ae0MUymk3boZBJmJoUS8jtu/BU3Coq69UectAFr64GBM+fcy73eBElQlrWp7ayura+sVnY0rd3dvf2jYPDlghjnAThTkHQ8KTAnDTUkxZ2IYxh4FLe90Xmt8eYCxKyezmJsBPASM+QVAqyTXq5dKg0gugHp+cps+yNOSXi75c5IS5v7j1E1+PW6RtGqWlOYy8TOSRHkaLjGa68fojATCIKhejaViSdBHJEMWp3osFjiAawQHuKspgIWTC9NzbJS+qYfcvWYNKfqfEcCAyEmgacqsyXFopeJ/3ndWPqXTkJYFEvM0GyQH1NThmYWm9knHCNJ4pAxIna1URDyCGSKlxdhWAvnrxMWrWqfVY9v6sV61d5HAVwDE5ABdjgAtTBDWiAJkDgETyDN/CuPWkv2of2NStd0fKeI/AH2vcPpm6lHg=</latexit> <latexit sha1_base64="KWmfCBMJGy46BFqyZVbSAliZW7g=">ACK3icbVDLSsNAFJ34rPEVdekm2BbqpiQV0WpG91VsA9oY5hMJ+3QySTMTAol5H/c+CsudOEDt/6H0zaL2Hpg4HDOucy9x4soEdKyPrW19Y3Nre3Cjr67t39waBwdt0UYc4RbKQh73pQYEoYbkiKe5GHMPAo7jW9mfmeCuSAhe5DTCDsBHDLiEwSlklyjUS4NK/0AypHnJ3fpozwv6eWSn5OUkItMUjfJx5XtGkWras1hrhI7I0WQoekar/1BiOIAM4koFKJnW5F0EsglQRSnej8WOIJoDIe4pyiDARZOMr81NctKGZh+yNVj0pyr+YkEBkJMA08lZ2uKZW8m/uf1YulfOwlhUSwxQ4uP/JiaMjRnxZkDwjGSdKoIRJyoXU0ghwiqerVQn28smrpF2r2hfVy/tasd7I6iAU3AGKsAGV6AObkETtACT+AFvIMP7Vl7076070V0TctmTsAfaD+/XpSmBQ=</latexit> <latexit sha1_base64="ZYkFvcOHAPMcQbcKwhPq3ujeh+c=">ACG3icbVDLSsNAFJ34rPEVdelmsCnUTUkqosuiG91VsA9oY5lMJ+3QySTMTIQS+h9u/BU3LhRxJbjwb5y0WdTWAwOHc85l7j1+zKhUjvNjrKyurW9sFrbM7Z3dvX3r4LApo0Rg0sARi0TbR5IwyklDUcVIOxYEhT4jLX90nfmtRyIkjfi9GsfEC9GA04BipLTUs6ole1DuhkgN/SC9nTyoU9s2cGcpAV7PmD3rKJTcaAy8TNSRHkqPesr24/wklIuMIMSdlxnVh5KRKYkYmZjeRJEZ4hAakoylHIZFeOr1tAkta6cMgEvpxBafq/ESKQinHoa+T2ZJy0cvE/7xOoJL6U8ThThePZRkDCoIpgVBftUEKzYWBOEBdW7QjxEAmGl6zR1Ce7iycukWa24Z5Xzu2qxdpXUQDH4ASUgQsuQA3cgDpoAyewAt4A+/Gs/FqfBifs+iKkc8cgT8wvn8BRCKfGg=</latexit> <latexit sha1_base64="UJ5UsvgNfPvuSUTDITK4OtV6Cvg=">ACRHicdZDLSsNAFIYn9VbjLerSTbAp1E1JKqLohvdVbAXSGOYTCft0MmFmUmhDycGx/AnU/gxoUibsVJW7C2emDg5/vP4Zz5vZgSLkzWSmsrK6tbxQ31a3tnd09bf+gxaOEIdxEY1Yx4McUxLipiC4k7MAw8itve8Cr32yPMOInCOzGOsRPAfkh8gqCQyNXstGvdAMoBp6f3mT34sRQy4Y/hxbAKHPTHzN3jX8sw9VKZtWclL4srJkogVk1XO2p24tQEuBQIAo5ty0zFk4KmSCI4kztJhzHEA1hH9tShjDA3EknIWR6WZKe7kdMvlDoEzo/kcKA83Hgyc78Rr7o5fAvz06Ef+GkJIwTgUM0XeQnVBeRnieq9wjDSNCxFBAxIm/V0QAyiITMXZUhWItfXhatWtU6rZ7d1kr1y1kcRXAEjkEFWOAc1ME1aIAmQOABvIA38K48Kq/Kh/I5bS0os5lD8KuUr2+xRbBF</latexit> <latexit sha1_base64="CZMu9i9mqYh9pdzTQnbewbqTXR0=">AB73icbVDLSsNAFJ3UV62vqks3g63gqiRVfOyKblxWsA9oQ5lMJ+3QySTO3Agl9CfcuFDErb/jzr9xkgZR64ELh3Pu5d57vEhwDb9aRWldW14rpY3Nre2d8u5eW4exoqxFQxGqrkc0E1yFnAQrBspRgJPsI43uU79zgNTmofyDqYRcwMyktznlICRutU+jBmQ6qBcsWt2BrxInJxUI7moPzRH4Y0DpgEKojWPceOwE2IAk4Fm5X6sWYRoRMyYj1DJQmYdpPs3hk+MsoQ+6EyJQFn6s+JhARaTwPdAYExvqvl4r/eb0Y/As34TKgUk6X+THAkOI0+fxkCtGQUwNIVRxcyumY6IBRNRKQvhMsXZ98uLpF2vOSe109t6pXGVx1FEB+gQHSMHnaMGukFN1EIUCfSIntGLdW89Wa/W27y1YOUz+gXrPcvduyPug=</latexit> <latexit sha1_base64="CZMu9i9mqYh9pdzTQnbewbqTXR0=">AB73icbVDLSsNAFJ3UV62vqks3g63gqiRVfOyKblxWsA9oQ5lMJ+3QySTO3Agl9CfcuFDErb/jzr9xkgZR64ELh3Pu5d57vEhwDb9aRWldW14rpY3Nre2d8u5eW4exoqxFQxGqrkc0E1yFnAQrBspRgJPsI43uU79zgNTmofyDqYRcwMyktznlICRutU+jBmQ6qBcsWt2BrxInJxUI7moPzRH4Y0DpgEKojWPceOwE2IAk4Fm5X6sWYRoRMyYj1DJQmYdpPs3hk+MsoQ+6EyJQFn6s+JhARaTwPdAYExvqvl4r/eb0Y/As34TKgUk6X+THAkOI0+fxkCtGQUwNIVRxcyumY6IBRNRKQvhMsXZ98uLpF2vOSe109t6pXGVx1FEB+gQHSMHnaMGukFN1EIUCfSIntGLdW89Wa/W27y1YOUz+gXrPcvduyPug=</latexit> <latexit sha1_base64="Ly2C3v3mHZfg5kOK31IrZoYRcrg=">ACt3icbVFNb9QwEHXCVwkfXeDIxWK3UpHQKhuWwt6qcoEDUpHYtmKzRI7jbKw6TrAnhZXlv8iBG/8GJxvKljKSrTdv3shvxmktuIYw/OX5N27eun1n525w7/6Dh7uDR49PdNUoyua0EpU6S4lmgks2Bw6CndWKkTIV7DQ9f9vWTy+Y0rySn2Bds2VJVpLnBJwVDL4sTeK64InMRQMyH5cEijS3Ly3X+D5KHDFlqBEmA/WpXGjWU3oOVmxRXbBay1JyfTSfO+MWCfIWO6cdKkhuiaKrBptjVql1oTj6AUOxwftNY1scEV8pKpv8lL4ataKou4ObWDibdHoj8vSJuavYzuyQTIYuoYu8HUw6cEQ9XGcDH7GWUWbkmgmi9mIQ1LA1RwKlgzuPWxA72827GxXuOyXBeKXck4I7d7jCk1Hpdpk7ZutT/1lryf7VFA/mbpeGyboBJunkobwSGCrefiDOuGAWxdoBQxZ1XTAu3awruqzdLmLVxcDnydXASjScvx9OP0fDwqF/HDnqKnqF9NEGv0SF6h47RHFv6n32qJf5Mz/xc7/YSH2v73mCroT/9Tdnf9X</latexit> <latexit sha1_base64="NZU1SuG24a2ZQjMLXDCxlDXMhQ=">ADYHicjVLPb9MwFHYTfpSwsRZucIloJ4aEqiR0g92mcYED0pDWbVJTIsdxWmvOD9lOobL8T3LjwIW/BDsJpaxD4kl+en7v8+fPfi8uKeHC8753LPvO3Xv3uw+chzu7j/Z6/cXvKgYwhNU0IJdxZBjSnI8EURQfFUyDLOY4sv4+p2pXy4x46TIz8WqxLMznOSEgSFTkX9znJ/GJYLEoVigQU8CDMoFnEqP6jP4uXQ0UWTQJDKj0pvw4rjEqJrOMfTZElKnsM85n8WitRGpDgVEuptxLyEjI4r7iSbB4r6Y2CV643OjJuHCjnBvyUFV/yNfTw2MC2nuaWYabqOFvoZmK5B/RarjFeo4hXZN6hu2wdhuUNeR2xhcN5b8ZfUNWu/FYOf/NGPUGWk1t7nbgt8EAtHYW9b6FSYGqDOcCUcj51PdKMZOQCYIoVs5mZ3TY9qVpi7uvM4mbFkyvXLh1dvOEhBnqyzWSCOT36yZ5G21aSXStzNJ8rISOEfNRWlFXVG4ZtrchDCMBF3pACJGtFYXLfRMIKFnsvmEY2NH6ydvBxfByH89Gn8KBien7Xd0wTPwHBwAH7wBJ+A9OAMTgDo/LNvasXatn3bX3rP7DdTqtGegL/MfvoLchwPOg=</latexit> <latexit sha1_base64="zG0+86ACKs0ZivmNXL/57ldlvG0=">AB83icbVDLSsNAFL2pr1pfVZduBlvBVUmr+NgV3eiugn1AE8pkOmHTiZhZiKU0N9w40IRt/6MO/GSRpErQcGDufcyz1zvIgzpW370yosLa+srhXSxubW9s75d29jgpjSWibhDyUPQ8rypmgbc0p71IUhx4nHa9yXqdx+oVCwU93oaUTfAI8F8RrA2klN1AqzHnp/czqDcsWu2RnQIqnpAI5WoPyhzMSRxQoQnHSvXrdqTdBEvNCKezkhMrGmEywSPaN1TgCo3yTLP0JFRhsgPpXlCo0z9uZHgQKlp4JnJNKL6Xif14/1v6FmzARxZoKMj/kxzpEKUFoCGTlGg+NQTyUxWRMZYqJNTaWshMsUZ9fXiSdRq1+Uju9a1SaV3kdRTiAQziGOpxDE26gBW0gEMEjPMOLFVtP1qv1Nh8tWPnOPvyC9f4FiwCRfw=</latexit> <latexit sha1_base64="y2FWsphi+t+VGdcXlc76o/Lki9c=">ADHicbVLbtQwFHXCq4RHp7BkY3WmokholAltobuqbGCBVKROW2kyRI7jzFh1HMt2CiPLH8CGX2HDAoTY8gHs+BucTDpMH1fK1fG956c3DgVjCodhn89/8bNW7fvrNwN7t1/8HC1s/boSJWVxGSIS1bKkxQpwignQ01IydCElSkjBynp6/r/vEZkYqW/FDPBkXaMJpTjHSrpSsesbvVhMaRLrKdFoMy6Qnqa5eWs/6Ge9wDXrAkbMvLPuGFeKCIRP0YSMsjMqFEcFUWPzqXFiHSEjubPSHA1SAk0qZQ1cpJaE/aj5zDs79RpK7LBJfq+LD/yBXV7t6ZFTQ6dsomXWb1zo4VNzH/TtudUL4geEsQWmEt0kG5wLNozr9Z42gkmn60abgFfBoAVd0MZB0vkTZyWuCsI1Zkip0SAUemyQ1BQzYoPlLTrY7nC+QrjhKhnMS+kermFTXZ4wqFBqVqSOWdtUl3t18breqNL5q7GhXFSacDx/UV4xqEtY3wyYUmwZjMHEJbUeYV46v4f1u7+zJewW8fO4pOvgqOoP3jR3ofdf23WsgCdgHWyCAXgJ9sAbcACGAHufva/ed+H/8X/5v/0f82pvtfOPAYXwv/9D6/i9XA=</latexit> <latexit sha1_base64="8oYhATVQxn+kCv7hSq1/xFVvwEs=">ACSHicbZBLS8NAFIUn9VXrq+rSTbAp1E1JKqLohvdVbAP6CNMpN26OTBzE2hPw8Ny7d+RvcuFDEnZO2YG17YeDwnXu5d4TcibBN+0zMbm1vZOdje3t39weJQ/PmnIBKE1knA9FysKSc+bQODhthYJiz+G06YzuUr85pkKywH+CSUi7Hh74zGUEg0J23i4ag1LHwzB03Pgh6cGFkSsa7gJaAuPEjv/M1DXW70YksSw8wWzbE5LXxXWXBTQvGp2/rXTD0jkUR8Ix1K2LTOEbowFMJpkutEkoaYjPCAtpX0sUdlN54GkehFRfq6Gwj1fNCndHEixp6UE89RnemdctlL4TqvHYF7042ZH0ZAfTJb5EZch0BPU9X7TFACfKIEJoKpW3UyxAITUNnVAjW8pdXRaNSti7LV4+VQvV2HkcWnaFzVEIWukZVdI9qI4Iekbv6BN9aS/ah/at/cxaM9p85hT9q0zmF4FDsTg=</latexit> <latexit sha1_base64="y2FWsphi+t+VGdcXlc76o/Lki9c=">ADHicbVLbtQwFHXCq4RHp7BkY3WmokholAltobuqbGCBVKROW2kyRI7jzFh1HMt2CiPLH8CGX2HDAoTY8gHs+BucTDpMH1fK1fG956c3DgVjCodhn89/8bNW7fvrNwN7t1/8HC1s/boSJWVxGSIS1bKkxQpwignQ01IydCElSkjBynp6/r/vEZkYqW/FDPBkXaMJpTjHSrpSsesbvVhMaRLrKdFoMy6Qnqa5eWs/6Ge9wDXrAkbMvLPuGFeKCIRP0YSMsjMqFEcFUWPzqXFiHSEjubPSHA1SAk0qZQ1cpJaE/aj5zDs79RpK7LBJfq+LD/yBXV7t6ZFTQ6dsomXWb1zo4VNzH/TtudUL4geEsQWmEt0kG5wLNozr9Z42gkmn60abgFfBoAVd0MZB0vkTZyWuCsI1Zkip0SAUemyQ1BQzYoPlLTrY7nC+QrjhKhnMS+kermFTXZ4wqFBqVqSOWdtUl3t18breqNL5q7GhXFSacDx/UV4xqEtY3wyYUmwZjMHEJbUeYV46v4f1u7+zJewW8fO4pOvgqOoP3jR3ofdf23WsgCdgHWyCAXgJ9sAbcACGAHufva/ed+H/8X/5v/0f82pvtfOPAYXwv/9D6/i9XA=</latexit> <latexit sha1_base64="zG0+86ACKs0ZivmNXL/57ldlvG0=">AB83icbVDLSsNAFL2pr1pfVZduBlvBVUmr+NgV3eiugn1AE8pkOmHTiZhZiKU0N9w40IRt/6MO/GSRpErQcGDufcyz1zvIgzpW370yosLa+srhXSxubW9s75d29jgpjSWibhDyUPQ8rypmgbc0p71IUhx4nHa9yXqdx+oVCwU93oaUTfAI8F8RrA2klN1AqzHnp/czqDcsWu2RnQIqnpAI5WoPyhzMSRxQoQnHSvXrdqTdBEvNCKezkhMrGmEywSPaN1TgCo3yTLP0JFRhsgPpXlCo0z9uZHgQKlp4JnJNKL6Xif14/1v6FmzARxZoKMj/kxzpEKUFoCGTlGg+NQTyUxWRMZYqJNTaWshMsUZ9fXiSdRq1+Uju9a1SaV3kdRTiAQziGOpxDE26gBW0gEMEjPMOLFVtP1qv1Nh8tWPnOPvyC9f4FiwCRfw=</latexit> <latexit sha1_base64="KWmfCBMJGy46BFqyZVbSAliZW7g=">ACK3icbVDLSsNAFJ34rPEVdekm2BbqpiQV0WpG91VsA9oY5hMJ+3QySTMTAol5H/c+CsudOEDt/6H0zaL2Hpg4HDOucy9x4soEdKyPrW19Y3Nre3Cjr67t39waBwdt0UYc4RbKQh73pQYEoYbkiKe5GHMPAo7jW9mfmeCuSAhe5DTCDsBHDLiEwSlklyjUS4NK/0AypHnJ3fpozwv6eWSn5OUkItMUjfJx5XtGkWras1hrhI7I0WQoekar/1BiOIAM4koFKJnW5F0EsglQRSnej8WOIJoDIe4pyiDARZOMr81NctKGZh+yNVj0pyr+YkEBkJMA08lZ2uKZW8m/uf1YulfOwlhUSwxQ4uP/JiaMjRnxZkDwjGSdKoIRJyoXU0ghwiqerVQn28smrpF2r2hfVy/tasd7I6iAU3AGKsAGV6AObkETtACT+AFvIMP7Vl7076070V0TctmTsAfaD+/XpSmBQ=</latexit> <latexit sha1_base64="Tn5ijQgOnAy4mPiBG3gQMhvSFH4=">ACE3icbVDLSsNAFJ3UV62vqEs3waZQXZSkio9d0Y0uhAr2AU0tk+mkHTp5MHMjlNB/cOvuHGhiFs37vwbkzSIWg8MnDnXu69xw4k2AYn0pubn5hcSm/XFhZXVvfUDe3mtIPBaEN4nNftG0sKWcebQADTtuBoNi1OW3Zo/PEb91RIZnv3cA4oF0XDzmMIhlnrqfkm3giHrWTCkgMuWi2FoO9Hl5Bb29IKe/gnm0dVE76lFo2Kk0GaJmZEiylDvqR9W3yehSz0gHEvZMY0AuhEWwAink4IVShpgMsID2omph10qu1F60QrxUpfc3wRPw+0VP3ZEWFXyrFrx5XJjvKvl4j/eZ0QnJNuxLwgBOqR6SAn5Br4WhKQ1meCEuDjmGAiWLyrRoZYAJxjIU0hNMER98nz5JmtWIeVA6vq8XaWRZHu2gXVRGJjpGNXSB6qiBCLpHj+gZvSgPypPyqrxNS3NK1rONfkF5/wJm4J1f</latexit> <latexit sha1_base64="ZYkFvcOHAPMcQbcKwhPq3ujeh+c=">ACG3icbVDLSsNAFJ34rPEVdelmsCnUTUkqosuiG91VsA9oY5lMJ+3QySTMTIQS+h9u/BU3LhRxJbjwb5y0WdTWAwOHc85l7j1+zKhUjvNjrKyurW9sFrbM7Z3dvX3r4LApo0Rg0sARi0TbR5IwyklDUcVIOxYEhT4jLX90nfmtRyIkjfi9GsfEC9GA04BipLTUs6ole1DuhkgN/SC9nTyoU9s2cGcpAV7PmD3rKJTcaAy8TNSRHkqPesr24/wklIuMIMSdlxnVh5KRKYkYmZjeRJEZ4hAakoylHIZFeOr1tAkta6cMgEvpxBafq/ESKQinHoa+T2ZJy0cvE/7xOoJL6U8ThThePZRkDCoIpgVBftUEKzYWBOEBdW7QjxEAmGl6zR1Ce7iycukWa24Z5Xzu2qxdpXUQDH4ASUgQsuQA3cgDpoAyewAt4A+/Gs/FqfBifs+iKkc8cgT8wvn8BRCKfGg=</latexit> <latexit sha1_base64="8oYhATVQxn+kCv7hSq1/xFVvwEs=">ACSHicbZBLS8NAFIUn9VXrq+rSTbAp1E1JKqLohvdVbAP6CNMpN26OTBzE2hPw8Ny7d+RvcuFDEnZO2YG17YeDwnXu5d4TcibBN+0zMbm1vZOdje3t39weJQ/PmnIBKE1knA9FysKSc+bQODhthYJiz+G06YzuUr85pkKywH+CSUi7Hh74zGUEg0J23i4ag1LHwzB03Pgh6cGFkSsa7gJaAuPEjv/M1DXW70YksSw8wWzbE5LXxXWXBTQvGp2/rXTD0jkUR8Ix1K2LTOEbowFMJpkutEkoaYjPCAtpX0sUdlN54GkehFRfq6Gwj1fNCndHEixp6UE89RnemdctlL4TqvHYF7042ZH0ZAfTJb5EZch0BPU9X7TFACfKIEJoKpW3UyxAITUNnVAjW8pdXRaNSti7LV4+VQvV2HkcWnaFzVEIWukZVdI9qI4Iekbv6BN9aS/ah/at/cxaM9p85hT9q0zmF4FDsTg=</latexit> <latexit sha1_base64="CZMu9i9mqYh9pdzTQnbewbqTXR0=">AB73icbVDLSsNAFJ3UV62vqks3g63gqiRVfOyKblxWsA9oQ5lMJ+3QySTO3Agl9CfcuFDErb/jzr9xkgZR64ELh3Pu5d57vEhwDb9aRWldW14rpY3Nre2d8u5eW4exoqxFQxGqrkc0E1yFnAQrBspRgJPsI43uU79zgNTmofyDqYRcwMyktznlICRutU+jBmQ6qBcsWt2BrxInJxUI7moPzRH4Y0DpgEKojWPceOwE2IAk4Fm5X6sWYRoRMyYj1DJQmYdpPs3hk+MsoQ+6EyJQFn6s+JhARaTwPdAYExvqvl4r/eb0Y/As34TKgUk6X+THAkOI0+fxkCtGQUwNIVRxcyumY6IBRNRKQvhMsXZ98uLpF2vOSe109t6pXGVx1FEB+gQHSMHnaMGukFN1EIUCfSIntGLdW89Wa/W27y1YOUz+gXrPcvduyPug=</latexit> <latexit sha1_base64="CZMu9i9mqYh9pdzTQnbewbqTXR0=">AB73icbVDLSsNAFJ3UV62vqks3g63gqiRVfOyKblxWsA9oQ5lMJ+3QySTO3Agl9CfcuFDErb/jzr9xkgZR64ELh3Pu5d57vEhwDb9aRWldW14rpY3Nre2d8u5eW4exoqxFQxGqrkc0E1yFnAQrBspRgJPsI43uU79zgNTmofyDqYRcwMyktznlICRutU+jBmQ6qBcsWt2BrxInJxUI7moPzRH4Y0DpgEKojWPceOwE2IAk4Fm5X6sWYRoRMyYj1DJQmYdpPs3hk+MsoQ+6EyJQFn6s+JhARaTwPdAYExvqvl4r/eb0Y/As34TKgUk6X+THAkOI0+fxkCtGQUwNIVRxcyumY6IBRNRKQvhMsXZ98uLpF2vOSe109t6pXGVx1FEB+gQHSMHnaMGukFN1EIUCfSIntGLdW89Wa/W27y1YOUz+gXrPcvduyPug=</latexit> <latexit sha1_base64="Ly2C3v3mHZfg5kOK31IrZoYRcrg=">ACt3icbVFNb9QwEHXCVwkfXeDIxWK3UpHQKhuWwt6qcoEDUpHYtmKzRI7jbKw6TrAnhZXlv8iBG/8GJxvKljKSrTdv3shvxmktuIYw/OX5N27eun1n525w7/6Dh7uDR49PdNUoyua0EpU6S4lmgks2Bw6CndWKkTIV7DQ9f9vWTy+Y0rySn2Bds2VJVpLnBJwVDL4sTeK64InMRQMyH5cEijS3Ly3X+D5KHDFlqBEmA/WpXGjWU3oOVmxRXbBay1JyfTSfO+MWCfIWO6cdKkhuiaKrBptjVql1oTj6AUOxwftNY1scEV8pKpv8lL4ataKou4ObWDibdHoj8vSJuavYzuyQTIYuoYu8HUw6cEQ9XGcDH7GWUWbkmgmi9mIQ1LA1RwKlgzuPWxA72827GxXuOyXBeKXck4I7d7jCk1Hpdpk7ZutT/1lryf7VFA/mbpeGyboBJunkobwSGCrefiDOuGAWxdoBQxZ1XTAu3awruqzdLmLVxcDnydXASjScvx9OP0fDwqF/HDnqKnqF9NEGv0SF6h47RHFv6n32qJf5Mz/xc7/YSH2v73mCroT/9Tdnf9X</latexit> PIRL - How it works f ( v I ) I v I res5 Dissimilar θ m I 0 Similar I t Should be similar m I g ( v I t ) v I t Similar m I 0 res5 θ Dissimilar Memory Bank M Unrelated (Negative) res5 θ 72

  28. Better self-supervised learning objective Accuracy on ImageNet-1K 73

  29. Object Detection • Outperforms ImageNet supervised pre-trained networks • Full fine-tuning, no bells & whistles • No extra data, changes in model architecture, fine-tuning schedule Initialization VOC07+12 VOC07 AP all AP 50 AP 75 AP all AP 50 AP 75 ImageNet 52.6 81.1 57.4 43.8 74.5 45.9 Supervised +2.3 +1.1 +1.4 PIRL 54.0 80.7 59.7 44.7 73.4 47.0 74

  30. Linear Classification • Linear classifiers on fixed features. Evaluate on ImageNet-1K CPCv2 75

  31. Easily Multi-task Transfer Dataset Method ImageNet-1M VOC07 Places205 iNaturalist Jigsaw 46.0 66.1 41.4 22.1 Rotation 48.9 63.9 47.6 23 PIRL (Rot) 60.2 77.1 47.6 31.2 PIRL (Jigsaw + Rot) 63.1 80.3 49.7 33.6 76

  32. The rise of contrastive learning

  33. Contrastive Learning • How to define what images are "related" and "unrelated"? Related and Unrelated Images

  34. Frames of a video Video & Audio Hadsell et al., 2005, DrLim AVID - Morgado et al., ECCV 2020 van der Oord et al., 2018, CPC GDT - Patrick et al., 2020

  35. Tracking Objects 80 Wang & Gupta, 2015, Unsupervised Learning of Visual Representations using Videos

  36. Nearby patches vs. distant patches of an Image Related (Positives) van der Oord et al., 2018, Hena ff et al., 2019 Contrastive Predictive Coding Unrelated (Negative)

  37. Patches of an image vs. patches of other images Related Wu et al., 2018, Instance Discrimination (Positives) He et al., 2019, MoCo Misra & van der Maaten, 2019, PIRL Chen et al., 2020, SimCLR Unrelated and lots more .... (Negative)

  38. Is "contrastive" really important?

  39. Contrastive learning -- what does it do? Negative samples Positive Sample Negative samples

  40. Contrastive learning -- what does it do? Negative samples Negative samples Positive Sample

  41. Contrastive learning -- what does it do? Creates groups in the feature space

  42. Contrastive learning -- what does it do? Creates groups in the feature space So does clustering ?!

  43. Swapping Assignments between Views (SwAV) Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, Armand Joulin

  44. Grouping Prototypes Similarity of dataset sample & prototypes Dataset (which cluster does a sample belong to?) See also - SeLa by Asano et al., 2019 89

  45. Grouping Prototypes } Dataset Codes 90

  46. Prototypes f θ Code 1 f θ Code 2

  47. Prototypes f θ Code 1 Predict f θ Code 2

  48. Prototypes Backprop f θ Code 1 Backprop f θ Code 2 Not contrastive!

  49. Key Results Linear Classifier Detection (Fixed Features) ImageNet Places iNaturalist VOC07+12 COCO Supervised 76.5 53.2 46.7 81.3 40.8 Prior self-supervised 71.1 (-5.4) 52.1 38.9 82.5 42.0 SwAV 75.3 (-1.2) 56.7 48.6 82.6 42.1 94

  50. Practical advantages of SwAV • Trains on 4-8 GPUs • Faster convergence than prior work (SimCLR, MoCov2) t n e d u t S % 0 y 0 • Smaller compute requirements. l d 1 n e i r f • 2x faster than MoCo-v2 on 8 GPUs • 72% after 100h vs. 71% after 200h • Better results Code & Models - https://github.com/facebookresearch/swav PyTorch Lightning implementation on the way 95

  51. Combining clustering with contrastive learning

  52. Audio Visual Instance Discrimination with Cross Modal Agreement (AVID + CMA) Pedro Morgado, Nuno Vasconcelos, Ishan Misra https://github.com/facebookresearch/AVID-CMA

  53. Contrastive (Audio Video Instance Discrimination) Positives Negatives d( ) < d( ) d( ) < d( ) } } Audio & Video Relate to other video/audio (same sample) using negatives 98

  54. Grouping using Audio-visual Agreements (CMA) Reference Positives Negatives i v j ) Video Similarity ( v T Positives d( ) < d( ) d( ) < d( ) Visual Negatives } Videos that are similar in Audio Similarity ( a T i a j ) Audio audio & video features Negatives Positive Set Negative Set 99

  55. Grouping using Audio-visual Agreements (CMA) Example 3 Example 2 Example 1 Reference Playing Violin Moving Train Dancing i v j ) Video Similarity ( v T Positives Moving Train Playing Violin Dancing Visual Negatives Fire Truck Station Playing Guitar Exercising Audio Similarity ( a T i a j ) Audio Negatives Positive Set Negative Set Moving Boat Playing Accordion Fishing with background music 100

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Computer Vision: Weakly-supervised learning from video and images Ivan Laptev

Computer Vision: Weakly-supervised learning from video and images Ivan Laptev

CSClub Saint Petersburg November 17, 2014 Computer Vision: Weakly-supervised learning from video and images Ivan Laptev ivan.laptev@inria.fr WILLOW, INRIA/ENS/CNRS, Paris Joint work with: Piotr Bojanowski Rmi Lajugie Maxime Oquab

1.03k views • 80 slides
Vision and Language Representation Learning Self Supervised Pretraining and Multi-Task Learning

Vision and Language Representation Learning Self Supervised Pretraining and Multi-Task Learning

Vision and Language Representation Learning Self Supervised Pretraining and Multi-Task Learning Jiasen Lu April 21, 2020 1 1 Vision and Language Visual Question Answering Image Captioning Visual Commonsense Reasoning Refer Expression 2

1.13k views • 53 slides
Persistent self-supervised learning principle: from stereo to monocular vision for obstacle

Persistent self-supervised learning principle: from stereo to monocular vision for obstacle

Persistent self-supervised learning principle: from stereo to monocular vision for obstacle avoidance by Kevin van Hecke, Guido de Croon, Laurens van der Maaten, Daniel Hennes, and Dario Izzo presented by Mike Kalaitzakis What is Persistent

503 views • 13 slides
Learning frameworks Self-supervised learning: (Auto)encoder networks Supervised learning Network

Learning frameworks Self-supervised learning: (Auto)encoder networks Supervised learning Network

Learning frameworks Self-supervised learning: (Auto)encoder networks Supervised learning Network must copy inputs to outputs through Assumes environment specifies correct output (targets) for each input a bottleneck (fewer hidden units)

407 views • 5 slides
Self-Supervised Feature Learning by Learning to Spot Artifacts Wonbin Kim Self-Supervised

Self-Supervised Feature Learning by Learning to Spot Artifacts Wonbin Kim Self-Supervised

Self-Supervised Feature Learning by Learning to Spot Artifacts Wonbin Kim Self-Supervised Learning To exploit different labelings that are freely available besides or within visual data, and to use them as intrinsic reward signals to

387 views • 24 slides
Stacking for supervised learning Stacking for supervised learning Niall Rooney, NIKEL,

Stacking for supervised learning Stacking for supervised learning Niall Rooney, NIKEL,

Stacking for supervised learning Stacking for supervised learning Niall Rooney, NIKEL, University of Ulster 1 Ensemble learning Ensemble learning l Postulate multiple hypotheses to explain the data l Shortcomings of single model learning

681 views • 19 slides
Machine Learning for NLP Supervised Learning Aurlie Herbelot 2019 Centre for Mind/Brain

Machine Learning for NLP Supervised Learning Aurlie Herbelot 2019 Centre for Mind/Brain

Machine Learning for NLP Supervised Learning Aurlie Herbelot 2019 Centre for Mind/Brain Sciences University of Trento 1 Supervised learning Supervised: you know the result of the task you want to perform. Supervised learning mostly

832 views • 61 slides
PCA CS 446 Supervised learning So far, weve done supervised learning: Given (( x i , y i )) ,

PCA CS 446 Supervised learning So far, weve done supervised learning: Given (( x i , y i )) ,

PCA CS 446 Supervised learning So far, weve done supervised learning: Given (( x i , y i )) , find f with f ( x i ) y i . k -nn, decision trees, . . . 1 / 18 Supervised learning So far, weve done supervised learning: Given (( x i , y

1.01k views • 56 slides
Seminar Important Developments in Computer Vision and Machine Learning Kickoff Meeting

Seminar Important Developments in Computer Vision and Machine Learning Kickoff Meeting

Seminar Current Topics in Computer Vision and Machine Learning Seminar Important Developments in Computer Vision and Machine Learning Kickoff Meeting 18.10.2019 Prof. Dr. Bastian Leibe RWTH Aachen University, Computer Vision Group

627 views • 33 slides
Deep Learning in Computer Vision Caner Hazrba Deep Learning in Action 24. June 15

Deep Learning in Computer Vision Caner Hazrba Deep Learning in Action 24. June 15

Deep Learning in Computer Vision Caner Hazrba Deep Learning in Action 24. June 15 Computer Vision Group 6 Postdocs, 16 PhD students Caner Hazrba | vision.in.tum.de Deep Learning in Computer Vision 2 Research in Computer Vision

960 views • 30 slides
Computer Vision II Bjoern Andres Machine Learning for Computer Vision TU Dresden 2020-05-22

Computer Vision II Bjoern Andres Machine Learning for Computer Vision TU Dresden 2020-05-22

Computer Vision II Bjoern Andres Machine Learning for Computer Vision TU Dresden 2020-05-22 Image decomposition So far, we have studied pixel classification , a problem whose feasible solutions define decisions at the pixels of an image

817 views • 27 slides
Semi-Supervised Learning Maria-Florina Balcan 03/30/2015 Readings: Semi-Supervised Learning.

Semi-Supervised Learning Maria-Florina Balcan 03/30/2015 Readings: Semi-Supervised Learning.

Semi-Supervised Learning Maria-Florina Balcan 03/30/2015 Readings: Semi-Supervised Learning. Encyclopedia of Machine Learning. Jerry Zhu, 2010 Combining Labeled and Unlabeled Data with Co- Training. Avrim Blum, Tom Mitchell. COLT

953 views • 45 slides
Self-supervised Learning for Vision-and-Language Licheng Yu, Yen-Chun Chen, Linjie Li Nowadays

Self-supervised Learning for Vision-and-Language Licheng Yu, Yen-Chun Chen, Linjie Li Nowadays

Self-supervised Learning for Vision-and-Language Licheng Yu, Yen-Chun Chen, Linjie Li Nowadays Machine Learning Algorithm Data Compute Nowadays Machine Learning Algorithm Data Compute Datasets + Labels MS COCOs Image Captioning:

3.23k views • 115 slides
RECSM Summer School: Machine Learning for Social Sciences Session 1.3: Supervised Learning and

RECSM Summer School: Machine Learning for Social Sciences Session 1.3: Supervised Learning and

RECSM Summer School: Machine Learning for Social Sciences Session 1.3: Supervised Learning and Model Accuracy Reto West Department of Political Science and International Relations University of Geneva 1 Supervised Learning Supervised

436 views • 42 slides
CSE 158 Lecture 1.5 Web Mining and Recommender Systems Supervised learning Regression

CSE 158 Lecture 1.5 Web Mining and Recommender Systems Supervised learning Regression

CSE 158 Lecture 1.5 Web Mining and Recommender Systems Supervised learning Regression What is supervised learning? Supervised learning is the process of trying to infer from labeled data the underlying function that produced the labels

751 views • 25 slides
CSE 158 Lecture 1.5 Web Mining and Recommender Systems Supervised learning Regression

CSE 158 Lecture 1.5 Web Mining and Recommender Systems Supervised learning Regression

CSE 158 Lecture 1.5 Web Mining and Recommender Systems Supervised learning Regression What is supervised learning? Supervised learning is the process of trying to infer from labeled data the underlying function that produced the labels

784 views • 33 slides
Millifluidics, 4. Actual surfaces The effect of the roughness q a q r rough flat wax surface

Millifluidics, 4. Actual surfaces The effect of the roughness q a q r rough flat wax surface

Millifluidics, 4. Actual surfaces The effect of the roughness q a q r rough flat wax surface Dettre &amp; Jonhson Super-hydrophobic surfaces 200 m Onda et al. Apparent contact angle Cos q * q * cos q q Onda et al. A textured surface

144 views • 10 slides
Will Start at 2:35 pm Data Visualization The Good, the Bad, the Weird Nam Wook Kim Mini-Courses

Will Start at 2:35 pm Data Visualization The Good, the Bad, the Weird Nam Wook Kim Mini-Courses

Will Start at 2:35 pm Data Visualization The Good, the Bad, the Weird Nam Wook Kim Mini-Courses January @ GSAS 2018 Goal Rules of thumbs to critique visualization design 45 Ways to Communicate Two Quantities [Santiago Ortiz 2012]

1.2k views • 105 slides
How to write a Letter to the Editor David P. Billings Disclaimer / Legal Fine Print The

How to write a Letter to the Editor David P. Billings Disclaimer / Legal Fine Print The

How to write a Letter to the Editor David P. Billings Disclaimer / Legal Fine Print The foregoing materials and information are provided solely for informational purposes, and are not offered as or intended to constitute legal advice, nor do

459 views • 9 slides
A friendly introduction to knots in three and four dimensions Arunima Ray Rice University SUNY

A friendly introduction to knots in three and four dimensions Arunima Ray Rice University SUNY

A friendly introduction to knots in three and four dimensions Arunima Ray Rice University SUNY Geneseo Mathematics Department Colloquium April 25, 2013 What is a knot? Take a piece of string, tie a knot in it, glue the two ends together.

570 views • 55 slides
Magnetic Field of a Wire Fundamental Laws for Calculating B-field Biot-Savart Law (long

Magnetic Field of a Wire Fundamental Laws for Calculating B-field Biot-Savart Law (long

Magnetic Field of a Wire Fundamental Laws for Calculating B-field Biot-Savart Law (long method, but works always) Amperes Law (high symmetry) B-Field of a Straight Wire For a thin straight conductor carrying current the

459 views • 20 slides
Reviews Using Off-The-Shelf Argumentation Mining Marco Passon*, Marco Lippi, Giuseppe Serra* ,

Reviews Using Off-The-Shelf Argumentation Mining Marco Passon*, Marco Lippi, Giuseppe Serra* ,

Predicting the Usefulness of Amazon Reviews Using Off-The-Shelf Argumentation Mining Marco Passon*, Marco Lippi, Giuseppe Serra* , Carlo Tasso* * University of Udine University of Modena and Reggio Emilia Artificial Intelligence Laboratory

471 views • 18 slides
Natural Language Processing Info 159/259 Lecture 3: Text classification 2 (Aug 30, 2018)

Natural Language Processing Info 159/259 Lecture 3: Text classification 2 (Aug 30, 2018)

Natural Language Processing Info 159/259 Lecture 3: Text classification 2 (Aug 30, 2018) David Bamman, UC Berkeley Bayes Rule Likelihood of really really the Prior belief that Y = positive worst movie ever (before you see

1.02k views • 54 slides
Why Do People Give? An Experimental Test of Pure and Impure Altruism Lise Vesterlund Mark

Why Do People Give? An Experimental Test of Pure and Impure Altruism Lise Vesterlund Mark

Why Do People Give? An Experimental Test of Pure and Impure Altruism Lise Vesterlund Mark Ottoni-Wilhelm Huan Xie University of Pittsburgh IUPUI and IU Lilly Family School of Philanthropy Concordia University Workshop in

802 views • 57 slides

More recommend