Lecture 27: Ethics in computer vision (part 2) 1 Announcements - PowerPoint PPT Presentation

Lecture 27: Ethics in computer vision (part 2) 1

Announcements • Last class :( • Extra office hours: • Today: end of class until 2pm • Thurs: 12:00 - 1:00pm. • PS8 grades out (regrade requests due Friday ) • PS9, PS10 due tonight. No late days allowed. 2

Garbage in, garbage out A machine learning algorithm will do whatever the training data tells it to do. If the data is bad or biased, the learned algorithm will be too. 3 Source: Isola, Torralba, Freeman


 
 
 
 
 
 Microsoft’s Tay chatbot Chatbot released on twitter. Learned from interactions with users Started mimicking offensive language, was shut down. 4 Image source: https://money.cnn.com/2016/03/30/technology/tay-tweets-microsoft/index.html

The Giraffe-Tree problem [“Measuring Machine Intelligence Through Visual Question Answering”, Zitnick et al., 2016] 5

Nearest neighbor baseline Train Test 6 Source: L. Zitnick

Nearest Neighbor A black and white cat Two zebras and a giraffe in a field. sitting in a bathroom sink. 7 Source: L. Zitnick See mscoco.org for image information

Image captioning An airplane is parked on the tarmac at an airport. A man riding a motorcycle on a beach. 8 Source: L. Zitnick

Results COCO Caption Challenge CIDEr-D Meteor ROUGE-L BLEU-4 Google [4] 0.943 0.254 0.53 0.309 MSR Captivator [9] 0.931 0.248 0.526 0.308 m-RNN [15] 0.917 0.242 0.521 0.299 MSR [8] 0.912 0.247 0.519 0.291 Nearest Neighbor [11] 0.886 0.237 0.507 0.280 m-RNN (Baidu/ UCLA) [16] 0.886 0.238 0.524 0.302 Berkeley LRCN [2] 0.869 0.242 0.517 0.277 Human [5] 0.854 0.252 0.484 0.217 Montreal/Toronto [10] 0.85 0.243 0.513 0.268 PicSOM [13] 0.833 0.231 0.505 0.281 MLBL [7] 0.74 0.219 0.499 0.26 ACVT [1] 0.709 0.213 0.483 0.246 NeuralTalk [12] 0.674 0.21 0.475 0.224 Tsinghua Bigeye [14] 0.673 0.207 0.49 0.241 MIL [6] 0.666 0.214 0.468 0.216 Brno University [3] 0.517 0.195 0.403 0.134 9 Source: L. Zitnick

Visual Question Answering Dataset Source: L. Zitnick

11 [“Colorful image colorization”, Zhang et al., ECCV 2016] Source: Isola, Torralba, Freeman

12 [“Colorful image colorization”, Zhang et al., ECCV 2016]

13 [“Colorful image colorization”, Zhang et al., ECCV 2016] Source: Isola, Torralba, Freeman

Generalization 14

<latexit sha1_base64="4gXZsPFg4PMoPOoO47pTNWzkob0=">AB9HicbVA9SwNBEN2LXzF+RS1tFoNgFe7SaBm0sYyQL0iOMLe3lyzZ2zt35wIh5HfYWChi64+x89+4Sa7QxAcDj/dmJkXpFIYdN1vp7C1vbO7V9wvHRweHZ+UT8/aJsk04y2WyER3AzBcCsVbKFDybqo5xIHknWB8v/A7E6NSFQTpyn3YxgqEQkGaCW/qUEoY0BIRBueJW3SXoJvFyUiE5GoPyVz9MWBZzhUyCMT3PTdGfgUbBJ+X+pnhKbAxDHnPUgUxN/5sefScXlklpFGibSmkS/X3xAxiY6ZxYDtjwJFZ9xbif14vw+jWnwmVZsgVWy2KMkxoYsEaCg0ZyinlgDTwt5K2Qg0MLQ5lWwI3vrLm6Rdq3pu1XusVep3eRxFckEuyTXxyA2pkwfSIC3CyBN5Jq/kzZk4L86787FqLTj5zDn5A+fzB45lkfA=</latexit> <latexit sha1_base64="4gXZsPFg4PMoPOoO47pTNWzkob0=">AB9HicbVA9SwNBEN2LXzF+RS1tFoNgFe7SaBm0sYyQL0iOMLe3lyzZ2zt35wIh5HfYWChi64+x89+4Sa7QxAcDj/dmJkXpFIYdN1vp7C1vbO7V9wvHRweHZ+UT8/aJsk04y2WyER3AzBcCsVbKFDybqo5xIHknWB8v/A7E6NSFQTpyn3YxgqEQkGaCW/qUEoY0BIRBueJW3SXoJvFyUiE5GoPyVz9MWBZzhUyCMT3PTdGfgUbBJ+X+pnhKbAxDHnPUgUxN/5sefScXlklpFGibSmkS/X3xAxiY6ZxYDtjwJFZ9xbif14vw+jWnwmVZsgVWy2KMkxoYsEaCg0ZyinlgDTwt5K2Qg0MLQ5lWwI3vrLm6Rdq3pu1XusVep3eRxFckEuyTXxyA2pkwfSIC3CyBN5Jq/kzZk4L86787FqLTj5zDn5A+fzB45lkfA=</latexit> <latexit sha1_base64="4gXZsPFg4PMoPOoO47pTNWzkob0=">AB9HicbVA9SwNBEN2LXzF+RS1tFoNgFe7SaBm0sYyQL0iOMLe3lyzZ2zt35wIh5HfYWChi64+x89+4Sa7QxAcDj/dmJkXpFIYdN1vp7C1vbO7V9wvHRweHZ+UT8/aJsk04y2WyER3AzBcCsVbKFDybqo5xIHknWB8v/A7E6NSFQTpyn3YxgqEQkGaCW/qUEoY0BIRBueJW3SXoJvFyUiE5GoPyVz9MWBZzhUyCMT3PTdGfgUbBJ+X+pnhKbAxDHnPUgUxN/5sefScXlklpFGibSmkS/X3xAxiY6ZxYDtjwJFZ9xbif14vw+jWnwmVZsgVWy2KMkxoYsEaCg0ZyinlgDTwt5K2Qg0MLQ5lWwI3vrLm6Rdq3pu1XusVep3eRxFckEuyTXxyA2pkwfSIC3CyBN5Jq/kzZk4L86787FqLTj5zDn5A+fzB45lkfA=</latexit> <latexit sha1_base64="4gXZsPFg4PMoPOoO47pTNWzkob0=">AB9HicbVA9SwNBEN2LXzF+RS1tFoNgFe7SaBm0sYyQL0iOMLe3lyzZ2zt35wIh5HfYWChi64+x89+4Sa7QxAcDj/dmJkXpFIYdN1vp7C1vbO7V9wvHRweHZ+UT8/aJsk04y2WyER3AzBcCsVbKFDybqo5xIHknWB8v/A7E6NSFQTpyn3YxgqEQkGaCW/qUEoY0BIRBueJW3SXoJvFyUiE5GoPyVz9MWBZzhUyCMT3PTdGfgUbBJ+X+pnhKbAxDHnPUgUxN/5sefScXlklpFGibSmkS/X3xAxiY6ZxYDtjwJFZ9xbif14vw+jWnwmVZsgVWy2KMkxoYsEaCg0ZyinlgDTwt5K2Qg0MLQ5lWwI3vrLm6Rdq3pu1XusVep3eRxFckEuyTXxyA2pkwfSIC3CyBN5Jq/kzZk4L86787FqLTj5zDn5A+fzB45lkfA=</latexit> <latexit sha1_base64="CL13TSGHTX83KUz8wOKXVyvItw=">ACVXicfZDbSgMxEIaz6nWs156EyCiJRdL9RLUS+8EStYD7RFZtNpDSbZJZkVy9Kn8FafS3wYwbRW8IQDgY9/jAzf5Ip6SiKXoNwbHxicqo0XZ6ZnZtfWFxavnBpbgXWRapSe5WAQyUN1kmSwqvMIuhE4WVydzjoX96jdTI159TLsKWha2RHCiAvXZ+jI94GgpvFSlSNhsV/QzyChtV7WYp2Gm2U5FrNCQUONeIo4xaBViSQmG/3MwdZiDuoIsNjwY0ulYx3LjP173S5p3U+meID9WvPwrQzvV04p0a6Nb97A3Ev3qNnDp7rUKaLCc04mNQJ1ecUj4n7elRUGq5wGElX5XLm7BgiAfUvnbmCTRfa8coT/P4okfdZqhBUrtZtE29Xw0PfndptbA/rPKM2n0ZNPOv6Z62+42K7GUTU+267sH4wyL7FVtsY2WMx2T47ZjVWZ4Jp9sie2HPwEryF4+HkhzUMRn9W2LcKF94Blr61EA=</latexit> <latexit sha1_base64="CL13TSGHTX83KUz8wOKXVyvItw=">ACVXicfZDbSgMxEIaz6nWs156EyCiJRdL9RLUS+8EStYD7RFZtNpDSbZJZkVy9Kn8FafS3wYwbRW8IQDgY9/jAzf5Ip6SiKXoNwbHxicqo0XZ6ZnZtfWFxavnBpbgXWRapSe5WAQyUN1kmSwqvMIuhE4WVydzjoX96jdTI159TLsKWha2RHCiAvXZ+jI94GgpvFSlSNhsV/QzyChtV7WYp2Gm2U5FrNCQUONeIo4xaBViSQmG/3MwdZiDuoIsNjwY0ulYx3LjP173S5p3U+meID9WvPwrQzvV04p0a6Nb97A3Ev3qNnDp7rUKaLCc04mNQJ1ecUj4n7elRUGq5wGElX5XLm7BgiAfUvnbmCTRfa8coT/P4okfdZqhBUrtZtE29Xw0PfndptbA/rPKM2n0ZNPOv6Z62+42K7GUTU+267sH4wyL7FVtsY2WMx2T47ZjVWZ4Jp9sie2HPwEryF4+HkhzUMRn9W2LcKF94Blr61EA=</latexit> <latexit sha1_base64="CL13TSGHTX83KUz8wOKXVyvItw=">ACVXicfZDbSgMxEIaz6nWs156EyCiJRdL9RLUS+8EStYD7RFZtNpDSbZJZkVy9Kn8FafS3wYwbRW8IQDgY9/jAzf5Ip6SiKXoNwbHxicqo0XZ6ZnZtfWFxavnBpbgXWRapSe5WAQyUN1kmSwqvMIuhE4WVydzjoX96jdTI159TLsKWha2RHCiAvXZ+jI94GgpvFSlSNhsV/QzyChtV7WYp2Gm2U5FrNCQUONeIo4xaBViSQmG/3MwdZiDuoIsNjwY0ulYx3LjP173S5p3U+meID9WvPwrQzvV04p0a6Nb97A3Ev3qNnDp7rUKaLCc04mNQJ1ecUj4n7elRUGq5wGElX5XLm7BgiAfUvnbmCTRfa8coT/P4okfdZqhBUrtZtE29Xw0PfndptbA/rPKM2n0ZNPOv6Z62+42K7GUTU+267sH4wyL7FVtsY2WMx2T47ZjVWZ4Jp9sie2HPwEryF4+HkhzUMRn9W2LcKF94Blr61EA=</latexit> <latexit sha1_base64="CL13TSGHTX83KUz8wOKXVyvItw=">ACVXicfZDbSgMxEIaz6nWs156EyCiJRdL9RLUS+8EStYD7RFZtNpDSbZJZkVy9Kn8FafS3wYwbRW8IQDgY9/jAzf5Ip6SiKXoNwbHxicqo0XZ6ZnZtfWFxavnBpbgXWRapSe5WAQyUN1kmSwqvMIuhE4WVydzjoX96jdTI159TLsKWha2RHCiAvXZ+jI94GgpvFSlSNhsV/QzyChtV7WYp2Gm2U5FrNCQUONeIo4xaBViSQmG/3MwdZiDuoIsNjwY0ulYx3LjP173S5p3U+meID9WvPwrQzvV04p0a6Nb97A3Ev3qNnDp7rUKaLCc04mNQJ1ecUj4n7elRUGq5wGElX5XLm7BgiAfUvnbmCTRfa8coT/P4okfdZqhBUrtZtE29Xw0PfndptbA/rPKM2n0ZNPOv6Z62+42K7GUTU+267sH4wyL7FVtsY2WMx2T47ZjVWZ4Jp9sie2HPwEryF4+HkhzUMRn9W2LcKF94Blr61EA=</latexit> Training data Test data What Google thinks are student bedrooms 15 Source: Isola, Torralba, Freeman