Quantity vs. Quality: Evaluating User Interest Profiles Using Ad - PowerPoint PPT Presentation

Quantity vs. Quality: Evaluating User Interest Profiles Using Ad Preference Managers Muhammad Ahmad Bashir Umar Farooq Maryam Shahid   Muhammad Fareed Za ff ar Christo Wilson

Online Tracking � 2

Online Tracking sports shoes shoes soccer shoes men’s soccer shoes soccer shoes soccer shoes   (phantom series) � 2

Inferences Used For Targeted Ads washingtonpost.com � 3

We Don’t Know What Ad Networks Infer � 4

Goals of the Study 1. Who knows what and how much? 2. How do users perceive interests inferred about them? 3. How are the interests inferred? 4. How do privacy practices impact amount of inferences drawn? � 5 � 5

Ad Preference Managers (APMs) • Transparency tools • Let users control the inferred interests about them � 6

Overview 1. Data collection 2. Interests inferred by di ff erent APMs 3. Perception of interests 4. Limitations & Conclusion � 7

Data Collection � 8

Data Collection • We recruited 220 participants • 82 from Pakistan (university students), 138 from US (crowdsource) � 8

Data Collection • We recruited 220 participants • 82 from Pakistan (university students), 138 from US (crowdsource) • Used our browser extension to A. Take a survey B. Contribute data from their APMs + Historical Data � 8

Data Collection • We recruited 220 participants • 82 from Pakistan (university students), 138 from US (crowdsource) • Used our browser extension to A. Take a survey B. Contribute data from their APMs + Historical Data Ethics • Obtained IRB from both LUMS and Northeastern University • Obtained informed consent. � 8

Browser Extension Background Foreground � 9

Browser Extension Basic Demographics Foreground Education Age Location Background � 9

Browser Extension Basic General Web Demographics Usage Foreground Education Age Location Background � 9

Browser Extension Online Ads & Basic General Web Privacy Practices Demographics Usage Foreground Education Age Location Background � 9

Browser Extension Online Ads & Basic General Web Privacy Practices Demographics Usage Foreground Education Age Location Historical Data Background Search History Browsing History � 9

Browser Extension Online Ads & Basic General Web Privacy Practices Demographics Usage Foreground Education Age Location Historical Data Ad Preference Managers Background Search History Browsing History � 9

Browser Extension Online Ads & Dynamic   Basic General Web Privacy Practices Questions Demographics Usage Foreground Education Age Location Historical Data Ad Preference Managers Background Search History Randomly Sampled Interests Browsing History � 9

Dynamic Questions � 10

Dynamic Questions � 11

Summary of Data Collection 220 participants (82 from Pakistan, 138 from US) For each participant, we have: � 12

Summary of Data Collection 220 participants (82 from Pakistan, 138 from US) For each participant, we have: Foreground Background � 12

Summary of Data Collection 220 participants (82 from Pakistan, 138 from US) For each participant, we have: Foreground Background • Survey 1. Basic demographics 2. General web usage 3. Interaction with Ads 4. Privacy practices 5. Knowledge about APMs 6. Relevance of interests � 12

Summary of Data Collection 220 participants (82 from Pakistan, 138 from US) For each participant, we have: Foreground Background • Interests from 4 APMS • Survey 1. Facebook 1. Basic demographics 2. Google 2. General web usage 3. BlueKai 3. Interaction with Ads 4. eXelate 4. Privacy practices 5. Knowledge about APMs • Browsing history (last 3 months) 6. Relevance of interests • Search term history (last 3 months) � 12

Goals of the Study 1. Who knows what and how much? • What inferences are drawn by each APM? • Does every APM infer the same information? 2. How do users perceive these interests inferred about them? � 13

Which APM Knows More? � 14

Which APM Knows More? Table: Interests gathered from 220 participants Inferred Interests APM Users Unique Total Avg. per User Google 213 594 9,013 42.3 Facebook 208 25,818 108,930 523.7 BlueKai 220 3,522 92,926 422.4 eXelate 218 139 1,941 8.9 � 14

Which APM Knows More? Table: Interests gathered from 220 participants Inferred Interests APM Users Unique Total Avg. per User Google 213 594 9,013 42.3 Facebook 208 25,818 108,930 523.7 BlueKai 220 3,522 92,926 422.4 eXelate 218 139 1,941 8.9 • Facebook gathers maximum interests, while eXelate has the least � 14

Which APM Knows More? Table: Interests gathered from 220 participants Inferred Interests APM Users Unique Total Avg. per User Google 213 594 9,013 42.3 Facebook 208 25,818 108,930 523.7 BlueKai 220 3,522 92,926 422.4 eXelate 218 139 1,941 8.9 • Facebook gathers maximum interests, while eXelate has the least • Bluekai had a profile on every user � 14

Which APM Knows More? Table: Interests gathered from 220 participants 1 Inferred Interests 0.8 APM Users Unique Total Avg. per User 0.6 CDF Google 213 594 9,013 42.3 0.4 Facebook 208 25,818 108,930 523.7 0.2 BlueKai 220 3,522 92,926 422.4 0 eXelate 218 139 1,941 8.9 1 10 100 1000 10000 # Interests • Fig: CDF of interests per user Facebook gathers maximum interests, while eXelate has the least • Bluekai had a profile on every user � 14

Which APM Knows More? Categories capped by Google Table: Interests gathered from 220 participants 1 Inferred Interests 0.8 APM Users Unique Total Avg. per User 0.6 CDF Google 213 594 9,013 42.3 0.4 Facebook 208 25,818 108,930 523.7 0.2 BlueKai 220 3,522 92,926 422.4 0 eXelate 218 139 1,941 8.9 1 10 100 1000 10000 # Interests • Fig: CDF of interests per user Facebook gathers maximum interests, while eXelate has the least • Bluekai had a profile on every user � 14

Canonicalization of Interests We cannot directly compare interests from di ff erent APMs • Synonyms: Real Estate, Property • Granularity: Sports, Tennis, Wimbledon   For fair comparison, we need to map interests to a common space � 15

Canonicalization of Interests We cannot directly compare interests from di ff erent APMs • Synonyms: Real Estate, Property • Granularity: Sports, Tennis, Wimbledon   For fair comparison, we need to map interests to a common space We used Open Directory Project (ODP) • Manually mapped raw interest to 465 ODP categories � 15

Canonicalization of Interests We cannot directly compare interests from di ff erent APMs • Synonyms: Real Estate, Property • Granularity: Sports, Tennis, Wimbledon   For fair comparison, we need to map interests to a common space FB Bluekai Soccer Softball We used Open Directory Project (ODP) • Manually mapped raw interest to 465 ODP categories � 15

Canonicalization of Interests We cannot directly compare interests from di ff erent APMs • Synonyms: Real Estate, Property • Granularity: Sports, Tennis, Wimbledon   For fair comparison, we need to map interests to a common space FB Bluekai Soccer Softball We used Open Directory Project (ODP) • Manually mapped raw interest to 465 ODP categories ODP   Sports Category � 15

Inferred Interests After ODP Mapping 1 0.8 0.6 CDF 0.4 0.2 0 1 10 100 1000 10000 # Interests Fig: CDF of raw interests per user � 16

Inferred Interests After ODP Mapping 1 1 0.8 0.8 0.6 CDF 0.6 CDF 0.4 0.4 0.2 0.2 0 0 1 10 100 1000 10000 0 100 200 300 # Interests # ODP Categories Fig: CDF of raw interests per user Fig: CDF of ODP categories per user � 16

Do APMs Infer Similar Interests? BlueKai eXelate FB Google 1 Fractional Overlap 0.75 0.5 0.25 0 Google FB eXelate BlueKai Fig: Per Participant overlap of ODP categorized interests (min, 5th, median, 95th, max) � 17

Do APMs Infer Similar Interests? BlueKai eXelate FB Google 1 Fractional Overlap 0.75 0.5 0.25 0 Google FB eXelate BlueKai Fig: Per Participant overlap of ODP categorized interests Median Google (min, 5th, median, 95th, max) user’s interest profile has 20% overlap with BlueKai � 17

Quantity vs. Quality: Evaluating User Interest Profiles Using Ad - PowerPoint PPT Presentation

Quantity vs. Quality: Evaluating User Interest Profiles Using Ad Preference Managers Muhammad Ahmad Bashir Umar Farooq Maryam Shahid Muhammad Fareed Za ff ar Christo Wilson Online Tracking 2 Online Tracking 2 Online

Evaluating User Interfaces Evaluating User Interfaces Lecture slides modified from Eileen

STATE WATERS Quantity, Quality, Connected Quantity How much? When? Where? Who has

Quality Sleep Series II : The Sleep Quality and Quantity of HK s s The Sleep Quality and

Evaluating cell lines as tumor models by comparison of genomic profiles Domcke, S. et al. Nat.

Evaluating MT Quality Evaluation of Why do we want to do it? Translation Quality - Want to

CSE440: Introduction to HCI Methods for Design, Prototyping and Evaluating User Interaction

Quantity and Quality of Primary and Secondary School Teachers in China Analysis from an

Unsupervised Learning There is no direct ground truth for the quantity of interest

S2 surface quality (almost car quality) Minimum X-Ray absorption Metal Profiles

Womens Health Initiative Profiles May 2019 Objectives of WHI Profiles Knowing the

UX/UI What is UX and UI? UX Process User Research User Research Creating User

An assessment of the drainage quality and quantity associated with recycled wastewater irrigation

CSE 440: Introduction to HCI User Interface Design, Prototyping, and Evaluation Lecture 06:

Enhancing the Quantity and Quality of ODA for LDCs and Addressing their Debt Problems

CSE 440: Introduction to HCI User Interface Design, Prototyping, and Evaluation James Fogarty

Evaluation of groundwater and surface water resources in quality and quantity at Binh Chanh

ad -heap: an Efficient Heap Data Structure for Asymmetric Multicore Processors Weifeng Liu and

Vectorial AdS/CFT and quantum higher spins Arkady Tseytlin Partition functions and Casimir

Recent analytical and numerical studies of asymptotically AdS spacetimes in spherical symmetry

The SeibergWitten theory Coulomb phase of N = 1 SO ( N ) N = 1, SO ( N ) F = N 2: SO ( N )

Research at the Boundary of Robotics and AI Prof: Peter Stone Department of Computer Science

Showing Relevant Ads via Context Multi-Armed Bandits D avid P al December 17, 2008

CLEF eHealth 2020 @clefehealth CLEF eHealth 2020 Task 1: Multilingual Information Extraction

COMMERCIAL STREET/ COMMERCIAL STREET/ ATLANTIC AVENUE ATLANTIC AVENUE BICYCLE FACILITIES