XRay Transparency for the data-driven Web. Mathias Lcuyer ! - - PowerPoint PPT Presentation

▶
xray
SMART_READER_LITE
LIVE PREVIEW

XRay Transparency for the data-driven Web. Mathias Lcuyer ! - - PowerPoint PPT Presentation

Dec 29, 2023 199 likes •685 views

XRay Transparency for the data-driven Web. Mathias Lcuyer ! Guillaume Ducoffe , Francis Lan , Andrei Papancea , Theofilos Petsios , Riley Spahn ! Augustin Chaintreau , and Roxana Geambasu ! ! Columbia University ! 1 Why this ad? Cedars Hotel

slide-1
SLIDE 1

XRay

Transparency for the data-driven Web.

1

Mathias Lécuyer! Guillaume Ducoffe, Francis Lan, Andrei Papancea, Theofilos Petsios, Riley Spahn! Augustin Chaintreau, and Roxana Geambasu!

!

Columbia University!

slide-2
SLIDE 2

Why this ad?

2

Cedars Hotel Loughborough! 36 Bedrooms, Restaurant; Bar Free WiFi, Parking, Best Rates www.thecedarshotel.com

slide-3
SLIDE 3

Why this ad?

3

Cedars Hotel Loughborough! 36 Bedrooms, Restaurant; Bar Free WiFi, Parking, Best Rates www.thecedarshotel.com

H

  • m
  • s

e x u a l i t y

slide-4
SLIDE 4

Why this ad?

4

Ralph Lauren Online! Shop! The official Site for Ralph Lauren Apparel, Accessories & More www.ralphlauren.com

slide-5
SLIDE 5

Why this ad?

5

Ralph Lauren Online! Shop! The official Site for Ralph Lauren Apparel, Accessories & More www.ralphlauren.com

P r e g n a n c y

slide-6
SLIDE 6

Did you know?!

  • Data Brokers can tell when you're sick,

tired and depressed (and sell the information) [CNN ’14]

  • Google Apps for Ed used institutional

emails to target ads in personal accounts? [SafeGov’14]

  • Credit companies use Facebook data to

decide loans? [CNN’13]

6

Did you know?

slide-7
SLIDE 7

Welcome to the big data world

7

  • Myriad of web services parties collect immense

information about us and use it for varied purposes

  • Data has lots of beneficial uses

Useful recommendations Powerful, predictive applications Improve business with effective product placement Improve public health, disaster response …

slide-8
SLIDE 8

Big data lacks transparency

8

  • We have no visibility into what services do with our

data: What is the data used for exactly? Is it being shared? With whom? Can we delete it?

  • Obscurity threatens to transform the data-driven web

into a breeding ground for data misuse.

  • No robust tools exist to reveal data (mis)uses, even

auditors cannot find answers.

slide-9
SLIDE 9

Question: can we build tools that reveal data misuse?

9

  • Which emails trigger which ads?
  • Which prior searches trigger which prices?
  • Does Facebook share our data with third-parties?
slide-10
SLIDE 10

Question: can we build tools that reveal data misuse?

10

Can we do taint tracking systems?

  • Lots of prior work, many successful systems (e.g.,

Taintdroid)

  • Assume a controlled environment (runtime, language, OS)
  • Need something for the complex and uncontrolled web
  • Which emails trigger which ads?
  • Which prior searches trigger which prices?
  • Does Facebook share our data with third-parties?
slide-11
SLIDE 11

XRay

11

  • First generic data tracking system for the Web

associate inputs (e.g., emails) outputs (e.g., ads)

  • It is accurate, scalable, and generic!

Works now on Gmail, Amazon and YouTube

  • Provides key building blocks for a new ecosystem
  • f tools to keep big data in check
slide-12
SLIDE 12

12

Overview

Motivation Design! Evaluation

slide-13
SLIDE 13

Goals

  • 1. Fine-grained, accurate data use prediction
  • Predict use at individual input level (e.g., emails)
  • 2. Scalability
  • Track many inputs (e.g., 100s of emails)
  • 3. Widely applicable and Self-Tuning
  • Applies to many services (e.g., gmail, amazon…)

13

slide-14
SLIDE 14

Web service model

14

Treat Web services as a black box

Data inputs (Di)! e.g., emails, searches, browsing

Targeted outputs (Ok)! e.g., ads, recommendations

slide-15
SLIDE 15

15

Treat Web services as a black box

Data inputs (Di)! e.g., emails, searches, browsing

XRay! (Differential Correlation)

Targeted outputs (Ok)! e.g., ads, recommendations

Web service model

Associations! Di -> Ok

slide-16
SLIDE 16

Differential Correlation

16

  • Key idea: correlate inputs with outputs
  • Populate extra accounts with subsets of inputs
  • Use shadow account observations to relate inputs to
  • utputs
slide-17
SLIDE 17

Differential Correlation

17

  • Key idea: correlate inputs with outputs
  • Populate extra accounts with subsets of inputs
  • Use shadow account observations to relate inputs to
  • utputs

D1, D2, D3

Main account

D2, D3 D1, D2 D1, D3

Shadow accounts

O1 O1

no O1

Association:! D2 -> O1

O1

slide-18
SLIDE 18

18

  • Key idea: correlate inputs with outputs
  • Populate extra accounts with subsets of inputs
  • Use shadow account observations to relate inputs to
  • utputs

D1, D2, D3

Main account

D2, D3 D1, D2 D1, D3

Shadow accounts

O1 O1

no O1

Association:! D2 -> O1

O1

It sounds like a lot of accounts…

Challenge: scaling

slide-19
SLIDE 19

Scalable algorithms

19

  • Theorem Under certain assumptions for any ε > 0

there exists an algorithm that requires C x log(N) accounts to correctly identify the inputs of a targeted ad with probability (1 − ε).

!

  • Algo1: Set Intersection (simple, not robust)
  • Algo2: Bayesian (more robust)
slide-20
SLIDE 20

Algo1: Set Intersection

20

D1, D2 D2, D3 D1, D3

Step 1

D1, D2 D2, D3 D1, D3

X

Step 2 Step 3 D2

Input: Output Ok (an ad), Inputs Dis

(emails), Observations x

Output: Targeted input Step1: Randomly assign emails to shadow accounts. Step2: Take the sets of emails from accounts where the ad appeared. Step3: Compute the intersection of these sets. Step4: if the intersection is non empty: it is the targeted emails else there is no targeting.

slide-21
SLIDE 21

Algo1: Set Intersection

21

  • We prove it needs a

logarithmic number of accounts in number of inputs for high probability of detection

Input: Output Ok (an ad), Inputs Dis

(emails), Observations x

Output: Targeted input Step1: Randomly assign emails to shadow accounts. Step2: Take the sets of emails from accounts where the ad appeared. Step3: Compute the intersection of these sets. Step4: if the intersection is non empty: it is the targeted emails else there is no targeting.

slide-22
SLIDE 22

Challenge: it needs tuning

22

  • Ads must never appear in the wrong accounts

Not true: email redundancy, cache Need a manual threshold to detect emails in a significant number of accounts

  • Doesn’t take low signal into account

Need hard coded minimum number of accounts that see an ad

  • Tuning is service specific and hard to do
slide-23
SLIDE 23

Algo2: Bayesian (XRay)

23

Input: Output Ok (an ad), Inputs Dis (emails), Observations x Output: Targeted input foreach input Di do compute prob. = apply_bayes end compute prob. = apply_bayes return Di with max

  • Bayes’ rule:
  • Probability of observations:

With Di targeted:

! !

slide-24
SLIDE 24
  • Bayes’ rule:

Input: Output Ok (an ad), Inputs Dis (emails), Observations x Output: Targeted input foreach input Di do compute prob. = apply_bayes end compute prob. = apply_bayes return Di with max

Algo2: Bayesian (XRay)

24

  • Probability of observations:

With Di targeted:

! !

slide-25
SLIDE 25

Input: Output Ok (an ad), Inputs Dis (emails), Observations x Output: Targeted input foreach input Di do compute prob. = apply_bayes end compute prob. = apply_bayes return Di with max

Algo2: Bayesian (XRay)

25

  • Bayes’ rule:
  • Probability of observations:

With Di targeted:

! !

  • Pin : probability to see ad if targeted

email in account

  • Pout : probability to see ad if targeted

email not in account

slide-26
SLIDE 26
  • Bayes’ rule:

Input: Output Ok (an ad), Inputs Dis (emails), Observations x Output: Targeted input foreach input Di do compute prob. = apply_bayes end compute prob. = apply_bayes return Di with max

Algo2: Bayesian (XRay)

26

  • Probability of observations:

With Di targeted:

! !

slide-27
SLIDE 27

Input: Output Ok (an ad), Inputs Dis (emails), Observations x Output: Targeted input foreach input Di do compute prob. = apply_bayes end compute prob. = apply_bayes return Di with max

Algo2: Bayesian (XRay)

27

  • Bayes’ rule:
  • Probability of observations:

With Di targeted:

! !

slide-28
SLIDE 28

Input: Output Ok (an ad), Inputs Dis (emails), Observations x Output: Targeted input foreach input Di do compute prob. = apply_bayes end compute prob. = apply_bayes return Di with max

Algo2: Bayesian (XRay)

28

  • If an email is targeted, we

can tell which one.

  • Challenge: what if no

tracked input is targeted?

slide-29
SLIDE 29

Input: Output Ok (an ad), Inputs Dis (emails), Observations x Output: Targeted input foreach input Di do compute prob. = apply_bayes end compute prob. = apply_bayes return Di with max

Algo2: Bayesian (XRay)

29

Without targeting:

  • Probability of observations:

With Di targeted:

! !

  • Bayes’ rule:
slide-30
SLIDE 30

Bayesian can self-tune

30

  • Automatic self-tuning with classic iterative

inference to learn the parameters

  • Many other challenges (input overlap,

different kind of targeting…).

slide-31
SLIDE 31

!

Shadow Account Manager (service specific)

!

Browser Plugin (gives visual feedback)

All inputs All outputs

XRay’s architecture

31

Get associations Tracked inputs

Correlation Engine (service agnostic)

Shadow outputs

XRay - backend

XRay - frontend

User account Shadow 1 … Shadow n

Web service

slide-32
SLIDE 32

Prototype

32

  • We built the prototype for Gmail, to associate ads to

the emails they target.

  • Applied correlation engine as-is to Amazon product

recommendations and YouTube video recommendations.

  • 0 lines of code to change to adapt the correlation

mechanisms.

slide-33
SLIDE 33

33

Talk overview

Motivation Design Evaluation

slide-34
SLIDE 34

Evaluation questions

34

How accurate is XRay?! Is XRay general, extensible and self-tuning? ! How does XRay scale with the number of inputs?! How can we manage input overlap?! Is XRay useful?

slide-35
SLIDE 35

#1 How accurate is XRay?

35

  • We measured recall and precision for XRay’s

associations on Gmail, YouTube and Amazon

  • We need Ground Truth:

Ground truth provided by Amazon and YouTube Manual labeling and validation for Gmail

slide-36
SLIDE 36

#1 How accurate is XRay?

36

0.2 0.4 0.6 0.8 1 10 20 30 40 50 60

Precision Number of Accounts

Gmail Amazon YouTube 0.2 0.4 0.6 0.8 1 10 20 30 40 50 60

Recall Number of Accounts

Gmail Amazon YouTube

Recall and precision are 80-90% for Gmail, YouTube, and Amazon.

Number of inputs (e.g., emails): 16

slide-37
SLIDE 37

#1 How accurate is XRay?

37

Number of inputs (e.g., emails): 16

0.2 0.4 0.6 0.8 1 10 20 30 40 50 60

Precision Number of Accounts

Gmail Amazon YouTube 0.2 0.4 0.6 0.8 1 10 20 30 40 50 60

Recall Number of Accounts

Gmail Amazon YouTube

15 .86 15 .84

Recall and precision are 80-90% for Gmail, YouTube, and Amazon.

slide-38
SLIDE 38

#2 How does XRay scale with the number of inputs?

38

5 10 15 20 25 2 4 8 16 32 64

Number of Accounts Number of Inputs (log)

Gmail Amazon Youtube

Logarithmic dependency, as theory predicted.

slide-39
SLIDE 39

5 10 15 20 25 2 4 8 16 32 64

Number of Accounts Number of Inputs (log)

Gmail 5 10 15 20 25 2 4 8 16 32 64

Number of Accounts Number of Inputs (log)

Gmail Amazon Youtube

#2 How does XRay scale with the number of inputs?

39

Gmail: 2x inputs, +3 accounts.

9 4 15 16 12 8 32 19

x2 +4 x2 x2 +3 +3

slide-40
SLIDE 40

#3 Is XRay useful?

40

  • We collected ads targeting a few sensitive

topics

debt pregnancy race

  • For each topics we have one email with

relevant keywords

  • We analyzed very strong associations

detected by XRay

various diseases sexual orientation divorce

slide-41
SLIDE 41

41 Topic Targeted ads Score Alzheimer Black Mold Allergy Symptoms? Expert to remove Black Mold. 0.99 Adult Assisted Living. Affordable Assisted Living. 0.99 Cancer Ford Warriors in Pink. Join The Fight. 1.0 Rosen Method Bodywork for physical or emotional pain. 0.98 Depression Shamanic healing over the phone. 0.99 Text Coach - Get the girl you want and Desire. 0.99 African American Racial Harassment? Learn your rights now. 0.99 Racial Harassment, Hearing racial slurs? 0.99 Homosexuality SF Gay Pride Hotel. Luxury Waterfront. 0.99 Cedars Hotel Loughborough, 36 Bedrooms, Restaurant, Bar. 0.96 Pregnancy Ralph Lauren Apparel. Official Online Store. 0.99 Find Baby Shower Invitations. Get up to 60% off here! 0.99 Divorce Law Attorneys specializing in special needs kids education. 0.99 Cerbone Law Firm, Helping Good People Thru Bad Times. 0.99 Debt / Loan Take a New Toyota Test Drive, Get a $50 Gift Card On The Spot. 0.99 Great Credit Cards Search. Apply for VISA, MasterCard... 0.99 Car Loan without Cosigner 100% Accepted. [...] 0.99 Car Loans w/ Bad Credit 100% Acceptance! [...] 0.99

Example associations

slide-42
SLIDE 42

42 Topic Targeted ads Score Alzheimer Black Mold Allergy Symptoms? Expert to remove Black Mold. 0.99 Adult Assisted Living. Affordable Assisted Living. 0.99 Cancer Ford Warriors in Pink. Join The Fight. 1.0 Rosen Method Bodywork for physical or emotional pain. 0.98 Depression Shamanic healing over the phone. 0.99 Text Coach - Get the girl you want and Desire. 0.99 African American Racial Harassment? Learn your rights now. 0.99 Racial Harassment, Hearing racial slurs? 0.99 Homosexuality SF Gay Pride Hotel. Luxury Waterfront. 0.99 Cedars Hotel Loughborough, 36 Bedrooms, Restaurant, Bar. 0.96 Pregnancy Ralph Lauren Apparel. Official Online Store. 0.99 Find Baby Shower Invitations. Get up to 60% off here! 0.99 Divorce Law Attorneys specializing in special needs kids education. 0.99 Cerbone Law Firm, Helping Good People Thru Bad Times. 0.99 Debt / Loan Take a New Toyota Test Drive, Get a $50 Gift Card On The Spot. 0.99 Great Credit Cards Search. Apply for VISA, MasterCard... 0.99 Car Loan without Cosigner 100% Accepted. [...] 0.99 Car Loans w/ Bad Credit 100% Acceptance! [...] 0.99

Example associations

slide-43
SLIDE 43

43 Topic Targeted ads Score Alzheimer Black Mold Allergy Symptoms? Expert to remove Black Mold. 0.99 Adult Assisted Living. Affordable Assisted Living. 0.99 Cancer Ford Warriors in Pink. Join The Fight. 1.0 Rosen Method Bodywork for physical or emotional pain. 0.98 Depression Shamanic healing over the phone. 0.99 Text Coach - Get the girl you want and Desire. 0.99 African American Racial Harassment? Learn your rights now. 0.99 Racial Harassment, Hearing racial slurs? 0.99 Homosexuality SF Gay Pride Hotel. Luxury Waterfront. 0.99 Cedars Hotel Loughborough, 36 Bedrooms, Restaurant, Bar. 0.96 Pregnancy Ralph Lauren Apparel. Official Online Store. 0.99 Find Baby Shower Invitations. Get up to 60% off here! 0.99 Divorce Law Attorneys specializing in special needs kids education. 0.99 Cerbone Law Firm, Helping Good People Thru Bad Times. 0.99 Debt / Loan Take a New Toyota Test Drive, Get a $50 Gift Card On The Spot. 0.99 Great Credit Cards Search. Apply for VISA, MasterCard... 0.99 Car Loan without Cosigner 100% Accepted. [...] 0.99 Car Loans w/ Bad Credit 100% Acceptance! [...] 0.99

Example associations

slide-44
SLIDE 44

44 Topic Targeted ads Score Alzheimer Black Mold Allergy Symptoms? Expert to remove Black Mold. 0.99 Adult Assisted Living. Affordable Assisted Living. 0.99 Cancer Ford Warriors in Pink. Join The Fight. 1.0 Rosen Method Bodywork for physical or emotional pain. 0.98 Depression Shamanic healing over the phone. 0.99 Text Coach - Get the girl you want and Desire. 0.99 African American Racial Harassment? Learn your rights now. 0.99 Racial Harassment, Hearing racial slurs? 0.99 Homosexuality SF Gay Pride Hotel. Luxury Waterfront. 0.99 Cedars Hotel Loughborough, 36 Bedrooms, Restaurant, Bar. 0.96 Pregnancy Ralph Lauren Apparel. Official Online Store. 0.99 Find Baby Shower Invitations. Get up to 60% off here! 0.99 Divorce Law Attorneys specializing in special needs kids education. 0.99 Cerbone Law Firm, Helping Good People Thru Bad Times. 0.99 Debt / Loan Take a New Toyota Test Drive, Get a $50 Gift Card On The Spot. 0.99 Great Credit Cards Search. Apply for VISA, MasterCard... 0.99 Car Loan without Cosigner 100% Accepted. [...] 0.99 Car Loans w/ Bad Credit 100% Acceptance! [...] 0.99

Example associations

slide-45
SLIDE 45

45 Topic Targeted ads Score Alzheimer Black Mold Allergy Symptoms? Expert to remove Black Mold. 0.99 Adult Assisted Living. Affordable Assisted Living. 0.99 Cancer Ford Warriors in Pink. Join The Fight. 1.0 Rosen Method Bodywork for physical or emotional pain. 0.98 Depression Shamanic healing over the phone. 0.99 Text Coach - Get the girl you want and Desire. 0.99 African American Racial Harassment? Learn your rights now. 0.99 Racial Harassment, Hearing racial slurs? 0.99 Homosexuality SF Gay Pride Hotel. Luxury Waterfront. 0.99 Cedars Hotel Loughborough, 36 Bedrooms, Restaurant, Bar. 0.96 Pregnancy Ralph Lauren Apparel. Official Online Store. 0.99 Find Baby Shower Invitations. Get up to 60% off here! 0.99 Divorce Law Attorneys specializing in special needs kids education. 0.99 Cerbone Law Firm, Helping Good People Thru Bad Times. 0.99 Debt / Loan Take a New Toyota Test Drive, Get a $50 Gift Card On The Spot. 0.99 Great Credit Cards Search. Apply for VISA, MasterCard... 0.99 Car Loan without Cosigner 100% Accepted. [...] 0.99 Car Loans w/ Bad Credit 100% Acceptance! [...] 0.99

Example associations

!

In a Subprime Bubble for Used Cars, Borrowers Pay Sky-High Rates

slide-46
SLIDE 46

Conclusion

  • Without transparency, the wonderful big-data Web

threatens to become a breeding place for deceptive and unfair practices

  • XRay: the first generic, scalable, and accurate

building block for revealing data targeting Relies on differential correlation, which has provable properties

  • We hope it will support the building of a new

generation of auditing tools to keep the big-data Web in check

46

slide-47
SLIDE 47

Code & Data Available

  • http://xray.cs.columbia.edu
  • Come to me for a demo

47