Robust Learning from Untrusted Sources Nikola Konstantinov - - PowerPoint PPT Presentation

Robust Learning from Untrusted Sources

Nikola Konstantinov Christoph H. Lampert ICML, June 2019

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 1 / 13

Motivation

Collecting data for machine learning applications

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 2 / 13

Motivation

Collecting data for machine learning applications

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 2 / 13

Motivation

Collecting data for machine learning applications

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 2 / 13

Motivation

Using multiple data sources

Crowdsourcing

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 3 / 13

Motivation

Using multiple data sources

Crowdsourcing

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 3 / 13

Motivation

Using multiple data sources

Web crawling

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 3 / 13

Motivation

Using multiple data sources

Data from personal devices

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 3 / 13

Motivation

Using multiple data sources

Data from different labs

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 3 / 13

Motivation

Using multiple data sources

Data from different labs How can we learn robustly from such data?

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 3 / 13

Motivation

Learning from untrusted sources

Motivation Untrusted sources can provide valuable data for training. Some of these data batches might be corrupted or irrelevant. Goal Naive approaches are to:

Simply train on all data. Train only on the trusted subset.

Can we do better?

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 4 / 13

Theory

Setup

Learning task Unknown target distribution DT on X × Y. Loss function L : Y × Y → R+. Want to learn a predictor h : X → Y from a hypothesis class H. Given Have a small reference dataset: ST = {

• xT

1 , yT 1

• , . . . ,
• xT

mT , yT mT

• } ∼ DT

Also given mi data points from each source i = 1, . . . , N: Si = {

• xi

1, yi 1

• , . . . ,
• xi

mi, yi mi

• } ∼ Di

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 5 / 13

Theory

Approach

Assign weights α = (α1, ..., αN) to the sources, N

i=1 αi = 1.

Minimize the α-weighted empirical loss: ˆ hα = argmin

h∈H

ˆ ǫα (h) = argmin

h∈H

 

N

• i=1

αi 1 mi

mi

• j=1

L

• h
• xi

j

• , yi

j

• 

 Want a small expected loss on the target distribution: ǫT

• ˆ

hα

• = EDT
• L(ˆ

hα(x), y)

• How to decide which sources are trustworthy?

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 6 / 13

Theory

Approach

Discrepancies between the sources (Kifer et al., VLDB 2004; Mohri et al., ALT 2012): discH (Di, DT) = sup

h∈H

|ǫi(h) − ǫT(h)| Small if H does not distinguish between the two learning tasks. Popular in the domain adaptation literature.

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 7 / 13

Theory

Bound on the expected loss

Given a hypothesis set H, let:

ˆ hα = argminh∈H ˆ ǫα(h) h∗

T = argminh∈H ǫT(h)

For any δ > 0, with probability at least 1 − δ: |ǫT(ˆ hα) − ǫT(h∗

T)| ≤

2

N

• i=1

αidiscH (Di, DT) + C (δ)

• N
• i=1

α2

i

mi + 4

N

• i=1

αiRi (H, L)

Similar bounds in Ben-David et al., ML 2010; Zhang et al., NIPS 2013.

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 8 / 13

Theory

Bound on the expected loss

Given a hypothesis set H, let:

ˆ hα = argminh∈H ˆ ǫα(h) h∗

T = argminh∈H ǫT(h)

For any δ > 0, with probability at least 1 − δ: |ǫT(ˆ hα) − ǫT(h∗

T)| ≤

2

N

• i=1

αidiscH (Di, DT) + C (δ)

• N
• i=1

α2

i

mi + 4

N

• i=1

αiRi (H, L)

Similar bounds in Ben-David et al., ML 2010; Zhang et al., NIPS 2013.

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 8 / 13

Theory

Bound on the expected loss

Given a hypothesis set H, let:

ˆ hα = argminh∈H ˆ ǫα(h) h∗

T = argminh∈H ǫT(h)

For any δ > 0, with probability at least 1 − δ: |ǫT(ˆ hα) − ǫT(h∗

T)| ≤

2

N

• i=1

αidiscH (Di, DT) + C (δ)

• N
• i=1

α2

i

mi + 4

N

• i=1

αiRi (H, L)

Similar bounds in Ben-David et al., ML 2010; Zhang et al., NIPS 2013.

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 8 / 13

Theory

Bound on the expected loss

Given a hypothesis set H, let:

ˆ hα = argminh∈H ˆ ǫα(h) h∗

T = argminh∈H ǫT(h)

For any δ > 0, with probability at least 1 − δ: |ǫT(ˆ hα) − ǫT(h∗

T)| ≤

2

N

• i=1

αidiscH (Di, DT) + C (δ)

• N
• i=1

α2

i

mi + 4

N

• i=1

αiRi (H, L)

Similar bounds in Ben-David et al., ML 2010; Zhang et al., NIPS 2013.

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 8 / 13

Theory

Bound on the expected loss

Given a hypothesis set H, let:

ˆ hα = argminh∈H ˆ ǫα(h) h∗

T = argminh∈H ǫT(h)

For any δ > 0, with probability at least 1 − δ: |ǫT(ˆ hα) − ǫT(h∗

T)| ≤

2

N

• i=1

αidiscH (Di, DT) + C (δ)

• N
• i=1

α2

i

mi + 4

N

• i=1

αiRi (H, L)

Similar bounds in Ben-David et al., ML 2010; Zhang et al., NIPS 2013.

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 8 / 13

Theory

Bound on the expected loss

Given a hypothesis set H, let:

ˆ hα = argminh∈H ˆ ǫα(h) h∗

T = argminh∈H ǫT(h)

For any δ > 0, with probability at least 1 − δ: |ǫT(ˆ hα) − ǫT(h∗

T)| ≤

2

N

• i=1

αidiscH (Di, DT) + C (δ)

• N
• i=1

α2

i

mi + 4

N

• i=1

αiRi (H, L)

Similar bounds in Ben-David et al., ML 2010; Zhang et al., NIPS 2013.

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 8 / 13

Theory

Algorithm

Theory suggests:

Select α by minimizing:

N

• i=1

αidiscH (Di, DT) + λ

• N
• i=1

α2

i

mi Find ˆ hα by minimizing the α-weighted empirical risk. Choose λ by cross-validation on the reference dataset.

Trade-off between exploiting trusted sources and using all data. In practice, work with the empirical discrepancies:

discH (Si, ST) = sup

h∈H

| 1 mi

mi

• j=1

L

• h
• xi

j

• , y i

j

• −

1 mT

mT

• j=1

L

• h
• xT

j

• , y T

j

• |

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 9 / 13

Theory

Experiments

Evaluate empirically on:

Multitask Dataset of Product Reviews 1. Animals with Attributes 2 2.

Some clean reference data for a target task is available. Have other subsets, some of which are corrupted. Experimented with various manipulations/problems with the data.

1Pentina et al., ICML 2017; McAuley et al., 2015 2Xian et al., TPAMI 2018 Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 10 / 13

Theory

Results

10 20 30 40 50 60

Number of corrupted sources

0.20 0.25 0.30 0.35 0.40

Average classification error

Ours Reference only All data Pregibon et al. Median of probs Feng et al. Yin et al. Batch norm

Figure: Animals with Attributes 2: RGB channels swapped

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 11 / 13

Theory

Summary

Data from different sources is naturally heterogeneous. Our method suppresses the effect of corrupted/irrelevant data. The approach is theoretically justified and shows good empirical performance. The algorithm can be applied even when the data is private and/or distributed.

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 12 / 13

Theory

Summary

Data from different sources is naturally heterogeneous. Our method suppresses the effect of corrupted/irrelevant data. The approach is theoretically justified and shows good empirical performance. The algorithm can be applied even when the data is private and/or distributed.

Thank you for your attention!

Poster 156

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 12 / 13

Theory

Summary

Data from different sources is naturally heterogeneous. Our method suppresses the effect of corrupted/irrelevant data. The approach is theoretically justified and shows good empirical performance. The algorithm can be applied even when the data is private and/or distributed.

Thank you for your attention!

Poster 156

Code available at: https://github.com/NikolaKon1994/Robust-Learning-from-Untrusted-Sources

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 12 / 13

References I

Ben-David, Shai et al. (2010). “A theory of learning from different domains”. In: Machine learning 79.1-2, pp. 151–175. Kifer, Daniel et al. (2004). “Detecting change in data streams”. In: Proceedings of the Thirtieth international conference on Very large data bases-Volume 30. McAuley, Julian et al. (2015). “Image-based recommendations on styles and substitutes”. In: 38th International ACM SIGIR Conference on Research and Development in Information

• Retrieval. ACM.

Mohri, Mehryar et al. (2012). “New analysis and algorithm for learning with drifting distributions”. In: International Conference on Algorithmic Learning Theory. Pentina, Anastasia et al. (2017). “Multi-task Learning with Labeled and Unlabeled Tasks”. In: International Conference on Machine Learning (ICML). Xian, Yongqin et al. (2018). “Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly”. In: IEEE transactions on pattern analysis and machine intelligence.

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 12 / 13

References II

Zhang, Chao et al. (2012). “Generalization bounds for domain adaptation”. In: Advances in neural information processing systems.

Konstantinov, Lampert; IST Austria Robust Learning from Untrusted Sources Poster 156 13 / 13