Nearly Tight Bounds for Robust Proper Learning of Halfspaces with a - - PowerPoint PPT Presentation
Nearly Tight Bounds for Robust Proper Learning of Halfspaces with a Margin Ilias Diakonikolas Daniel M. Kane Pasin Manurangsi UW Madison UC San Diego Google Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
from distribution 𝓔 +
+ + + +
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
from distribution 𝓔 +
+ + + +
+
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
from distribution 𝓔
A halfspace w with “small” classifjcation error +
+ + + +
+
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
from distribution 𝓔
A halfspace w with “small” classifjcation error +
+ + + +
= minw Pr(x, y)~𝓔 [<w, x>・y ＜ 0]
+
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
from distribution 𝓔
A halfspace w with “small” classifjcation error +
+ + + +
= minw Pr(x, y)~𝓔 [<w, x>・y ＜ 0]
+ w*
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
from distribution 𝓔
A halfspace w with “small” classifjcation error + +
+ + + +
= minw Pr(x, y)~𝓔 [<w, x>・y ＜ 0]
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
from distribution 𝓔
A halfspace w with “small” classifjcation error + +
+ + + +
= minw Pr(x, y)~𝓔 [<w, x>・y ＜ 0]
classifjcation error at most 𝛽・OPT + ε
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
from distribution 𝓔
A halfspace w with “small” classifjcation error + +
+ + + +
= minw Pr(x, y)~𝓔 [<w, x>・y ＜ 0]
classifjcation error at most 𝛽・OPT + ε
w
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
from distribution 𝓔
A halfspace w with “small” classifjcation error + +
+ + + +
= minw Pr(x, y)~𝓔 [<w, x>・y ＜ 0]
classifjcation error at most 𝛽・OPT + ε
w
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
from distribution 𝓔
A halfspace w with “small” classifjcation error + +
+ + + +
= minw Pr(x, y)~𝓔 [<w, x>・y ＜ 0]
classifjcation error at most 𝛽・OPT + ε
[Arora et al.’97] Unless NP = RP, no poly-time 𝛽-learner for all constants 𝛽.
w* w
[Guruswami-Raghavendra’ 06, Feldman et al.’06] Even weak learning is NP-hard.
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
from distribution 𝓔
A halfspace w with “small” classifjcation error +
+ + + +
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
from distribution 𝓔
A halfspace w with “small” classifjcation error +
+ + + +
= minw Pr(x, y)~𝓔 [<w, x>・y ＜ 𝛿]
w*
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
from distribution 𝓔
A halfspace w with “small” classifjcation error +
+ + + +
= minw Pr(x, y)~𝓔 [<w, x>・y ＜ 𝛿]
w*
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
from distribution 𝓔
A halfspace w with “small” classifjcation error +
+ + + +
= minw Pr(x, y)~𝓔 [<w, x>・y ＜ 𝛿]
𝛿 𝛿 w*
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
from distribution 𝓔
A halfspace w with “small” classifjcation error OPT𝛿 = Min 𝛿-margin error among all halfspaces = minw Pr(x, y)~𝓔 [<w, x>・y ＜ 𝛿]
An algorithm is a 𝛽-learner if it outputs w with classifjcation error at most 𝛽・OPT𝛿 + ε +
+ + + +
𝛿 𝛿 w*
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
from distribution 𝓔
A halfspace w with “small” classifjcation error OPT𝛿 = Min 𝛿-margin error among all halfspaces = minw Pr(x, y)~𝓔 [<w, x>・y ＜ 𝛿]
An algorithm is a 𝛽-learner if it outputs w with classifjcation error at most 𝛽・OPT𝛿 + ε
halfspace to ℓ2 noise +
+ + + +
𝛿 𝛿 w*
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
from distribution 𝓔
A halfspace w with “small” classifjcation error OPT𝛿 = Min 𝛿-margin error among all halfspaces = minw Pr(x, y)~𝓔 [<w, x>・y ＜ 𝛿]
An algorithm is a 𝛽-learner if it outputs w with classifjcation error at most 𝛽・OPT𝛿 + ε
halfspace to ℓ2 noise
+
+ + + +
𝛿 𝛿 w*
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
[Shalev-Shwaruz, Shamir & Sridharan’09]
improper 1-learner that runs in poly(d/ε)・exp(Õ(1/𝛿)) time, takes poly(d/ε)・exp(Õ(1/𝛿)) samples [Ben-David & Simon’00] proper 1-learner that runs in poly(d)・exp(Õ(log(1/ε)/𝛿2)) time, takes O(1/ε2𝛿2) samples
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
[Shalev-Shwaruz, Shamir & Sridharan’09]
improper 1-learner that runs in poly(d/ε)・exp(Õ(1/𝛿)) time, takes poly(d/ε)・exp(Õ(1/𝛿)) samples
Approximation ratio: 𝛽 = 1
[Ben-David & Simon’00] proper 1-learner that runs in poly(d)・exp(Õ(log(1/ε)/𝛿2)) time, takes O(1/ε2𝛿2) samples
Approximation ratio: 𝛽 = 1
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
[Shalev-Shwaruz, Shamir & Sridharan’09]
improper 1-learner that runs in poly(d/ε)・exp(Õ(1/𝛿)) time, takes poly(d/ε)・exp(Õ(1/𝛿)) samples
Approximation ratio: 𝛽 = 1 Approximation ratio: 𝛽 = 1
[Ben-David & Simon’00] proper 1-learner that runs in poly(d)・exp(Õ(log(1/ε)/𝛿2)) time, takes O(1/ε2𝛿2) samples
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
[Shalev-Shwaruz, Shamir & Sridharan’09]
improper 1-learner that runs in poly(d/ε)・exp(Õ(1/𝛿)) time takes poly(d/ε)・exp(Õ(1/𝛿)) samples
Approximation ratio: 𝛽 = 1 Approximation ratio: 𝛽 = 1
[Ben-David & Simon’00] proper 1-learner that runs in poly(d)・exp(Õ(log(1/ε)/𝛿2)) time, takes O(1/ε2𝛿2) samples
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
[Shalev-Shwaruz, Shamir & Sridharan’09]
improper 1-learner that runs in poly(d/ε)・exp(Õ(1/𝛿)) time takes poly(d/ε)・exp(Õ(1/𝛿)) samples
Approximation ratio: 𝛽 = 1
Output hypothesis is not a halfspace
Approximation ratio: 𝛽 = 1
[Ben-David & Simon’00] proper 1-learner that runs in poly(d)・exp(Õ(log(1/ε)/𝛿2)) time, takes O(1/ε2𝛿2) samples
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
[Shalev-Shwaruz, Shamir & Sridharan’09]
improper 1-learner that runs in poly(d/ε)・exp(Õ(1/𝛿)) time takes poly(d/ε)・exp(Õ(1/𝛿)) samples
Approximation ratio: 𝛽 = 1
Output hypothesis is not a halfspace
Approximation ratio: 𝛽 = 1
[Ben-David & Simon’00] proper 1-learner that runs in poly(d)・exp(Õ(log(1/ε)/𝛿2)) time, takes O(1/ε2𝛿2) samples
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
[Shalev-Shwaruz, Shamir & Sridharan’09]
improper 1-learner that runs in poly(d/ε)・exp(Õ(1/𝛿)) time takes poly(d/ε)・exp(Õ(1/𝛿)) samples
Approximation ratio: 𝛽 = 1
Output hypothesis is not a halfspace Theorem 1 proper 1.01-learner that runs in poly(d/ε)・exp(Õ(1/𝛿2))-time takes O(1/ε2𝛿2) samples
Approximation ratio: 𝛽 = 1
[Ben-David & Simon’00] proper 1-learner that runs in poly(d)・exp(Õ(log(1/ε)/𝛿2)) time, takes O(1/ε2𝛿2) samples
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
[Shalev-Shwaruz, Shamir & Sridharan’09]
improper 1-learner that runs in poly(d/ε)・exp(Õ(1/𝛿)) time takes poly(d/ε)・exp(Õ(1/𝛿)) samples
Approximation ratio: 𝛽 = 1
Output hypothesis is not a halfspace Theorem 1 proper 1.01-learner that runs in poly(d/ε)・exp(Õ(1/𝛿2))-time takes O(1/ε2𝛿2) samples
Approximation ratio: any 𝛽 > 1 Approximation ratio: 𝛽 = 1
[Ben-David & Simon’00] proper 1-learner that runs in poly(d)・exp(Õ(log(1/ε)/𝛿2)) time, takes O(1/ε2𝛿2) samples
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
[Shalev-Shwaruz, Shamir & Sridharan’09]
improper 1-learner that runs in poly(d/ε)・exp(Õ(1/𝛿)) time takes poly(d/ε)・exp(Õ(1/𝛿)) samples
Approximation ratio: 𝛽 = 1
Output hypothesis is not a halfspace Theorem 1 proper 1.01-learner that runs in poly(d/ε)・exp(Õ(1/𝛿2))-time takes O(1/ε2𝛿2) samples
Approximation ratio: any 𝛽 > 1 Approximation ratio: 𝛽 = 1
[Ben-David & Simon’00] proper 1-learner that runs in poly(d)・exp(Õ(log(1/ε)/𝛿2)) time, takes O(1/ε2𝛿2) samples
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
[Shalev-Shwaruz, Shamir & Sridharan’09]
improper 1-learner that runs in poly(d/ε)・exp(Õ(1/𝛿)) time takes poly(d/ε)・exp(Õ(1/𝛿)) samples
Approximation ratio: 𝛽 = 1
Output hypothesis is not a halfspace Theorem 1 proper 1.01-learner that runs in poly(d/ε)・exp(Õ(1/𝛿2))-time takes O(1/ε2𝛿2) samples
Approximation ratio: any 𝛽 > 1 Approximation ratio: 𝛽 = 1
[Ben-David & Simon’00] proper 1-learner that runs in poly(d)・exp(Õ(log(1/ε)/𝛿2)) time, takes O(1/ε2𝛿2) samples
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
[Shalev-Shwaruz, Shamir & Sridharan’09]
improper 1-learner that runs in poly(d/ε)・exp(Õ(1/𝛿)) time takes poly(d/ε)・exp(Õ(1/𝛿)) samples
Approximation ratio: 𝛽 = 1
Output hypothesis is not a halfspace Theorem 1 proper 1.01-learner that runs in poly(d/ε)・exp(Õ(1/𝛿2))-time takes O(1/ε2𝛿2) samples
Approximation ratio: any 𝛽 > 1 Approximation ratio: 𝛽 = 1
Theorem 2 Assuming Exponential Time Hypothesis, for any constant 𝛽 > 1, no proper 𝛽-learner runs in poly(d/ε)・exp(O(1/𝛿2-o(1))) time [Ben-David & Simon’00] proper 1-learner that runs in poly(d)・exp(Õ(log(1/ε)/𝛿2)) time, takes O(1/ε2𝛿2) samples
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
[Shalev-Shwaruz, Shamir & Sridharan’09]
improper 1-learner that runs in poly(d/ε)・exp(Õ(1/𝛿)) time takes poly(d/ε)・exp(Õ(1/𝛿)) samples
Approximation ratio: 𝛽 = 1
Output hypothesis is not a halfspace Theorem 1 proper 1.01-learner that runs in poly(d/ε)・exp(Õ(1/𝛿2))-time takes O(1/ε2𝛿2) samples
Approximation ratio: any 𝛽 > 1 Approximation ratio: 𝛽 = 1
Theorem 2 Assuming Exponential Time Hypothesis, for any constant 𝛽 > 1, no proper 𝛽-learner runs in poly(d/ε)・exp(O(1/𝛿2-o(1))) time [Ben-David & Simon’00] proper 1-learner that runs in poly(d)・exp(Õ(log(1/ε)/𝛿2)) time, takes O(1/ε2𝛿2) samples
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
[Shalev-Shwaruz, Shamir & Sridharan’09]
improper 1-learner that runs in poly(d/ε)・exp(Õ(1/𝛿)) time takes poly(d/ε)・exp(Õ(1/𝛿)) samples
Approximation ratio: 𝛽 = 1
Output hypothesis is not a halfspace Theorem 1 proper 1.01-learner that runs in poly(d/ε)・exp(Õ(1/𝛿2))-time takes O(1/ε2𝛿2) samples
Approximation ratio: any 𝛽 > 1
Theorem 2 Assuming Exponential Time Hypothesis, for any constant 𝛽 > 1, no proper 𝛽-learner runs in poly(d/ε)・exp(O(1/𝛿2-o(1))) time
Approximation ratio: 𝛽 = 1
Theorem 3 Assuming W[1] ≠ FPT, for any function f, no proper 1-learner runs in poly(d/ε)・f(1/𝛿) time
Approximation ratio: 𝛽 = 1
[Ben-David & Simon’00] proper 1-learner that runs in poly(d)・exp(Õ(log(1/ε)/𝛿2)) time, takes O(1/ε2𝛿2) samples
Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi
[Shalev-Shwaruz, Shamir & Sridharan’09]
improper 1-learner that runs in poly(d/ε)・exp(Õ(1/𝛿)) time takes poly(d/ε)・exp(Õ(1/𝛿)) samples
Approximation ratio: 𝛽 = 1
Output hypothesis is not a halfspace Theorem 1 proper 1.01-learner that runs in poly(d/ε)・exp(Õ(1/𝛿2))-time takes O(1/ε2𝛿2) samples
Approximation ratio: any 𝛽 > 1
Theorem 2 Assuming Exponential Time Hypothesis, for any constant 𝛽 > 1, no proper 𝛽-learner runs in poly(d/ε)・exp(O(1/𝛿2-o(1))) time
Approximation ratio: 𝛽 = 1
Theorem 3 Assuming W[1] ≠ FPT, for any function f, no proper 1-learner runs in poly(d/ε)・f(1/𝛿) time
Approximation ratio: 𝛽 = 1
Also results for large approximation ratio 𝛽 [Ben-David & Simon’00] proper 1-learner that runs in poly(d)・exp(Õ(log(1/ε)/𝛿2)) time, takes O(1/ε2𝛿2) samples