Nearly Tight Bounds for Robust Proper Learning of Halfspaces with a - PowerPoint PPT Presentation

Nearly Tight Bounds for Robust Proper Learning of Halfspaces with a Margin Ilias Diakonikolas Daniel M. Kane Pasin Manurangsi UW Madison UC San Diego Google Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi

Agnostic Proper Learning of Halfspaces Input + + - - + - Labeled samples (x 1 , y 1 ), (x 2 , y 2 ), … ∈ 𝓒 (d) × {±1} + from distribution 𝓔 + - + - - - + - - - + - Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi

Agnostic Proper Learning of Halfspaces Input + + - - + - Labeled samples (x 1 , y 1 ), (x 2 , y 2 ), … ∈ 𝓒 (d) × {±1} + from distribution 𝓔 + - Positive real number ε - + - - - + - - - + - Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi

Agnostic Proper Learning of Halfspaces Input + + - - + - Labeled samples (x 1 , y 1 ), (x 2 , y 2 ), … ∈ 𝓒 (d) × {±1} + from distribution 𝓔 + - Positive real number ε - + Output A halfspace w with “small” classifjcation error - - - + - - - + - Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi

Agnostic Proper Learning of Halfspaces Input + + - - + - Labeled samples (x 1 , y 1 ), (x 2 , y 2 ), … ∈ 𝓒 (d) × {±1} + from distribution 𝓔 + - Positive real number ε - + Output A halfspace w with “small” classifjcation error - - - + - - - + - OPT = Min classifjcation error among all halfspaces = min w Pr (x, y)~ 𝓔 [<w, x> ・ y ＜ 0] Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi

Agnostic Proper Learning of Halfspaces w* Input + + - - + - Labeled samples (x 1 , y 1 ), (x 2 , y 2 ), … ∈ 𝓒 (d) × {±1} + from distribution 𝓔 + - Positive real number ε - + Output A halfspace w with “small” classifjcation error - - - + - - - + - OPT = Min classifjcation error among all halfspaces = min w Pr (x, y)~ 𝓔 [<w, x> ・ y ＜ 0] Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi

Agnostic Proper Learning of Halfspaces w* Input + + - - + - Labeled samples (x 1 , y 1 ), (x 2 , y 2 ), … ∈ 𝓒 (d) × {±1} + from distribution 𝓔 + - Positive real number ε - + Output A halfspace w with “small” classifjcation error - - - + - - - + - OPT = Min classifjcation error among all halfspaces = min w Pr (x, y)~ 𝓔 [<w, x> ・ y ＜ 0] Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi

Agnostic Proper Learning of Halfspaces w* Input + + - - + - Labeled samples (x 1 , y 1 ), (x 2 , y 2 ), … ∈ 𝓒 (d) × {±1} + from distribution 𝓔 + - Positive real number ε - + Output A halfspace w with “small” classifjcation error - - - + - An algorithm is a 𝛽 -learner if it outputs w with - classifjcation error at most 𝛽 ・ OPT + ε - + - OPT = Min classifjcation error among all halfspaces = min w Pr (x, y)~ 𝓔 [<w, x> ・ y ＜ 0] Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi

Agnostic Proper Learning of Halfspaces w* Input + w + - - + - Labeled samples (x 1 , y 1 ), (x 2 , y 2 ), … ∈ 𝓒 (d) × {±1} + from distribution 𝓔 + - Positive real number ε - + Output A halfspace w with “small” classifjcation error - - - + - An algorithm is a 𝛽 -learner if it outputs w with - classifjcation error at most 𝛽 ・ OPT + ε - + - OPT = Min classifjcation error among all halfspaces = min w Pr (x, y)~ 𝓔 [<w, x> ・ y ＜ 0] Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi

Agnostic Proper Learning of Halfspaces w* Input + w + - - + - Labeled samples (x 1 , y 1 ), (x 2 , y 2 ), … ∈ 𝓒 (d) × {±1} + from distribution 𝓔 + - Positive real number ε - + Output A halfspace w with “small” classifjcation error - - - + - An algorithm is a 𝛽 -learner if it outputs w with - classifjcation error at most 𝛽 ・ OPT + ε - + - OPT = Min classifjcation error among all halfspaces = min w Pr (x, y)~ 𝓔 [<w, x> ・ y ＜ 0] Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi

Agnostic Proper Learning of Halfspaces w* Input + w + - - + - Labeled samples (x 1 , y 1 ), (x 2 , y 2 ), … ∈ 𝓒 (d) × {±1} + from distribution 𝓔 + - Positive real number ε - + Output A halfspace w with “small” classifjcation error - - - + - An algorithm is a 𝛽 -learner if it outputs w with - classifjcation error at most 𝛽 ・ OPT + ε - + - Bad news: [Arora et al.’97] Unless NP = RP, no poly-time 𝛽 -learner for all constants 𝛽 . OPT = Min classifjcation error among all halfspaces = min w Pr (x, y)~ 𝓔 [<w, x> ・ y ＜ 0] [Guruswami-Raghavendra’ 06, Feldman et al.’06] Even weak learning is NP-hard. Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi

Agnostic Proper Learning of Halfspaces with a Margin Input + + - - + - Labeled samples (x 1 , y 1 ), (x 2 , y 2 ), … ∈ 𝓒 (d) × {±1} + from distribution 𝓔 + - Positive real number ε - + Output A halfspace w with “small” classifjcation error - - - + - - - + - Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi

Agnostic Proper Learning of Halfspaces with a Margin w* Input + + - - + - Labeled samples (x 1 , y 1 ), (x 2 , y 2 ), … ∈ 𝓒 (d) × {±1} + from distribution 𝓔 + - Positive real number ε - + Output A halfspace w with “small” classifjcation error - - - + - - - + - OPT 𝛿 = Min 𝛿 -margin error among all halfspaces = min w Pr (x, y)~ 𝓔 [<w, x> ・ y ＜ 𝛿 ] Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi

Agnostic Proper Learning of Halfspaces with a Margin w* Input + + - - + - Labeled samples (x 1 , y 1 ), (x 2 , y 2 ), … ∈ 𝓒 (d) × {±1} + from distribution 𝓔 + - Positive real number ε - + Output A halfspace w with “small” classifjcation error - - - + - - - + - OPT 𝛿 = Min 𝛿 -margin error among all halfspaces = min w Pr (x, y)~ 𝓔 [<w, x> ・ y ＜ 𝛿 ] Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi

Agnostic Proper Learning of Halfspaces with a Margin w* Input + + - - + - Labeled samples (x 1 , y 1 ), (x 2 , y 2 ), … ∈ 𝓒 (d) × {±1} + from distribution 𝓔 + - Positive real number ε - 𝛿 𝛿 + Output A halfspace w with “small” classifjcation error - - - + - - - + - OPT 𝛿 = Min 𝛿 -margin error among all halfspaces = min w Pr (x, y)~ 𝓔 [<w, x> ・ y ＜ 𝛿 ] Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi

Agnostic Proper Learning of Halfspaces with a Margin w* Input + + - - + - Labeled samples (x 1 , y 1 ), (x 2 , y 2 ), … ∈ 𝓒 (d) × {±1} + from distribution 𝓔 + - Positive real number ε - 𝛿 𝛿 + Output A halfspace w with “small” classifjcation error - - - + - An algorithm is a 𝛽 -learner if it outputs w with - classifjcation error at most 𝛽 ・ OPT 𝛿 + ε - + - OPT 𝛿 = Min 𝛿 -margin error among all halfspaces = min w Pr (x, y)~ 𝓔 [<w, x> ・ y ＜ 𝛿 ] Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi

Agnostic Proper Learning of Halfspaces with a Margin w* Input + + - - + - Labeled samples (x 1 , y 1 ), (x 2 , y 2 ), … ∈ 𝓒 (d) × {±1} + from distribution 𝓔 + - Positive real number ε - 𝛿 𝛿 + Output A halfspace w with “small” classifjcation error - - - + - An algorithm is a 𝛽 -learner if it outputs w with - classifjcation error at most 𝛽 ・ OPT 𝛿 + ε - + - Margin Assumption OPT 𝛿 = Min 𝛿 -margin error among all halfspaces - “Robustness” of the optimal halfspace to ℓ 2 noise = min w Pr (x, y)~ 𝓔 [<w, x> ・ y ＜ 𝛿 ] Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi

Agnostic Proper Learning of Halfspaces with a Margin w* Input + + - - + - Labeled samples (x 1 , y 1 ), (x 2 , y 2 ), … ∈ 𝓒 (d) × {±1} + from distribution 𝓔 + - Positive real number ε - 𝛿 𝛿 + Output A halfspace w with “small” classifjcation error - - - + - An algorithm is a 𝛽 -learner if it outputs w with - classifjcation error at most 𝛽 ・ OPT 𝛿 + ε - + - Margin Assumption OPT 𝛿 = Min 𝛿 -margin error among all halfspaces - “Robustness” of the optimal halfspace to ℓ 2 noise = min w Pr (x, y)~ 𝓔 [<w, x> ・ y ＜ 𝛿 ] - Variants used in Perceptron, SVMs Nearly Tight Bound for Robust Proper Learning of Halfspaces with a Margin Diakonikolas, Kane, Manurangsi