Sample Complexity and Expressiveness Roi Livni and Yishay Mansour - PowerPoint PPT Presentation

Graph Based- Discriminators Sample Complexity and Expressiveness Roi Livni and Yishay Mansour

Discrimination • A discriminator is provided with two data sets. • 𝑇 1 ∼ 𝑄 1 • 𝑇 2 ∼ 𝑄 2 • Decide if 𝑄 1 and 𝑄 2 are different. • If not, provide a certificate.

Motivation: Synthetic Data Generation Goodfellow et al. ’ 14 https://thispersondoesnotexist.com/

Discrimination: Learning Lens • A learner is defined by a class 𝐼 ⊆ 0,1 𝑌 • Given labelled sample from some distribution 𝑄 over 𝑌 × 0,1 • Learner returns ℎ ∈ 𝐼 such that 𝑄 (𝑦,𝑧) ℎ 𝑦 ≠ 𝑧 ≤ min ℎ∈𝐼 𝑄 (𝑦,𝑧) ℎ 𝑦 ≠ 𝑧 + 𝜗 • If sup 𝐹 𝑦∼𝑄 1 ℎ 𝑦 − 𝐹 𝑦∼𝑄 2 ℎ 𝑦 > 𝜗 ℎ∈𝐼 • Learner succeeds.

Learning as a discrimination task • Discriminator is defined by a class of distinguishers 𝐼 ⊆ 0,1 𝑌 Integral Probability Metric: (Muller ’ 97) 𝐽𝑄𝑁 𝐼 𝑄 1 , 𝑄 2 = sup |𝐹 𝑦∼𝑄 1 ℎ 𝑦 − 𝐹 𝑦∼𝑄 2 ℎ 𝑦 | ℎ∈𝐼 • If 𝐽𝑄𝑁 𝐼 𝑄 1 , 𝑄 2 > 𝜗 -- return ℎ ∈ 𝐼 with 𝐽𝑄𝑁 𝐼 𝑄 1 , 𝑄 2 > 𝜗/2 • If not, may fail. (return EQUIVALENT).

Higher order discrimination • Instead of considering hypotheses classes, what if we take other types of statistical tests: • Example: Collision test • Estimate probability to draw the same point twice. If different – declare distinct. • If not, may fail (return equivalent).

Higher order discrimination • Instead of considering hypotheses classes, what if we take other types of distinguishers: • More generally: Take a family G = {𝑕: 𝑕: 𝑌 2 → 0,1 } 𝐽𝑄𝑁 𝐻 𝑄 1 , 𝑄 2 = sup 𝐹 𝑦 1, 𝑦 2 )∼𝑄 1 2 𝑕 𝑦 1 , 𝑦 2 − 𝐹 𝑦 1, 𝑦 2 )∼𝑄 2 2 𝑕 𝑦 1 , 𝑦 2 𝑕∈𝐻 • Are graph-based distinguishers stronger than classical distinguishers? • Sample Complexity?

Expressive power of graph-based discriminators THEOREM: Let X be an infinite domain. There exists a graph g such that: For every hypothesis class H with finite VC dimension and 𝜗 > 0 , there are two distributions 𝑄 𝑡𝑧𝑜 , 𝑄 𝑠𝑓𝑏𝑚 such that 𝐽𝑄𝑁 𝐼 𝑞 𝑡𝑧𝑜 , 𝑞 𝑠𝑓𝑏𝑚 < 𝜗 and, [𝑕(𝑦 1 , 𝑦 2 )] > 1 𝐹 (𝑦 1 ,𝑦 2 )∼𝑞 𝑡𝑧𝑜 [𝑕 𝑦 1 ,𝑦 2 )] − 𝐹 (𝑦 1 ,𝑦 2 )∼𝑞 𝑠𝑓𝑏𝑚 2 2 4 (L, Mansour ’ 19)

Finite Version If |X|=N, there is a graph g such that for every class H there are two ● distributions that are H-indistinguishable, g-distinguishable unless: 𝑊𝐷 𝐼 = Ω(𝜗 2 log 𝑂) (L, Mansour ’ 19) ○ Optimal : For every graph-based class G with finite capacity there is a ● hypothesis class H with VC dimension 𝑃(𝜗 2 log 𝑂) such that 𝐽𝑄𝑁 𝐷 𝑞 𝑡𝑧𝑜 , p 𝑠𝑓𝑏𝑚 > 1 4 ⇒ 𝐽𝑄𝑁 𝐻 𝑞 𝑡𝑧𝑜 , p 𝑠𝑓𝑏𝑚 > 𝜗 (Alon, L, Mansour) Given a graph g how many sets are needed to separate every dense set from every ○ sparse set?

Sample complexity of graph-based discriminators For a family of graph G. ● Given samples from two unknown distributions 𝑄 1 , 𝑄 2 : Decide if ● 𝐽𝑄𝑁 𝐻 𝑄 1 , 𝑄 2 > 𝜗 How many examples are needed? ● Recall: ● For an hypothesis class, a discriminator can decide if 𝐽𝑄𝑁 𝐼 𝑄 1 , 𝑄 2 > 𝜗 , if and ○ only if H has finite VC dimension. Θ 𝑊𝐷 𝐼 /𝜗 2 are needed ○

The graph-VC dimension The graph VC dimension is obtained by considering the projections of the ● graph by fixing a vertex. Namely, for every x consider the hypothesis class 𝐼 𝑦 = 𝑕 𝑦,⋅ : 𝑌 → 0,1 : 𝑕 ∈ 𝐻 Then: 𝑕𝑊𝐷 𝐷 = sup 𝑊𝐷(𝐼 𝑦 ) ● 𝑦∈𝑌 𝑃(𝑕𝑊𝐷 𝐷 ) are sufficient. ● Ω( 𝑕𝑊𝐷 𝐷 ) are necessary. ● (L, Mansour ’ 19)