Sample Complexity and Expressiveness Roi Livni and Yishay Mansour - - PowerPoint PPT Presentation

Graph Based- Discriminators Sample Complexity and Expressiveness

Roi Livni and Yishay Mansour

Discrimination

• A discriminator is provided with

two data sets.

• 𝑇1 ∼ 𝑄

1

• 𝑇2 ∼ 𝑄2
• Decide if 𝑄

1 and 𝑄2 are different.

• If not, provide a certificate.

Motivation: Synthetic Data Generation

Goodfellow et al.’14

https://thispersondoesnotexist.com/

Discrimination: Learning Lens

• A learner is defined by a class 𝐼 ⊆ 0,1 𝑌
• Given labelled sample from some distribution 𝑄 over 𝑌 × 0,1
• Learner returns ℎ ∈ 𝐼 such that

𝑄(𝑦,𝑧) ℎ 𝑦 ≠ 𝑧 ≤ min

ℎ∈𝐼 𝑄(𝑦,𝑧) ℎ 𝑦 ≠ 𝑧 + 𝜗

• If sup

ℎ∈𝐼

𝐹𝑦∼𝑄1 ℎ 𝑦 − 𝐹𝑦∼𝑄2 ℎ 𝑦 > 𝜗

• Learner succeeds.

Learning as a discrimination task

• Discriminator is defined by a class of distinguishers 𝐼 ⊆ 0,1 𝑌

Integral Probability Metric: 𝐽𝑄𝑁𝐼 𝑄

1, 𝑄2 = sup ℎ∈𝐼

|𝐹𝑦∼𝑄1 ℎ 𝑦 − 𝐹𝑦∼𝑄2 ℎ 𝑦 |

• If 𝐽𝑄𝑁𝐼 𝑄

1, 𝑄2 > 𝜗 -- return ℎ ∈ 𝐼 with 𝐽𝑄𝑁𝐼 𝑄 1, 𝑄2 > 𝜗/2

• If not, may fail. (return EQUIVALENT).

(Muller’97)

Higher order discrimination

• Instead of considering hypotheses classes, what if we take other types
• f statistical tests:
• Example: Collision test
• Estimate probability to draw the same point twice. If different–

declare distinct.

• If not, may fail (return equivalent).

Higher order discrimination

• Instead of considering hypotheses classes, what if we take other types
• f distinguishers:
• More generally: Take a family G = {𝑕: 𝑕: 𝑌2 → 0,1 }

𝐽𝑄𝑁𝐻 𝑄

1, 𝑄2 = sup 𝑕∈𝐻

𝐹 𝑦1,𝑦2)∼𝑄1

2 𝑕 𝑦1, 𝑦2

− 𝐹 𝑦1,𝑦2)∼𝑄2

2 𝑕 𝑦1, 𝑦2

• Are graph-based distinguishers stronger than classical distinguishers?
• Sample Complexity?

Expressive power of graph-based discriminators

THEOREM: Let X be an infinite domain. There exists a graph g such that: For every hypothesis class H with finite VC dimension and 𝜗 > 0, there are two distributions 𝑄

𝑡𝑧𝑜, 𝑄𝑠𝑓𝑏𝑚 such that

𝐽𝑄𝑁𝐼 𝑞𝑡𝑧𝑜, 𝑞𝑠𝑓𝑏𝑚 < 𝜗 and, 𝐹(𝑦1,𝑦2)∼𝑞𝑡𝑧𝑜

2

[𝑕 𝑦1,𝑦2 )] − 𝐹(𝑦1,𝑦2)∼𝑞𝑠𝑓𝑏𝑚

2

[𝑕(𝑦1, 𝑦2)] > 1 4 (L, Mansour’19)

Finite Version

• If |X|=N, there is a graph g such that for every class H there are two

distributions that are H-indistinguishable, g-distinguishable unless:

○

𝑊𝐷 𝐼 = Ω(𝜗2 log 𝑂) (L, Mansour’19)

• Optimal: For every graph-based class G with finite capacity there is a

hypothesis class H with VC dimension 𝑃(𝜗2 log 𝑂) such that

𝐽𝑄𝑁𝐷 𝑞𝑡𝑧𝑜, p𝑠𝑓𝑏𝑚 > 1

4 ⇒ 𝐽𝑄𝑁𝐻 𝑞𝑡𝑧𝑜, p𝑠𝑓𝑏𝑚 > 𝜗

(Alon, L, Mansour)

○

Given a graph g how many sets are needed to separate every dense set from every sparse set?

Sample complexity of graph-based discriminators

• For a family of graph G.
• Given samples from two unknown distributions 𝑄

1, 𝑄2: Decide if

• How many examples are needed?
• Recall:

○

For an hypothesis class, a discriminator can decide if 𝐽𝑄𝑁𝐼 𝑄

1, 𝑄2 > 𝜗, if and

• nly if H has finite VC dimension.

○

Θ 𝑊𝐷 𝐼 /𝜗2 are needed

𝐽𝑄𝑁𝐻 𝑄

1, 𝑄2 > 𝜗

The graph-VC dimension

• The graph VC dimension is obtained by considering the projections of the

graph by fixing a vertex. Namely, for every x consider the hypothesis class

𝐼𝑦 = 𝑕 𝑦,⋅ : 𝑌 → 0,1 : 𝑕 ∈ 𝐻

• Then: 𝑕𝑊𝐷 𝐷 = sup

𝑦∈𝑌

𝑊𝐷(𝐼𝑦)

• 𝑃(𝑕𝑊𝐷 𝐷 ) are sufficient.
• Ω( 𝑕𝑊𝐷 𝐷 ) are necessary.

(L, Mansour’19)