Learning Algorithms from Natural Lower Bounds CCC 2016 Marco - - PowerPoint PPT Presentation

Learning Algorithms from Natural Lower Bounds

CCC 2016 Marco Carmosino (UCSD) Russell Impagliazzo (UCSD) Valentine Kabanets (SFU) Antonina Kolokolova (MUN) June 11, 2016

Natural Proof of Circuit Lower Bounds for C

⇓

Learning Algorithm for C

Natural Proof

?

= Barrier However: Natural Proof = Algorithm

Not A Surprise As long as we use natural proofs we have to cope with a duality: any lower bound proof must implicitly argue a proportionately strong upper bound. – Razborov, Rudich 1997 With this duality in mind, it is no coincidence that the technical lemmas of [H˚ as87, Smo87, Raz87] yield much of the machinery for the learning result of [LMN93]. – Razborov, Rudich 1997

Theorem (Main Application of this work)

There is a quasi-polynomial time membership-query learning algorithm for AC0[p] Open Since:

Theorem ([LMN93])

There is a quasi-polynomial time uniform-sample learning algorithm for AC0

Algorithms Lower Bounds

Reminder: Circuits

Complexity Petting Zoo

An AC0 Circuit

Circuit Zoo P/poly Unrestricted poly-size circuits ⊆ TC0 Add MAJORITY gates ⊆ ACC0 Add counting modulo any m ⊆ AC0[p] Add counting modulo prime p ⊆ AC0 AND, OR, NOT, constant-depth, poly-size

Circuit Lower Bounds

The Final Frontier

P/poly ??!?!?!?!?!!?? ⊆ TC0 ???? ⊆ ACC0 ⊃ NEXP [Wil11] ⊆ AC0[p] ACC0 [Raz87, Smo87] ⊆ AC0 AC0[2] [FSS81]

Reminder: Pseudorandom Generators

Reminder: Natural Proofs

Natural Lower Bounds Against C

Razborov Rudich 97

Natural Proofs embed an efficient algorithm for some problem R: input: (truth table of) f : {0, 1}n → {0, 1} task: Distinguish: ∀f of C-circuit complexity ≤ u(n), f ∈ R versus Pr[random f ∈ R] > 1/5

= Function on n bits

Prior Work

Prior Work

C-Lower Bounds to Algorithms

C-Meta-Algorithms

(SAT, Learning, Compression)

⇑

◮ Formula-SAT [San10] ◮ AC0-SAT [IMP12] ◮ AC0-Learning [LMN93] ◮ AC0-Compression [CKK+15] ◮ AC0[p]-Compression [Sri15]

Prior Work

C-Algorithms to Lower Bounds

◮ fast C-LEARN =

⇒ EXP C [FK09, HH11, KKO13]

◮ fast C-SAT =

⇒ NEXP C [Wil10]

◮ NEXP ACC

[Wil11]

◮ C-Compression =

⇒ NEXP C [CKK+15]

(SAT, Learning, Compression)

⇓

C-Lower Bounds

Prior Work: The Pattern C-Meta-Algorithms

(SAT, Learning, Compression)

⇑

(SAT, Learning, Compression)

⇓

C-Lower Bounds

C-Meta-Algorithms

(SAT? Learning? Compression?)

?⇑?

???

Randomized C-Learning Algorithm

⇑

(for “powerful enough” circuit classes C)

(C closed under polysize AC0[p]-reductions)

C-Learning f

Learning Model: Membership Queries: may query f (x) for any x Uniform PAC: find circuit H s.t. Prx[H(x) = f (x)] ≈ 1

Razborov Rudich 97

Natural Lower Bound Against C

⇓

Inverting Algorithm for every g ∈ C

MESSAGE: No Natural Lower Bounds for C ⊇ TC0

This Work

Natural Lower Bound Against C

⇓

Learning Algorithm for every g ∈ C

MESSAGE: Hmmm.

⇓

Natural Lower Bound for AC0[p] [RS]

⇓

Quasi-Polytime Learning Algorithm for AC0[p]

Learning Algorithms

Compare & Contrast

[LMN] Algorithm for AC0

◮ Uniform PAC Model ◮ npoly(log n) runtime ◮ Switching Lemma

(an exp(n1/Ω(d)) lower bound for depth d)

Our Algorithm for AC0[p]

◮ Membership queries ◮ npoly(log n) runtime ◮ RS Lower Bound

(an exp(n1/Ω(d)) lower bound for depth d)

The Proof

Tools

◮ NW Generator ◮ XOR Lemma ◮ Natural Properties

Notation

Agree-o-Meter

Let f : {0, 1}n → {0, 1}. Define:

Hardness

Hardness

Very Hard Somewhat Hard

(not to scale)

NW Generator

Definition

NW Generator

Definition

(stretch)

(seed)

NW Generator

Definition

NW Generator

Definition

Almost Disjoint Subset Multiplexer

NW Generator

Definition

Almost Disjoint Subset Multiplexer

NW Generator

THEOREM: If h is a very hard function, then NWh(s) is a PRG. PROOF: Assume NWh(s) is NOT a PRG. Then ∃D:

NW Generator

Using D:

Yao’s XOR Lemma

h⊕k( x1, . . . , xk) = h( x1) ⊕ · · · ⊕ h( xk) THEOREM: If h is a somewhat hard function, then h⊕k is a very hard function. PROOF: Assume h⊕k is NOT very hard. Then ∃C:

Yao’s XOR Lemma

Proof

Using C:

Hardness to Randomness

Theorem: if h is a somewhat hard function, then NWh⊕k(s) is a PRG. Proof: Assume NWh⊕k(s) is NOT a PRG. Then compose the previous two proofs!

Main Proof Idea: Play to Lose PRG: ⊥ = victory Learning: circuit = victory

Learning Algorithm

Idea: if f ∈ C...

Figure 1: Composed arguments for easy f

What’s Missing?

CONDITION: ∀f ∈ C ∃D a UNIFORM circuit such that...

Figure 2: The circuit we need to learn f

NATURAL PROPERTY R vs C: ∀f ′ ∈ C[u]

COMPARE

Figure 3: HAVE vs WANT

. . . . . . . . . . . . . . . . . .

Key Lemma f ∈ C[poly] = ⇒ ∀s gs ∈ C[exp(ℓ−1)] Requirement: gs ∈ R size(gs) ≤ usefulness(log(ℓ))

Proof of Key Lemma:

Almost Disjoint Subset Multiplexer

Runtime Analysis

Learning Runtime = poly(ℓ)

(Recall, ℓ is the stretch of our PRG)

= Function on n bits

???

???

???

???

???

Constraints

• 1. Natural Property:

C-size(gs) ≤ u(log(ℓ))

recall: f ∈ C[poly] = ⇒ ∀s gs ∈ C[exp(ℓ−1)]

• 2. Runtime (want to minimize):

poly(ℓ)

Learning Algorithm Runtime

Usefulness vs. Size

Usefulness

◮ Exponential 2Ω(n) ◮ Subexponential 2nǫ ◮ Superpolynomial nω(1)

Learning Runtime

◮ Polynomial ◮ Quasipolynomial ◮ Subexponential

Poly(n) lb

2𝑜 lb

Exp runtime

Poly runtime

Randomized C-Learning Algorithm

⇑

(for “powerful enough” circuit classes C)

(C closed under polysize AC0[p]-reductions)

Main Application

Learning AC0[p] In quasipolynomial time quasi-poly

Summary

◮ Natural Proofs =

⇒ Learning (and Compression) Algorithms

◮ BPP-Natural Proofs for C ⇐

⇒ Randomized Compression for C

(answers open question of [CKKSZ15]

◮ Randomized quasi-polytime learning algorithm

for AC0[p] for any prime p

Future Work

◮ Natural Proof of Williams’ ACC0 lower bound? ◮ Natural Proofs for C

?

= ⇒ C-SAT Algorithms

◮ Other learning models? ◮ Banish serendipity? ◮ Derandomization? ◮ Limits to “grey-box” algorithm design?

Ruiwen Chen, Valentine Kabanets, Antonina Kolokolova, Ronen Shaltiel, and David Zuckerman. Mining circuit lower bound proofs for meta-algorithms. Computational Complexity, 24(2):333–392, 2015. Lance Fortnow and Adam R. Klivans. Efficient learning algorithms yield circuit lower bounds.

• J. Comput. Syst. Sci., 75(1):27–36, 2009.

Merrick L. Furst, James B. Saxe, and Michael Sipser. Parity, circuits, and the polynomial-time hierarchy. In 22nd Annual Symposium on Foundations of Computer Science, Nashville, Tennessee, USA, 28-30 October 1981, pages 260–270. IEEE Computer Society, 1981. Johan H˚ astad. Computational Limitations of Small-depth Circuits. MIT Press, Cambridge, MA, USA, 1987. Ryan C. Harkins and John M. Hitchcock.

Exact learning algorithms, betting games, and circuit lower bounds. In Luca Aceto, Monika Henzinger, and Jir´ ı Sgall, editors, Automata, Languages and Programming - 38th International Colloquium, ICALP 2011, Zurich, Switzerland, July 4-8, 2011, Proceedings, Part I, volume 6755 of Lecture Notes in Computer Science, pages 416–423. Springer, 2011. Russell Impagliazzo, William Matthews, and Ramamohan Paturi. A satisfiability algorithm for ac0. In Yuval Rabani, editor, Proceedings of the Twenty-Third Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2012, Kyoto, Japan, January 17-19, 2012, pages 961–972. SIAM, 2012. Adam Klivans, Pravesh Kothari, and Igor Carboni Oliveira. Constructing hard functions using learning algorithms.

In Proceedings of the 28th Conference on Computational Complexity, CCC 2013, K.lo Alto, California, USA, 5-7 June, 2013, pages 86–97. IEEE, 2013. Nathan Linial, Yishay Mansour, and Noam Nisan. Constant depth circuits, fourier transform, and learnability.

• J. ACM, 40(3):607–620, 1993.
• A. A. Razborov.

Lower bounds on the size of bounded depth circuits over a complete basis with logical addition. Mathematical notes of the Academy of Sciences of the USSR, 41(4):333–338, 1987. Rahul Santhanam. Fighting perebor: New and improved algorithms for formula and QBF satisfiability. In 51th Annual IEEE Symposium on Foundations of Computer Science, FOCS 2010, October 23-26, 2010, Las Vegas, Nevada, USA, pages 183–192. IEEE Computer Society, 2010.

Roman Smolensky. Algebraic methods in the theory of lower bounds for boolean circuit complexity. In Proceedings of the 19th Annual ACM Symposium on Theory of Computing, 1987, New York, New York, USA, pages 77–82, 1987. Srikanth Srinivasan. A compression algorithm for $acˆ0[\oplus]$ circuits using certifying polynomials. Electronic Colloquium on Computational Complexity (ECCC), 22:142, 2015. Ryan Williams. Improving exhaustive search implies superpolynomial lower bounds. In Leonard J. Schulman, editor, Proceedings of the 42nd ACM Symposium on Theory of Computing, STOC 2010, Cambridge, Massachusetts, USA, 5-8 June 2010, pages 231–240. ACM, 2010.

Ryan Williams. Non-uniform ACC circuit lower bounds. In Proceedings of the 26th Annual IEEE Conference on Computational Complexity, CCC 2011, San Jose, California, June 8-10, 2011, pages 115–125. IEEE Computer Society, 2011.