Limits on Representing Functions by Linear Combinations of Simple - - PowerPoint PPT Presentation

Limits on Representing Functions by Linear Combinations of Simple Functions

Ryan Williams MIT

simple simple simple simple

∑

𝑔 ∶ 0,1 𝑜 → 0,1 ? ≡

simple simple

Let 𝓓 be a class of “simple” functions (take Boolean inputs, but need not be Boolean-valued)

The ℝ-linear Representation Problem

simple simple simple simple simple simple

∑

Which “interesting” functions 𝒈 can(not) be represented by “short” ℝ-linear combinations of functions from 𝓓?

𝑔 ∶ 0,1 𝑜 → 0,1 ≡

poly(𝒐) “size”?

2 −𝜌 −𝑓 𝜚

Call this a ∑ ∘ 𝓓 circuit

Note: If 𝓓 spans the vector space of all functions 𝒈 ∶ 𝟏, 𝟐 𝒐 → ℝ then there is always a ∑ ∘ 𝓓 circuit of ≤ 𝟑𝒐 size…

The ℝ-linear Representation Problem

Which “interesting” functions 𝒈 can(not) be represented by “short” ℝ-linear combinations of functions from 𝓓? If 𝓓 is the class of 𝟑𝒐 𝑩𝑶𝑬 functions on 𝒐 variables: ∑ ∘ 𝑩𝑶𝑬 ≡ 𝟏/𝟐 polynomials over ℝ If 𝓓 is the class of 𝟑𝒐 𝑸𝑩𝑺𝑱𝑼𝒁 functions on 𝒐 variables: ∑ ∘ 𝑸𝑩𝑺𝑱𝑼𝒁 ≡ −𝟐/𝟐 polynomials over ℝ (Fourier analysis of Boolean functions) These are well-understood: 𝓓 is a basis for the vector space of functions 𝑔 ∶ 0,1 𝑜 → ℝ ⇒ the ℝ-linear representation of 𝒈 is unique, so the “shortest” is also the “longest”… More interesting cases: representations are not unique

• 1. Linear Threshold Functions [𝑴𝑼𝑮]
• 2. Rectified Linear Units [𝑺𝒇𝑴𝑽]

3. 𝑯𝑮(𝒒)-Polynomials of Degree-𝒆 [𝑸𝑷𝑴𝒁𝒆 𝒒 ] (𝒒 prime and 𝒆 ≥ 𝟑)

This Paper: Three Simple Classes

• There are ≫ 𝟑𝒐 functions on 𝒐 variables,

so ℝ-linear representations are not unique

𝟑𝚰 𝒐𝟑 LTFs, 𝒒𝚰 𝒐𝒆 degree-𝒆 polys, ∞ ReLU functions

• ℝ-linear Representations have been studied!

∑ ∘ 𝑴𝑼𝑮 = Special Case of Depth-2 Threshold Circuits ∑ ∘ 𝑺𝒇𝑴𝑽 = “Depth-2 Neural Net with ReLU activation” ∑ ∘ 𝑸𝑷𝑴𝒁𝒆[𝒒] = “Higher-Order” Fourier Analysis for 𝒆 ≥ 𝟑

For all three classes:

Depth-Two LTF Circuits (𝑴𝑼𝑮 ∘ 𝑴𝑼𝑮): Major problem to find “nice” functions without 𝑜𝑙-gate 𝑀𝑈𝐺 ∘ 𝑀𝑈𝐺 circuits, for all 𝑙

Sums of Linear Threshold Functions

We prove: Thm ∀𝒍, ∃𝒈𝒍 ∈ 𝑶𝑸 without 𝒐𝒍-size ∑ ∘ 𝑴𝑼𝑮

[Hajnal et al.’91] exp(n) depth-two lower bounds for small 𝑥𝑗’s

• Def. 𝑔

𝑜: 0,1 𝑜 → 0,1 is an LTF if ∃ 𝑥1, … 𝑥𝑜, 𝑢 ∈ ℝ such that

∀ 𝑦1, … , 𝑦𝑜 ∈ 0,1 𝑜, 𝒈 𝒚𝟐, … , 𝒚𝒐 = 𝟐 ⇔ ∑𝒋 𝒙𝒋𝒚𝒋 ≥ 𝒖 [Roychowdhury-Orlitsky-Siu’94] What about ∑ ∘ 𝑴𝑼𝑮? Special case of 𝑴𝑼𝑮 ∘ 𝑴𝑼𝑮: the linear form for output LTF must always evaluate to 0 or 1 Still, no 𝒐𝟐.𝟔-gate lower bounds were known for ∑ ∘ 𝑴𝑼𝑮!

Thm ∃𝒈 ∈ 𝑶𝑼𝑱𝑵𝑭[𝒐𝒎𝒑𝒉∗𝒐] without 𝒒𝒑𝒎𝒛(𝒐)-size ∑ ∘ 𝑴𝑼𝑮

Note: It is a major open problem to prove ∃𝒈 ∈ 𝑶𝑸 without 𝒐𝒍-size (unrestricted) circuits

∑ ∘ 𝑺𝒇𝑴𝑽 generalizes ∑ ∘ 𝑴𝑼𝑮

Sums of ReLUs

We can generalize the ∑ ∘ 𝑴𝑼𝑮 limits to ∑ ∘ 𝑺𝒇𝑴𝑽: Thm ∀𝒍, ∃𝒈𝒍 ∈ 𝑶𝑸 without 𝒐𝒍-size ∑ ∘ 𝑺𝒇𝑴𝑽

∑ ∘ 𝑺𝒇𝑴𝑽 = “Depth-Two Neural Nets with ReLU Activations” Very widely studied, thousands of references

• Def. 𝑔

𝑜: ℝ𝑜 → ℝ+ is a ReLU if ∃ 𝑥1, … 𝑥𝑜, 𝑢 ∈ ℝ such that

∀ 𝑦1, … , 𝑦𝑜 ∈ ℝ𝑜, 𝒈 𝒚𝟐, … , 𝒚𝒐 = 𝐧𝐛𝐲(𝟏, ∑𝒋 𝒙𝒋𝒚𝒋 + 𝒖) Several recent references [see paper] give lower bounds for some “weird” 𝒈: ℝ𝑜 → ℝ which vary sharply / sensitive

Thm ∃𝒈 ∈ 𝑶𝑼𝑱𝑵𝑭[𝒐𝒎𝒑𝒉∗𝒐] without 𝒒𝒑𝒎𝒛(𝒐)-size ∑ ∘ 𝑺𝒇𝑴𝑽

Again: major open problem to prove ∃𝒈 ∈ 𝑶𝑸 without 𝒐𝒍-size (unrestricted) circuits No lower bounds known for discrete-domain / Boolean functions (note: “most sensitive” Boolean fn PARITY has O(n)-size ∑∘ 𝑴𝑼𝑮)

Compelling Conjecture [“Degree-Two Uncertainty Principle”]: 𝑩𝑶𝑬 (on 𝒐 inputs) requires 𝒐𝝏 𝟐 -size ∑∘ 𝑸𝑷𝑴𝒁𝟑[𝟑]

Sums of Low-Degree GF(p)-Polys

We prove:

Thm ∀𝒆, 𝒍, ∀𝒒 prime, ∃𝒈𝒍 ∈ 𝑶𝑸 without 𝒐𝒍-size ∑∘ 𝑸𝑷𝑴𝒁𝒆[𝒒] Known: 𝑩𝑶𝑬 requires Ω(2𝑜)-size ∑∘ 𝑸𝑷𝑴𝒁𝟐 𝟑 ∑∘ 𝑸𝑷𝑴𝒁𝒆[𝒒]: Linear combination of 𝑔: 0,1 𝑜 → {0,1, … , 𝑞 − 1} where for every 𝑔 there is a degree-𝑒 polynomial 𝑟(𝑦) such that ∀𝑦 ∈ 0,1 𝑜, 𝒈 𝒚 = 𝒓 𝒚 mod 𝒒

No non-trivial lower bounds were known for ∑ ∘ 𝑸𝑷𝑴𝒁𝟑[𝒒]

Thm ∃𝒈 ∈ 𝑶𝑼𝑱𝑵𝑭[𝒐𝒎𝒑𝒉∗𝒐] without 𝒒𝒑𝒎𝒛(𝒐)-size ∑∘ 𝑸𝑷𝑴𝒁𝒆[𝒒] for all fixed 𝒆 and fixed prime 𝒒 𝑩𝑶𝑬 has O(2𝑜/2)-size ∑∘ 𝑸𝑷𝑴𝒁𝟑[𝟑] Case of 𝒆 = 𝟑, 𝒒 = 𝟑 is already very interesting!

Key Theorem: Let 𝓓 be a class of functions 𝒈 ∶ 𝟏, 𝟐 𝒐 → ℝ. Assume: there is an 𝜻 > 𝟏 and an algorithm 𝑩 so that for any given 𝒈𝟐, … , 𝒈𝟓 ∈ 𝓓, 𝑩 can compute the “sum-product” ෍

𝒃∈ 𝟏,𝟐 𝒐

ෑ

𝒋=𝟐 𝟓

𝒈𝒋(𝒃) in 𝟑𝒐 𝟐−𝜻 time. Then: ∀𝒍, ∃𝒈 ∈ 𝑶𝑸 without 𝒐𝒍-size ∑∘ 𝓓, and ∃𝒈 ∈ 𝑶𝑼𝑱𝑵𝑭 𝒐𝒎𝒑𝒉∗𝒐 without 𝒒𝒑𝒎𝒛(𝒐)-size ∑∘ 𝓓 Applies the new Easy Witness Lemma of [Murray-W’18]

A Key Theorem

A new instance of “Circuit Analysis Algorithms ⇒ Circuit Lower Bounds” We show how to compute sum-products in 𝟑𝒐 𝟐−𝜻 time for LTFs, ReLUs, and low-degree polynomials

[Murray-W’18] ⇒ ∀𝒍, ∃𝒈 ∈ 𝑶𝑸 without 𝒐𝒍-size unrestricted circuits

Major Ideas in the Key Theorem

Assume: (1) There is a 𝟑𝒐 𝟐−𝜻 -time sum-product algorithm 𝑩 for 𝓓 (2) For some fixed 𝒍, all 𝒈 ∈ 𝑶𝑸 have 𝒐𝒍-size ∑∘ 𝓓 Goal: Derive a contradiction. (1) and (2) ⇒ Given (unrestricted) circuit 𝑼 with 𝒐 inputs and 𝒏 size Can guess-and-check 𝒏𝒍-size ∑∘ 𝓓 computing 𝑼, in 𝟑𝒐 𝟐−𝜻 𝒏𝑷 𝟐 time (1) ⇒ Can solve Circuit-UNSAT in nondeterministic 𝟑𝒐 𝟐−𝜻 𝒏𝑷 𝟐 time

We can even solve #Circuit-SAT, because we can compute ∑𝒃∈ 𝟏,𝟐 𝒐(∑∘ 𝓓 𝒃 ) = ∑ ∑𝒃 𝓓(𝒃) by solving sum-product for 𝒐𝒍 times

Contradicts (2) when ∑∘ 𝓓 can be simulated by Boolean circuits!

Note: to guess, we need that the coefficients in our linear combinations have “small” bit complexity, WLOG

The proof crucially relies on ∑∘ 𝓓 computing a circuit exactly

Sum-Product Algorithm for LTF

Uses (old) fact that #Subset-Sum is solvable in 𝒒𝒑𝒎𝒛 𝒐 ⋅ 𝟑𝒐/𝟑 time! Thm [HS’76] #Subset-Sum on 𝒐 numbers is in 𝒒𝒑𝒎𝒛 𝒐 ⋅ 𝟑𝒐/𝟑 time Proof Given 𝒙𝟐, … , 𝒙𝒐, 𝒖, we want to know the number of 𝑻 ⊆ [𝒐] such that ∑𝒋∈𝑻 𝒙𝒋 = 𝒖 Takes 𝒒𝒑𝒎𝒛 𝒐 ⋅ 𝟑𝒐/𝟑 time in total

• 1. Enumerate all possible 𝟑𝒐/𝟑 subsets 𝑻 of {𝒙𝟐, … , 𝒙𝒐/𝟑}.

Make a list 𝑴𝟐 of the 𝟑𝒐/𝟑 subset sums, and SORT all sums in 𝑴𝟐

• 2. Enumerate all possible 𝟑𝒐/𝟑 subsets 𝑼 of {𝒙𝒐/𝟑+𝟐, … , 𝒙𝒐}.

For each 𝑼 summing to a value 𝒘, BINARY SEARCH for a value 𝒘′ in 𝑴𝟐 such that 𝒘 + 𝒘′ = 𝒖

• 3. To compute the total number of subsets summing to 𝒖:

For each sum value 𝒘′ appearing in 𝑴𝟐, store the number 𝒐𝒘′ of subsets in 𝑴𝟐 which have value 𝒘′. Later, if value 𝒘′ is found in the binary search, add 𝒐𝒘′ to a running sum.

Sum-Product Algorithm for LTF

Uses (old) fact that #Subset-Sum is solvable in 𝒒𝒑𝒎𝒛 𝒐 ⋅ 𝟑𝒐/𝟑 time! Thm For any 𝒈𝟐, … , 𝒈𝟓 ∈ 𝑴𝑼𝑮, we can compute ෍

𝒃∈ 𝟏,𝟐 𝒐

ෑ

𝒋=𝟐 𝟓

𝒈𝒋(𝒃) Proof An Exact LTF (𝑭𝑴𝑼𝑮) has the form 𝒉 𝒚 = 𝟐 ⇔ ∑𝒋 𝒙𝒋𝒚𝒋 = 𝒖 So we can write [HP, CCC’10]: Every 𝑴𝑼𝑮 on 𝒐 inputs can be written as ∑𝒒𝒑𝒎𝒛 𝒐 𝑭𝑴𝑼𝑮

෍

𝒃∈ 𝟏,𝟐 𝒐

ෑ

𝒋=𝟐 𝟓

𝒈𝒋(𝒃) = ෍

𝒃∈ 𝟏,𝟐 𝒐

ෑ

𝒋=𝟐 𝟓

෍

𝒒𝒑𝒎𝒛 𝒐

𝒉𝒋,𝒌(𝒃)

for 𝑭𝑴𝑼𝑮s 𝒉𝒋,𝒌

= ෍

𝒃∈ 𝟏,𝟐 𝒐

෍

𝒒𝒑𝒎𝒛 𝒐

ෑ

𝒋=𝟐 𝟓

𝒉𝒋,𝒌′ 𝒃

Simple algebra:

= ෍

𝒒𝒑𝒎𝒛 𝒐

෍

𝒃∈{𝟏,𝟐}𝒐

ෑ

𝒋=𝟐 𝟓

𝒉𝒋,𝒌′ 𝒃

Can compute in 𝒒𝒑𝒎𝒛 𝒐 ⋅ 𝟑𝒐/𝟑 time! Each ς𝒋=𝟐

𝟓

𝒉𝒋,𝒌′ 𝒚 = 𝒊 𝒚 for some 𝑭𝑴𝑼𝑮 𝒊

in 𝒒𝒑𝒎𝒛 𝒐 ⋅ 𝟑𝒐/𝟑 time.

#Subset-Sum in 𝒒𝒑𝒎𝒛 𝒐 ⋅ 𝟑𝒐/𝟑 time ⇒ ∑𝑏 𝑕 𝑏 in 𝒒𝒑𝒎𝒛 𝒐 ⋅ 𝟑𝒐/𝟑 time

Sum-Product Algorithm for Polys

Uses (recent) fact that counting Boolean roots of 𝒐-variable degree-𝒆 GF(𝒒)-polynomials is solvable in 𝟑𝒐 𝟐−𝟐/𝑷(𝒒𝒆) time [LPTWY’17] Thm For any 𝒈𝟐, … , 𝒈𝟓 ∈ 𝑸𝑷𝑴𝒁𝒆[𝒒], we can compute ෍

𝒃∈ 𝟏,𝟐 𝒐

ෑ

𝒋=𝟐 𝟓

𝒈𝒋(𝒃) Proof Idea Reduce the sum-product problem on four degree-𝒆 polys to counting Boolean roots of 𝑷(𝟐) degree-𝑷(𝒒𝒆) polys in 𝒒𝒑𝒎𝒛 𝒐 ⋅ 𝟑

𝒐 𝟐−

𝟐 𝑷 𝒒𝒆

time.

Algorithm uses a derandomized version of Razborov-Smolensky’s probabilistic representation of AC0[𝒒] by low-degree GF(𝒒) polynomials, along with a divide-and-conquer approach for fast evaluation

Open Problems

We proved fixed-polynomial lower bounds for functions in 𝑶𝑸 Can we prove 𝑻𝑩𝑼 requires 𝒐𝒍-size ∑∘ 𝑴𝑼𝑮, for all k? New ways of deriving strong lower bounds from “old” approaches Open even when 𝒆 = 𝟑. Could our approaches say anything?

Reviewer asked: Can “split-and-list” be viewed as a lower bound method? The alg. for polys uses Razborov-Smolensky, which is used in lower bounds!

For each 𝒍, there is an 𝒈 ∈ 𝑶𝑼𝑱𝑵𝑭 𝒐𝒉 𝒍 without 𝒐𝒍-size ∑∘ 𝑴𝑼𝑮

Current algorithms-to-lower bounds connections don’t seem to point a way Constant Degree Hypothesis [Barrington-Straubing-Therien’90]:

For each fixed 𝒆, 𝑩𝑶𝑬 does not have 𝑵𝑷𝑬𝒒 ∘ 𝑵𝑷𝑬𝒓 ∘ 𝑩𝑶𝑬𝒆 circuits of 𝟑𝒑 𝒐 size

The (old) algorithm for #Subset-Sum splits the instance of 𝒐 items into two parts of size 𝒐/𝟑 each, lists all 𝟑𝒐/𝟑 subsums separately, sorts the two lists and binary searches for the overall subset sum.

Thank you!