Limits on Representing Functions by Linear Combinations of Simple - PowerPoint PPT Presentation

Limits on Representing Functions by Linear Combinations of Simple Functions ∑ 𝑔 ∶ 0,1 𝑜 → 0,1 ? ≡ simple simple simple simple simple simple Ryan Williams MIT

The ℝ -linear Representation Problem Let 𝓓 be a class of “simple” functions (take Boolean inputs, but need not be Boolean-valued) Which “interesting” functions 𝒈 can(not) be represented by “short” ℝ -linear combinations of functions from 𝓓 ? ∑ 𝑔 ∶ 0,1 𝑜 → 0,1 poly( 𝒐 ) “size” ? ≡ −𝜌 2 𝜚 −𝑓 Call this a ∑ ∘ 𝓓 circuit simple simple simple simple simple simple Note: If 𝓓 spans the vector space of all functions 𝒈 ∶ 𝟏, 𝟐 𝒐 → ℝ then there is always a ∑ ∘ 𝓓 circuit of ≤ 𝟑 𝒐 size…

The ℝ -linear Representation Problem Which “interesting” functions 𝒈 can(not) be represented by “short” ℝ -linear combinations of functions from 𝓓 ? If 𝓓 is the class of 𝟑 𝒐 𝑩𝑶𝑬 functions on 𝒐 variables: ∑ ∘ 𝑩𝑶𝑬 ≡ 𝟏/𝟐 polynomials over ℝ If 𝓓 is the class of 𝟑 𝒐 𝑸𝑩𝑺𝑱𝑼𝒁 functions on 𝒐 variables: ∑ ∘ 𝑸𝑩𝑺𝑱𝑼𝒁 ≡ −𝟐/𝟐 polynomials over ℝ (Fourier analysis of Boolean functions) These are well-understood: 𝓓 is a basis for the vector space of functions 𝑔 ∶ 0,1 𝑜 → ℝ ⇒ the ℝ -linear representation of 𝒈 is unique, so the “shortest” is also the “longest”… More interesting cases: representations are not unique

This Paper: Three Simple Classes 1. Linear Threshold Functions [ 𝑴𝑼𝑮 ] 2. Rectified Linear Units [ 𝑺𝒇𝑴𝑽 ] 𝑯𝑮 ( 𝒒 )- Polynomials of Degree- 𝒆 [ 𝑸𝑷𝑴𝒁𝒆 𝒒 ] 3. ( 𝒒 prime and 𝒆 ≥ 𝟑 ) For all three classes: There are ≫ 𝟑 𝒐 functions on 𝒐 variables, • so ℝ -linear representations are not unique 𝟑 𝚰 𝒐 𝟑 LTFs, 𝒒 𝚰 𝒐 𝒆 degree- 𝒆 polys, ∞ ReLU functions • ℝ -linear Representations have been studied! ∑ ∘ 𝑴𝑼𝑮 = Special Case of Depth-2 Threshold Circuits ∑ ∘ 𝑺𝒇𝑴𝑽 = “Depth -2 Neural Net with ReLU activation” ∑ ∘ 𝑸𝑷𝑴𝒁𝒆[𝒒] = “Higher - Order” Fourier Analysis for 𝒆 ≥ 𝟑

Sums of Linear Threshold Functions 𝑜 : 0,1 𝑜 → 0,1 is an LTF if ∃ 𝑥 1 , … 𝑥 𝑜 , 𝑢 ∈ ℝ such that Def. 𝑔 ∀ 𝑦 1 , … , 𝑦 𝑜 ∈ 0,1 𝑜 , 𝒈 𝒚 𝟐 , … , 𝒚 𝒐 = 𝟐 ⇔ ∑ 𝒋 𝒙 𝒋 𝒚 𝒋 ≥ 𝒖 Depth-Two LTF Circuits ( 𝑴𝑼𝑮 ∘ 𝑴𝑼𝑮 ): Major problem to find “nice” functions without 𝑜 𝑙 -gate 𝑀𝑈𝐺 ∘ 𝑀𝑈𝐺 circuits, for all 𝑙 [Hajnal et al.’91] exp(n) depth-two lower bounds for small 𝑥 𝑗 ’s [Roychowdhury-Orlitsky- Siu’94] What about ∑ ∘ 𝑴𝑼𝑮 ? Special case of 𝑴𝑼𝑮 ∘ 𝑴𝑼𝑮 : the linear form for output LTF must always evaluate to 0 or 1 Still, no 𝒐 𝟐.𝟔 -gate lower bounds were known for ∑ ∘ 𝑴𝑼𝑮 ! We prove: Thm ∀𝒍 , ∃𝒈 𝒍 ∈ 𝑶𝑸 without 𝒐 𝒍 -size ∑ ∘ 𝑴𝑼𝑮 Thm ∃𝒈 ∈ 𝑶𝑼𝑱𝑵𝑭[𝒐 𝒎𝒑𝒉 ∗ 𝒐 ] without 𝒒𝒑𝒎𝒛(𝒐) -size ∑ ∘ 𝑴𝑼𝑮 Note: It is a major open problem to prove ∃𝒈 ∈ 𝑶𝑸 without 𝒐 𝒍 -size (unrestricted) circuits

Sums of ReLUs 𝑜 : ℝ 𝑜 → ℝ + is a ReLU if ∃ 𝑥 1 , … 𝑥 𝑜 , 𝑢 ∈ ℝ such that Def. 𝑔 ∀ 𝑦 1 , … , 𝑦 𝑜 ∈ ℝ 𝑜 , 𝒈 𝒚 𝟐 , … , 𝒚 𝒐 = 𝐧𝐛𝐲(𝟏, ∑ 𝒋 𝒙 𝒋 𝒚 𝒋 + 𝒖) ∑ ∘ 𝑺𝒇𝑴𝑽 generalizes ∑ ∘ 𝑴𝑼𝑮 ∑ ∘ 𝑺𝒇𝑴𝑽 = “Depth -Two Neural Nets with ReLU Activations” Very widely studied, thousands of references Several recent references [see paper] give lower bounds for some “weird” 𝒈: ℝ 𝑜 → ℝ which vary sharply / sensitive No lower bounds known for discrete-domain / Boolean functions (note: “most sensitive” Boolean fn PARITY has O(n)-size ∑∘ 𝑴𝑼𝑮 ) We can generalize the ∑ ∘ 𝑴𝑼𝑮 limits to ∑ ∘ 𝑺𝒇𝑴𝑽 : Thm ∀𝒍 , ∃𝒈 𝒍 ∈ 𝑶𝑸 without 𝒐 𝒍 -size ∑ ∘ 𝑺𝒇𝑴𝑽 Thm ∃𝒈 ∈ 𝑶𝑼𝑱𝑵𝑭[𝒐 𝒎𝒑𝒉 ∗ 𝒐 ] without 𝒒𝒑𝒎𝒛(𝒐) -size ∑ ∘ 𝑺𝒇𝑴𝑽 Again: major open problem to prove ∃𝒈 ∈ 𝑶𝑸 without 𝒐 𝒍 -size (unrestricted) circuits

Sums of Low-Degree GF(p)-Polys ∑∘ 𝑸𝑷𝑴𝒁𝒆[𝒒] : Linear combination of 𝑔: 0,1 𝑜 → {0,1, … , 𝑞 − 1} where for every 𝑔 there is a degree- 𝑒 polynomial 𝑟(𝑦) such that ∀𝑦 ∈ 0,1 𝑜 , 𝒈 𝒚 = 𝒓 𝒚 mod 𝒒 Case of 𝒆 = 𝟑, 𝒒 = 𝟑 is already very interesting! Compelling Conjecture [“Degree - Two Uncertainty Principle”]: 𝑩𝑶𝑬 (on 𝒐 inputs) requires 𝒐 𝝏 𝟐 -size ∑∘ 𝑸𝑷𝑴𝒁𝟑[𝟑] Known: 𝑩𝑶𝑬 requires Ω(2 𝑜 ) -size ∑∘ 𝑸𝑷𝑴𝒁𝟐 𝟑 𝑩𝑶𝑬 has O(2 𝑜/2 ) -size ∑∘ 𝑸𝑷𝑴𝒁𝟑[𝟑] No non-trivial lower bounds were known for ∑ ∘ 𝑸𝑷𝑴𝒁𝟑[𝒒] We prove: Thm ∀𝒆, 𝒍, ∀𝒒 prime, ∃𝒈 𝒍 ∈ 𝑶𝑸 without 𝒐 𝒍 -size ∑∘ 𝑸𝑷𝑴𝒁𝒆[𝒒] Thm ∃𝒈 ∈ 𝑶𝑼𝑱𝑵𝑭[𝒐 𝒎𝒑𝒉 ∗ 𝒐 ] without 𝒒𝒑𝒎𝒛(𝒐) -size ∑∘ 𝑸𝑷𝑴𝒁𝒆[𝒒] for all fixed 𝒆 and fixed prime 𝒒

A Key Theorem A new instance of “ Circuit Analysis Algorithms ⇒ Circuit Lower Bounds ” Key Theorem: Let 𝓓 be a class of functions 𝒈 ∶ 𝟏, 𝟐 𝒐 → ℝ . Assume: there is an 𝜻 > 𝟏 and an algorithm 𝑩 so that for any given 𝒈 𝟐 , … , 𝒈 𝟓 ∈ 𝓓 , 𝑩 can compute the “sum - product” 𝟓 ෍ ෑ 𝒈 𝒋 (𝒃) 𝒃∈ 𝟏,𝟐 𝒐 𝒋=𝟐 in 𝟑 𝒐 𝟐−𝜻 time. Then: ∀𝒍 , ∃𝒈 ∈ 𝑶𝑸 without 𝒐 𝒍 -size ∑∘ 𝓓 , and ∃𝒈 ∈ 𝑶𝑼𝑱𝑵𝑭 𝒐 𝒎𝒑𝒉 ∗ 𝒐 without 𝒒𝒑𝒎𝒛(𝒐) -size ∑∘ 𝓓 Applies the new Easy Witness Lemma of [Murray- W’18] We show how to compute sum-products in 𝟑 𝒐 𝟐−𝜻 time for LTFs, ReLUs, and low-degree polynomials

Major Ideas in the Key Theorem Assume: (1) There is a 𝟑 𝒐 𝟐−𝜻 -time sum-product algorithm 𝑩 for 𝓓 (2) For some fixed 𝒍 , all 𝒈 ∈ 𝑶𝑸 have 𝒐 𝒍 -size ∑∘ 𝓓 Goal: Derive a contradiction. (1) and (2) ⇒ Given (unrestricted) circuit 𝑼 with 𝒐 inputs and 𝒏 size Can guess-and-check 𝒏 𝒍 -size ∑∘ 𝓓 computing 𝑼 , in 𝟑 𝒐 𝟐−𝜻 𝒏 𝑷 𝟐 time Note: to guess, we need that the coefficients in our linear combinations have “small” bit complexity, WLOG (1) ⇒ Can solve Circuit-UNSAT in nondeterministic 𝟑 𝒐 𝟐−𝜻 𝒏 𝑷 𝟐 time We can even solve #Circuit-SAT, because we can compute ∑ 𝒃∈ 𝟏,𝟐 𝒐 (∑∘ 𝓓 𝒃 ) = ∑ ∑ 𝒃 𝓓(𝒃) by solving sum-product for 𝒐 𝒍 times [Murray- W’18] ⇒ ∀𝒍 , ∃𝒈 ∈ 𝑶𝑸 without 𝒐 𝒍 -size unrestricted circuits Contradicts (2) when ∑∘ 𝓓 can be simulated by Boolean circuits! The proof crucially relies on ∑∘ 𝓓 computing a circuit exactly

Sum-Product Algorithm for LTF Uses (old) fact that #Subset-Sum is solvable in 𝒒𝒑𝒎𝒛 𝒐 ⋅ 𝟑 𝒐/𝟑 time! Thm [HS’76] #Subset-Sum on 𝒐 numbers is in 𝒒𝒑𝒎𝒛 𝒐 ⋅ 𝟑 𝒐/𝟑 time Proof Given 𝒙 𝟐 , … , 𝒙 𝒐 , 𝒖 , we want to know the number of 𝑻 ⊆ [𝒐] such that ∑ 𝒋∈𝑻 𝒙 𝒋 = 𝒖 1. Enumerate all possible 𝟑 𝒐/𝟑 subsets 𝑻 of {𝒙 𝟐 , … , 𝒙 𝒐/𝟑 } . Make a list 𝑴 𝟐 of the 𝟑 𝒐/𝟑 subset sums, and SORT all sums in 𝑴 𝟐 2. Enumerate all possible 𝟑 𝒐/𝟑 subsets 𝑼 of {𝒙 𝒐/𝟑+𝟐 , … , 𝒙 𝒐 } . For each 𝑼 summing to a value 𝒘 , BINARY SEARCH for a value 𝒘′ in 𝑴 𝟐 such that 𝒘 + 𝒘′ = 𝒖 3. To compute the total number of subsets summing to 𝒖 : For each sum value 𝒘′ appearing in 𝑴 𝟐 , store the number 𝒐 𝒘′ of subsets in 𝑴 𝟐 which have value 𝒘′ . Later, if value 𝒘′ is found in the binary search, add 𝒐 𝒘′ to a running sum. Takes 𝒒𝒑𝒎𝒛 𝒐 ⋅ 𝟑 𝒐/𝟑 time in total

Sum-Product Algorithm for LTF Uses (old) fact that #Subset-Sum is solvable in 𝒒𝒑𝒎𝒛 𝒐 ⋅ 𝟑 𝒐/𝟑 time! Thm For any 𝒈 𝟐 , … , 𝒈 𝟓 ∈ 𝑴𝑼𝑮 , we can compute 𝟓 in 𝒒𝒑𝒎𝒛 𝒐 ⋅ 𝟑 𝒐/𝟑 time. ෍ ෑ 𝒈 𝒋 (𝒃) 𝒃∈ 𝟏,𝟐 𝒐 𝒋=𝟐 Proof An Exact LTF ( 𝑭𝑴𝑼𝑮 ) has the form 𝒉 𝒚 = 𝟐 ⇔ ∑ 𝒋 𝒙 𝒋 𝒚 𝒋 = 𝒖 #Subset-Sum in 𝒒𝒑𝒎𝒛 𝒐 ⋅ 𝟑 𝒐/𝟑 time ⇒ ∑ 𝑏 𝑕 𝑏 in 𝒒𝒑𝒎𝒛 𝒐 ⋅ 𝟑 𝒐/𝟑 time [HP, CCC’10]: Every 𝑴𝑼𝑮 on 𝒐 inputs can be written as ∑ 𝒒𝒑𝒎𝒛 𝒐 𝑭𝑴𝑼𝑮 𝟓 𝟓 for 𝑭𝑴𝑼𝑮 s 𝒉 𝒋,𝒌 So we can write ෍ ෑ 𝒈 𝒋 (𝒃) = ෍ ෑ ෍ 𝒉 𝒋,𝒌 (𝒃) 𝒃∈ 𝟏,𝟐 𝒐 𝒃∈ 𝟏,𝟐 𝒐 𝒋=𝟐 𝒋=𝟐 𝒒𝒑𝒎𝒛 𝒐 𝟓 𝟓 Simple algebra: = ෍ ෍ ෑ 𝒉 𝒋,𝒌′ 𝒃 = ෍ ෍ ෑ 𝒉 𝒋,𝒌′ 𝒃 𝒃∈ 𝟏,𝟐 𝒐 𝒃∈{𝟏,𝟐} 𝒐 𝒒𝒑𝒎𝒛 𝒐 𝒋=𝟐 𝒒𝒑𝒎𝒛 𝒐 𝒋=𝟐 Can compute in 𝒒𝒑𝒎𝒛 𝒐 ⋅ 𝟑 𝒐/𝟑 time! 𝟓 Each ς 𝒋=𝟐 𝒉 𝒋,𝒌′ 𝒚 = 𝒊 𝒚 for some 𝑭𝑴𝑼𝑮 𝒊