The minimum Euclidean norm point in a polytope: Wolfe's method is - PowerPoint PPT Presentation

The minimum Euclidean norm point in a polytope: Wolfe's method is exponential Luis Rademacher, UC Davis Simons Institute Join work with Jesús de Loera, Jamie Haddock

Minimum norm point in a polytope • Given 𝑞 1 , … , 𝑞 𝑜 ∈ 𝑆 𝑒 , determine argmin 𝑦∈𝑄 𝑦 for 𝑄 = conv(𝑞 1 , … , 𝑞 𝑜 ) . • A convex quadratic program.

Projection onto a simplex: not so easy • Only polynomial time algorithms I know are general purpose convex programming algorithms like the ellipsoid method. • No strongly polynomial time algorithm known.

Wolfe’s method (Wolfe ’74, Lawson Hanson ‘74) • A combinatorial algorithm to find the minimum norm point in a polytope. • Not the same as Frank-Wolfe

Motivation • In machine learning: • Fujishige-Wolfe algorithm • one of the most practical algorithms for submodular function minimization . • Wolfe’s method is a subroutine in it • Optimal loading of recursive neural networks [CDKSV ‘95] • Non-negative least squares

This talk: • Complexity of Wolfe’s method • Relationship between Linear Programming (LP) and minimum norm point problem.

Related work • [Lawson, Hanson ‘74] An algorithm very similar to Wolfe’s. • [Fujishige, Hayashi, Isotani ’06] Polynomial time reduction from LP to minimum norm point in a polytope. • [Chakrabarty, Jain, Kothari ‘14] [Lacoste-Julien, Jaggi ‘15] Rates of convergence of Wolfe’s method.

Our results • LP reduces in strongly polynomial time to “minimum norm point in a simplex ”. • A (strongly) polynomial time algorithm for minimum norm point in a simplex would give a (strongly) polynomial time algorithm for general LPs. • Step 1: LP reduc es to “membership in a V - polytope”. • Step 2: “Membership in a V - polytope” reduces to “distance to a simplex”.

Our results • Wolfe’s method takes exponential time in the worst case. • We construct explicit sets of points in every dimension. • Similar in spirit to Klee-Minty cubes for LP.

Idea of Wolfe’s method • Given points 𝑄 ⊆ 𝑆 𝑒 , maintain the following invariant : A subset 𝑇 ⊆ 𝑄 whose vertices determine a simplex, and the minimum norm point, 𝑦 , in the simplex. • Start with any point in 𝑄 as 𝑇 and 𝑦 . • Alternate between the following two steps: • Find point 𝑞 ∈ 𝑄 such that 𝑇 ← 𝑑𝑝𝑜𝑤 𝑇 ∪ 𝑞 contains a better point (entering rule). Add 𝑞 to 𝑇 . • Let 𝑦 “ follow gravity ” (towards 0) within 𝑑𝑝𝑜𝑤 𝑇 while dimension of current face decreases. Let 𝑇 be the (vertices of the) current face of 𝑦 .

Inefficiency of Wolfe’s method with “ minnorm ” entering rule • A point, 𝑏 , enters current set 𝑇 , leaves, and then re-enters. • 𝑏 < 𝑞 < 𝑟 < 𝑠 < |𝑡| S . a ap apq pq pqr qr qrs rs rsa

Exponential lower bound • Replace point 𝑏 and 1 st coordinate by subspace and a set of points in it, constructed recursively. • Seq 𝑇 : 𝑄 𝑒 − 2 → ⋯ → 𝑠 𝑒 𝑡 𝑒 → 𝑄 𝑒 − 2 𝑠 𝑒 𝑡 𝑒

Open questions • Find exponential lower bound for other “entering rules.” ( linopt) • For submodular function minimization, polytopes are so- called “base polytopes.” • Our exponential example is not of that kind. • Could Wolfe’s method be faster on base polytopes? • Smoothed analysis of Wolfe’s method.