Reading group: Latent Optimized GANs (Game theory brings guns to - - PowerPoint PPT Presentation

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Reading group: Latent Optimized GANs (Game theory brings guns to GANs)

Michal ˇ Sustr

• Dept. of Computer Science and Engineering

Faculty of Electrical Engineering Czech Technical University michal.sustr@aic.fel.cvut.cz

January 9th, 2020

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 1 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Reading group VKR

The reading group is based on two papers:

Differentiable Game Mechanics

• A. Letcher, D. Balduzzi, S. Racaniere, J. Martens, J. Foerster, K. Tuyls, T. Graepel

Journal of Machine Learning Research (JMLR), v20 (84) 1-40, 2019 Arxiv , Reading group LOGAN: Latent Optimisation for Generative Adversarial Networks

• Y. Wu, J. Donahue, D. Balduzzi, K. Simonyan, T. Lillicrap

Submission to ICLR2020 conference (rejected) Openreview , Arxiv Other relevant literature: Stable Opponent Shaping in Differentiable Games

• A. Letcher, J. Foerster, D. Balduzzi, T. Rocktaschel, S. Whiteson

ICLR 2019 poster, Openreview , Arxiv Uncoupled Dynamics Do Not Lead to Nash Equilibrium Andreu Mas-Colell and Sergiu Hart, Paper NeurIPS 2019 workshop: Bridging Game Theory and Deep Learning

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 2 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Motivation

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 3 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Motivation

GANs involve interplay of multiple objectives: min

θD

max

θG

Ex∼p(x) [hD(D (x; θD))] + Ez∼p(z) [hG(D (G (z; θG) ; θD))] The field of GT deals with such min-max optimization problems.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 4 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Motivation

We can use GT-inspired algorithms to improve the results! (mode collapse)

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 5 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Motivation

Higher learning rates and convergence to local minima

GRADIENT DESCENT SGA 0.032 0.1 0.01 learning rate Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 6 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Motivation

Applying latent optimization leads to more diverse image generation. (a) (b)

Figure: Samples from BigGAN-deep (a) and LOGAN (b) with similarly high IS.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 7 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Broader application than just GANs

Normal optimization: single objective Multiple objectives: pareto-optimality Multiple objectives: equilibrium outcomes Possible use-cases other than GANs: Proximal gradient TD learning, multi-level optimization, synthetic gradients, hierarchical reinforcement learning, intrinsic curiosity, and imaginative agents.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 8 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Outline

Background GT basic concepts Computation of NE (Un)constrained games (Un)stable fixed points Dynamics in games Fixed points and equilibria Differentiable Game Mechanics Game decomposition Optimization on decomposed games Symplectic gradient adjustment Experiments LOGAN GANs in general Latent optimization Comparison to SGA Computing the hessian update Experiments

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 9 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Normal form games

Normal form games: Solution concepts: Pareto efficient, (course) correlated eq., Stackelberg eq., Nash eq., Stable fixed points, ...

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 10 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Nash equilibria

Strategy profile where no player has incentive to deviate from their current strategy.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 11 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Nash equilibria

Strategy profile where no player has incentive to deviate from their current strategy. Strategy profile that consists of strategies which are mutual best responses.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 11 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Dynamics, i.e. vector fields

What about learning in games?

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 12 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Learning dynamics in games

. Iterative algorithms can be described with game dynamics:

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 13 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Learning dynamics in games

. Iterative algorithms can be described with game dynamics: Also called evolutionary / population / replicator dynamics

(successful strategies are rewarded by high reproductive rates, so become more likely to participate in subsequent playings of the game.) Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 13 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Learning dynamics in games

. Iterative algorithms can be described with game dynamics: Also called evolutionary / population / replicator dynamics

(successful strategies are rewarded by high reproductive rates, so become more likely to participate in subsequent playings of the game.)

Can be described with differential equations: dxa dt = xa

• (Ay)a − xTAy
• dya

dt = ya

• (Bx)a − xTBy
• Michal ˇ

Sustr Reading group: Latent Optimized GANs January 9th, 2020 13 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Examples of dynamics

(From Daan Bloembergen thesis: Analyzing reinforcement learning algorithms using evolutionary game theory) Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 14 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Uncoupled dynamics do not lead to NE

From Hart and Mas-Colell (2003): Coupled dynamics ˙ xi = Fi(x; (uj)j∈N) Uncoupled dynamics ˙ xi = Fi(x; ui)

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 15 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Uncoupled dynamics do not lead to NE

From Hart and Mas-Colell (2003): Coupled dynamics ˙ xi = Fi(x; (uj)j∈N) Uncoupled dynamics ˙ xi = Fi(x; ui) There exist no uncoupled dynamics that guarantee convergence to NE.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 15 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Uncoupled dynamics do not lead to NE

From Hart and Mas-Colell (2003): Coupled dynamics ˙ xi = Fi(x; (uj)j∈N) Uncoupled dynamics ˙ xi = Fi(x; ui) There exist no uncoupled dynamics that guarantee convergence to NE. Exceptions (special families of games): two-player zero-sum games, two-player potential games, and others.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 15 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Uncoupled dynamics do not lead to NE

From Hart and Mas-Colell (2003): Coupled dynamics ˙ xi = Fi(x; (uj)j∈N) Uncoupled dynamics ˙ xi = Fi(x; ui) There exist no uncoupled dynamics that guarantee convergence to NE. Exceptions (special families of games): two-player zero-sum games, two-player potential games, and others. There exist uncoupled dynamics converging to correlated equilibria.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 15 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Uncoupled dynamics do not lead to NE

From Hart and Mas-Colell (2003): Coupled dynamics ˙ xi = Fi(x; (uj)j∈N) Uncoupled dynamics ˙ xi = Fi(x; ui) There exist no uncoupled dynamics that guarantee convergence to NE. Exceptions (special families of games): two-player zero-sum games, two-player potential games, and others. There exist uncoupled dynamics converging to correlated equilibria. There exist uncoupled dynamics that are most of the time in ǫ-NE, but do not converge to NE.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 15 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Uncoupled dynamics do not lead to NE

From Hart and Mas-Colell (2003): Coupled dynamics ˙ xi = Fi(x; (uj)j∈N) Uncoupled dynamics ˙ xi = Fi(x; ui) There exist no uncoupled dynamics that guarantee convergence to NE. Exceptions (special families of games): two-player zero-sum games, two-player potential games, and others. There exist uncoupled dynamics converging to correlated equilibria. There exist uncoupled dynamics that are most of the time in ǫ-NE, but do not converge to NE. Therefore: GANs should have coupled dynamics to converge.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 15 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

(Un)stable fixed points (1/2)

A fixed point satisfies: f (x0) = x0 Stable (attractive) fixed point x, f (x), f (f (x)), f (f (f (x))), ... converges to x0 (Banach FP) Unstable fixed point ǫ-neighbourhood of x0 does not contract to x0 but diverges.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 16 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

(Un)stable fixed points (1/2)

A fixed point satisfies: f (x0) = x0 Stable (attractive) fixed point x, f (x), f (f (x)), f (f (f (x))), ... converges to x0 (Banach FP) Unstable fixed point ǫ-neighbourhood of x0 does not contract to x0 but diverges. Example: golden ratio φ = 1.61809339887... (solution to a+b

a

= a

b = φ → φ2 − φ − 1 = 0)

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 16 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

(Un)stable fixed points (1/2)

A fixed point satisfies: f (x0) = x0 Stable (attractive) fixed point x, f (x), f (f (x)), f (f (f (x))), ... converges to x0 (Banach FP) Unstable fixed point ǫ-neighbourhood of x0 does not contract to x0 but diverges. Example: golden ratio φ = 1.61809339887... (solution to a+b

a

= a

b = φ → φ2 − φ − 1 = 0)

Other solution φ′ = −0.61809339887...

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 16 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

(Un)stable fixed points (1/2)

A fixed point satisfies: f (x0) = x0 Stable (attractive) fixed point x, f (x), f (f (x)), f (f (f (x))), ... converges to x0 (Banach FP) Unstable fixed point ǫ-neighbourhood of x0 does not contract to x0 but diverges. Example: golden ratio φ = 1.61809339887... (solution to a+b

a

= a

b = φ → φ2 − φ − 1 = 0)

Other solution φ′ = −0.61809339887... Iterative method to find φ: φt+1 = 1 + 1 φt

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 16 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

(Un)stable fixed points (1/2)

A fixed point satisfies: f (x0) = x0 Stable (attractive) fixed point x, f (x), f (f (x)), f (f (f (x))), ... converges to x0 (Banach FP) Unstable fixed point ǫ-neighbourhood of x0 does not contract to x0 but diverges. Example: golden ratio φ = 1.61809339887... (solution to a+b

a

= a

b = φ → φ2 − φ − 1 = 0)

Other solution φ′ = −0.61809339887... Iterative method to find φ: φt+1 = 1 + 1 φt

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 16 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

(Un)stable fixed points (2/2)

Stable: Unstable: .

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 17 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Takeaways

Things to consider: What is the “right” solution concept? Is it robust (stable)? Can we compute it efficiently? On which family of games? Can we approximate it using iterative algorithms?

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 18 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Motivation (1/2)

In single objective problems gradient descent (GD) converges to local minima. Even escapes saddle points almost surely! Lee et. al (2017) However, simultaneous/alternating GD is not guaranteed to converge even to local minima! Cyclic behaviour!

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 19 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Motivation (1/2)

In single objective problems gradient descent (GD) converges to local minima. Even escapes saddle points almost surely! Lee et. al (2017) However, simultaneous/alternating GD is not guaranteed to converge even to local minima! Cyclic behaviour! Therefore: convergence to local minima is difficult!

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 19 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Motivation (2/2)

Simultaneous GD vs SGA

• Michal ˇ

Sustr Reading group: Latent Optimized GANs January 9th, 2020 20 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Differentiable games

The normal form games we talked about so far were constrained to probability simplex

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 21 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Differentiable games

The normal form games we talked about so far were constrained to probability simplex This does not make sense in settings such as GANs - weights are not on simplex!

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 21 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Differentiable games

The normal form games we talked about so far were constrained to probability simplex This does not make sense in settings such as GANs - weights are not on simplex! Introduce differentiable games: Set of players N = {1, . . . , n} Continuously twice differentiable losses {ℓi : Rd → R}n

i=1

NN parameters w = (w1, . . . , wn) ∈ Rd where n

i=1 di = d

Player i controls wi ∈ Rdi, and aims to minimize its loss

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 21 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Dynamics, Jacobian

Simultaneous gradient is the gradient of the losses w/ respect to the parameters of the respective players: ξ(w) = (∇w1ℓ1, . . . , ∇wnℓn) ∈ Rd. Dynamics of the game mean following −ξ with infinitesimal steps.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 22 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Dynamics, Jacobian

Simultaneous gradient is the gradient of the losses w/ respect to the parameters of the respective players: ξ(w) = (∇w1ℓ1, . . . , ∇wnℓn) ∈ Rd. Dynamics of the game mean following −ξ with infinitesimal steps. The Jacobian of a game with dynamics ξ is the (d × d)-matrix of second-derivatives J(w) := ∇w · ξ(w)⊺. Concretely: J(w) =      ∇2

w1ℓ1

∇2

w1,w2ℓ1

· · · ∇2

w1,wnℓ1

∇2

w2,w1ℓ2

∇2

w2ℓ2

· · · ∇2

w2,wnℓ2

. . . . . . ∇2

wn,w1ℓn

∇2

wn,w2ℓn

· · · ∇2

wnℓn

     where ∇2

wi,wjℓk is the (di × dj)-block of 2nd-order derivatives.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 22 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Fixed points and equilibria

local Nash equilibrium: NE only within a neighbourhood: if, for all i, there exists a neighborhood Ui of w∗

i

such that ℓi(w′

i, w∗ −i) ≥ ℓi(w∗ i , w∗ −i) for w′ i ∈ Ui.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 23 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Fixed points and equilibria

local Nash equilibrium: NE only within a neighbourhood: if, for all i, there exists a neighborhood Ui of w∗

i

such that ℓi(w′

i, w∗ −i) ≥ ℓi(w∗ i , w∗ −i) for w′ i ∈ Ui.

A fixed point w∗ with ξ(w∗) = 0 is stable if J(w∗) 0 and J(w∗) is invertible unstable if J(w∗) ≺ 0 strict saddle if J(w∗) has an eigenvalue with negative real part (Strict saddles are a subset of unstable fixed points.)

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 23 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Fixed points and equilibria

local Nash equilibrium: NE only within a neighbourhood: if, for all i, there exists a neighborhood Ui of w∗

i

such that ℓi(w′

i, w∗ −i) ≥ ℓi(w∗ i , w∗ −i) for w′ i ∈ Ui.

A fixed point w∗ with ξ(w∗) = 0 is stable if J(w∗) 0 and J(w∗) is invertible unstable if J(w∗) ≺ 0 strict saddle if J(w∗) has an eigenvalue with negative real part (Strict saddles are a subset of unstable fixed points.) Properties if general-sum: local NE

• ⇐

⇒ stable if zero-sum: local NE ⇐ ⇒ stable

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 23 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Decomposition of games (1/2)

Any matrix decomposes uniquely as M = S + A where S = 1

2(M + M⊺) and A = 1 2(M − M⊺)

S − S⊺ ≡ 0 is symmetric and A + A⊺ ≡ 0 is antisymmetric.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 24 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Decomposition of games (2/2)

We can apply this decomposition to the jacobian J(w) = S(w) + A(w) Let’s define: potential game: the jacobian is symmetric, i.e. A(w) ≡ 0. hamiltonian game: the jacobian is antisymmetric, i.e. S(w) ≡ 0.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 25 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Optimization on the respective game types

If game is potential: Well studied games in literature. Simultaneous gradient descent on the losses corresponds to gradient descent on a single function. GD on ξ converges to a fixed point that is a local minimum or a saddle.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 26 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Optimization on the respective game types

If game is potential: Well studied games in literature. Simultaneous gradient descent on the losses corresponds to gradient descent on a single function. GD on ξ converges to a fixed point that is a local minimum or a saddle. If game is hamiltonian: Novel contribution of the paper, analyzes properties of hamiltonian games Let H(w) := 1

2ξ(w)2 2

(by theorem 4) GD on H converges to stable fixed point Further gradient can be computed as ∇H = A⊺ξ

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 26 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Symplectic gradient adjustment

We’d like to solve general games, i.e. J(w) = S(w) + A(w) This paper introduces symplectic gradient adjustment (SGA): ξλ := ξ + λ · A⊺ξ.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 27 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

SGA desiderata

SGA satisfies following desiderata:

1 compatible1 with game dynamics: ξλ, ξ = α1 · ξ2; 2 compatible with potential dynamics:

if the game is a potential game then ξλ, ∇φ = α2 · ∇φ2;

3 compatible with Hamiltonian dynamics:

If the game is Hamiltonian then ξλ, ∇H = α3 · ∇H2;

4 attracted to stable equilibria:

in neighborhoods where S ≻ 0, require θ(ξλ, ∇H) ≤ θ(ξ, ∇H);

5 repelled by unstable equilibria:

in neighborhoods where S ≺ 0, require θ(ξλ, ∇H) ≥ θ(ξ, ∇H). for some α1, α2, α3 > 0.

1Two nonzero vectors are compatible if they have positive inner product. Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 28 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

SGA pseudocode

Input: losses L = {ℓi}n

i=1, weights W = {wi}n i=1

ξ ←

• gradient(ℓi, wi) for (ℓi, wi) ∈ (L, W)
• A⊺ξ ← get sym adj(L, W)

// appendix A if align then ∇H ←

• gradient( 1

2ξ2, w) for w ∈ W)

• λ ← sign
• 1

d ξ, ∇HA⊺ξ, ∇H + ǫ

• // ǫ = 1

10

else λ ← 1 end if Output: ξ + λ · A⊺ξ // plug into any optimizer

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 29 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

SGA (pytorch)

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 30 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Experiment: higher learning rates

GRADIENT DESCENT SGA 0.032 0.1 0.01 learning rate

SGA allows faster and more robust convergence to stable fixed points than vanilla gradient descent in the presence of ’rotational forces’, by bending the direction of descent towards the fixed point.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 31 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Experiment: convergence with various learning rates

LEARNING RATE LEARNING RATE STEPS TO CONVERGE LOSS AFTER 250 STEPS OMD SGA OMD SGA

Comparison of SGA with optimistic mirror descent. SGA with λ = 1 (Left): iterations to convergence, with maximum 250 iters. (Right): average absolute value of losses over the last 10 iterations, i.e. iterations 240-250

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 32 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Experiment: convergence to all modes (1/2)

Ground truth:

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 33 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Experiment: convergence to all modes (2/2)

GRADIENT DESCENT 4000 6000 8000 2000 Iteration: SGA without ALIGNMENT SGA with ALIGNMENT CONSENSUS OPTIMIZATION CONSENSUS OPTIMIZATION with ALIGNMENT

learning rate 9e-5 learning rate 9e-5 learning rate 9e-5 learning rate 1e-4 learning rate 9.25e-5

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 34 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

My experiments: learning in NFGs - 2x2 game

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 35 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

My experiments: learning in NFGs - 10x10 game

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 36 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Notes

Adjustment update independently discovered in Global Convergence to the Equilibrium of GAS using Variational Inequalities, I. Gemp, S. Mahadevan Arxiv (called Crossing-the-Curl) Things I’ve left out:

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 37 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Notes

Adjustment update independently discovered in Global Convergence to the Equilibrium of GAS using Variational Inequalities, I. Gemp, S. Mahadevan Arxiv (called Crossing-the-Curl) Things I’ve left out: Sign and magnitude of the adjustment λ

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 37 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Notes

Adjustment update independently discovered in Global Convergence to the Equilibrium of GAS using Variational Inequalities, I. Gemp, S. Mahadevan Arxiv (called Crossing-the-Curl) Things I’ve left out: Sign and magnitude of the adjustment λ Properties between Hamiltonian and zero-sum games

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 37 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Notes

Adjustment update independently discovered in Global Convergence to the Equilibrium of GAS using Variational Inequalities, I. Gemp, S. Mahadevan Arxiv (called Crossing-the-Curl) Things I’ve left out: Sign and magnitude of the adjustment λ Properties between Hamiltonian and zero-sum games Jacobian-vector product can be computed efficiently

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 37 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Notes

Adjustment update independently discovered in Global Convergence to the Equilibrium of GAS using Variational Inequalities, I. Gemp, S. Mahadevan Arxiv (called Crossing-the-Curl) Things I’ve left out: Sign and magnitude of the adjustment λ Properties between Hamiltonian and zero-sum games Jacobian-vector product can be computed efficiently Type consistency check of the game reveals if gradient adjustment is needed

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 37 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Notes

Adjustment update independently discovered in Global Convergence to the Equilibrium of GAS using Variational Inequalities, I. Gemp, S. Mahadevan Arxiv (called Crossing-the-Curl) Things I’ve left out: Sign and magnitude of the adjustment λ Properties between Hamiltonian and zero-sum games Jacobian-vector product can be computed efficiently Type consistency check of the game reveals if gradient adjustment is needed Relation to differential and symplectic geometry and Hodge decomposition

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 37 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Takeaways

Multiobjective optimization: Simultaneous gradient is not guaranteed to converge to local minima in general games (only in potential games). Causes mode collapse and mode-hopping Symplectic gradient adjustment Simple plug-n-play adjustment for optimization Supports multiplayer settings n ≥ 2 Implementation for Tensorflow/Pytorch available Computes stable fixed points which are not necessarily local NE

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 38 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Motivation

Applying latent optimization leads to more diverse image generation. (a) (b)

Figure: Samples from BigGAN-deep (a) and LOGAN (b) with similarly high IS.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 39 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Gradient update in GANs

min

θD

max

θG

Ex∼p(x) [hD(D (x; θD))] + Ez∼p(z) [hG(D (G (z; θG) ; θD))]

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 40 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Gradient update in GANs

min

θD

max

θG

Ex∼p(x) [hD(D (x; θD))] + Ez∼p(z) [hG(D (G (z; θG) ; θD))] LOGAN settings hD(t) = −t, hG(t) = t: min

θD

max

θG

Ex∼p(x) [−D (x; θD))] + Ez∼p(z) [D (G (z; θG) ; θD)] Notation: f (z) := D (G (z; θG) ; θD) Gradient update for discriminator and generator: ξ = ∂f (z) ∂θD , −∂f (z) ∂θG T (1)

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 40 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

SGA for GANs

SGA has extra term: ξλ := ξ + λ · A⊺ ξ Applying SGA to GANs: ξλ :=        ∂f (z) ∂θD + λ ∂2f (z) ∂θG ∂θD T ∂f (z) ∂θG −∂f (z) ∂θG + λ ∂2f (z) ∂θD ∂θG T ∂f (z) ∂θD       

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 41 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Latent optimized sources

Idea: instead of using z, use latent optimized z′ := z + ∆z

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 42 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Computing ∆z

Two approaches to compute ∆z: Gradient Descent: ∆z = α g Natural Gradient Descent (approximation of 2nd order method): ∆z = α β + g2 g with g = ∂f (z)

∂z .

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 43 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Latent Optimised GANs with Automatic Differentiation

Input: data distribution p(x) latent distribution p(z) D (·; θD), G (·; θG) learning rate α, batch size N repeat Initialise discriminator and generator parameters θD, θG for i = 1 to N do Sample z ∼ p(z), x ∼ p(x) Compute the gradient ∂D(G(z))

∂z

and obtain ∆z using GD or NGD Compute the optimized latent z′ ← [z + ∆z] a Compute generator loss L(i)

G = −D(G(z′))

Compute discriminator loss L(i)

D = D(G(z′)) − D(x)

end for Compute batch losses LG = 1

N

N

i=1 L(i) G and LD = 1 N

N

i=1 L(i) D

Update θD and θG with the gradients ∂LD

∂θD , ∂LG ∂θG

until reaches the maximum training steps

a[·] indicates clipping the value between −1 and 1

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 44 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Latent optimized dynamics

How does z′ = z + ∆z = z + α ∂f (z)

∂z

change the original dynamics ξ =

• ∂f (z)

∂θD , − ∂f (z) ∂θG

T ?

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 45 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Latent optimized dynamics

How does z′ = z + ∆z = z + α ∂f (z)

∂z

change the original dynamics ξ =

• ∂f (z)

∂θD , − ∂f (z) ∂θG

T ? ξ′ =    

∂f (z′) ∂θD

+

• ∂∆z

∂θD

T

∂f (z′) ∂∆z

− ∂f (z′)

∂θG

−

• ∂∆z

∂θG

T

∂f (z′) ∂∆z

    =    

∂f (z′) ∂θD

+ α

• ∂2f (z)

∂z∂θD

T

∂f (z′) ∂z′

− ∂f (z′)

∂θG

− α

• ∂2f (z)

∂z∂θG

T

∂f (z′) ∂z′

   

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 45 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Comparison of the two dynamics

SGA: ξλ =        ∂f (z) ∂θD + λ ∂2f (z) ∂θG ∂θD T ∂f (z) ∂θG −∂f (z) ∂θG + λ ∂2f (z) ∂θD ∂θG T ∂f (z) ∂θD        Latent optimization: ξ′ =        ∂f (z′) ∂θD + α ∂2f (z) ∂z∂θD T ∂f (z′) ∂z′ −∂f (z′) ∂θG − α ∂2f (z) ∂z∂θG T ∂f (z′) ∂z′       

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 46 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Comparison of the two dynamics

SGA: ξλ =        ∂f (z) ∂θD + λ ∂2f (z) ∂θG ∂θD T ∂f (z) ∂θG −∂f (z) ∂θG + λ ∂2f (z) ∂θD ∂θG T ∂f (z) ∂θD        Latent optimization: ξ′ =        ∂f (z′) ∂θD + α ∂2f (z) ∂z∂θD T ∂f (z′) ∂z′ −∂f (z′) ∂θG − α ∂2f (z) ∂z∂θG T ∂f (z′) ∂z′       

Approximates SGA using only second-order derivatives with respect to the latent z and parameters of the discriminator and generator separately.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 46 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Comparison of the two dynamics

SGA: ξλ =        ∂f (z) ∂θD + λ ∂2f (z) ∂θG ∂θD T ∂f (z) ∂θG −∂f (z) ∂θG + λ ∂2f (z) ∂θD ∂θG T ∂f (z) ∂θD        Latent optimization: ξ′ =        ∂f (z′) ∂θD + α ∂2f (z) ∂z∂θD T ∂f (z′) ∂z′ −∂f (z′) ∂θG − α ∂2f (z) ∂z∂θG T ∂f (z′) ∂z′       

Approximates SGA using only second-order derivatives with respect to the latent z and parameters of the discriminator and generator separately. The second order terms involving parameters of both the discriminator and the generator – which are extremely expensive to compute – are not used.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 46 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Comparison of the two dynamics

SGA: ξλ =        ∂f (z) ∂θD + λ ∂2f (z) ∂θG ∂θD T ∂f (z) ∂θG −∂f (z) ∂θG + λ ∂2f (z) ∂θD ∂θG T ∂f (z) ∂θD        Latent optimization: ξ′ =        ∂f (z′) ∂θD + α ∂2f (z) ∂z∂θD T ∂f (z′) ∂z′ −∂f (z′) ∂θG − α ∂2f (z) ∂z∂θG T ∂f (z′) ∂z′       

Approximates SGA using only second-order derivatives with respect to the latent z and parameters of the discriminator and generator separately. The second order terms involving parameters of both the discriminator and the generator – which are extremely expensive to compute – are not used. For latent z’s with dimensions typically used in GANs, the second order terms can be computed efficiently.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 46 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Comparison of the two dynamics

SGA: ξλ =        ∂f (z) ∂θD + λ ∂2f (z) ∂θG ∂θD T ∂f (z) ∂θG −∂f (z) ∂θG + λ ∂2f (z) ∂θD ∂θG T ∂f (z) ∂θD        Latent optimization: ξ′ =        ∂f (z′) ∂θD + α ∂2f (z) ∂z∂θD T ∂f (z′) ∂z′ −∂f (z′) ∂θG − α ∂2f (z) ∂z∂θG T ∂f (z′) ∂z′       

Approximates SGA using only second-order derivatives with respect to the latent z and parameters of the discriminator and generator separately. The second order terms involving parameters of both the discriminator and the generator – which are extremely expensive to compute – are not used. For latent z’s with dimensions typically used in GANs, the second order terms can be computed efficiently. In short, latent optimisation efficiently couples the gradients of the D and G, as prescribed by SGA, but using the much lower-dimensional latent source z.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 46 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Evaluation

Evaluation of GANs is hard

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 47 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Evaluation

Evaluation of GANs is hard If we could specify it well enough, it could become the objective!

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 47 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Evaluation

Evaluation of GANs is hard If we could specify it well enough, it could become the objective! Best response against a fixed discriminator may not be what we want.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 47 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Evaluation

Evaluation of GANs is hard If we could specify it well enough, it could become the objective! Best response against a fixed discriminator may not be what we want. Scores used in LOGAN: Inception Score (IS) a measure the “objectness” of a generated image, computed by model with Inception architecture (Salimans 2016)

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 47 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Evaluation

Evaluation of GANs is hard If we could specify it well enough, it could become the objective! Best response against a fixed discriminator may not be what we want. Scores used in LOGAN: Inception Score (IS) a measure the “objectness” of a generated image, computed by model with Inception architecture (Salimans 2016) Fr´ echet Inception Distance (FID) todo evaluate the similarity between two dataset of images (Heusel 2017)

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 47 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Experiments: IS vs FID

. BigGAN: architecture based on residual blocks regularisation mechanisms and self-attention

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 48 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Experiments: IS vs FID

. BigGAN: architecture based on residual blocks regularisation mechanisms and self-attention LOGAN improves the adversarial dynamics during training.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 48 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Experiments: IS vs FID

. BigGAN: architecture based on residual blocks regularisation mechanisms and self-attention LOGAN improves the adversarial dynamics during training. No need to optimize latents for evaluation.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 48 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 49 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Samples - High IS (points C,D)

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 50 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Samples - High IS (points C,D)

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 51 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Samples - High IS (points C,D)

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 52 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Samples - High IS (points C,D)

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 53 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Samples - Low FID (points A,B)

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 54 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Samples - Low FID (points A,B)

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 55 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Samples - Low FID (points A,B)

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 56 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Samples - Low FID (points A,B)

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 57 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Negative review

The LOGAN paper was not accepted to ICLR 2020. Main criticism: authors do not compare to SGA. The reviewers claim that computing the Hessian vector product is not that expensive and they should’ve done it.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 58 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Negative review

The LOGAN paper was not accepted to ICLR 2020. Main criticism: authors do not compare to SGA. The reviewers claim that computing the Hessian vector product is not that expensive and they should’ve done it. Authors did not cite concurrent submission to ICLR 2020 :)

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 58 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Notes

I did not talk about: Relation with Unrolled GANs, and that SGA can be seen as approximating Unrolled GANs Relation with stochastic approximation with two timescales.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 59 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Takeaways

Mainly experimental paper. Introduces a simplfication of the SGA update. + Coupling the generator/discriminator via latent optimization improves sample quality.

• Latent optimization has a higher cost per iteration

(they claim about 3x slower).

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 60 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Conclusion

Multiobjective optimization is an area of game theory. Game theory offers various solution concepts. It’s not clear what is the “best” solution concept. It’s hard to converge to Nash equilibria. We might be satisfied with stable fixed points. It’s possible to update simultaneous gradient descent to converge to stable FP. This is called symplectic gradient adjustment (SGA). SGA can be applied to GANs of any architecture. LOGAN is an approximation of SGA to GANs. LOGAN generates higher quality and more diverse images.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 61 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Questions?

Hannan consistency of SGA? Convergence to correlated equilibria? Relation to algorithms like exploitability descent? Find a simple failure case: finds local NE but not a global NE What GANs (do not) use coupling and how it influences performance?

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 62 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Strategy profile σ = (σi, σ−i) is an ǫ-Nash equilibrium (ǫ-NE) if (∀i ∈ N) : ui(σ) ≥ max

σ′

i ∈Σi

ui(σ′

i, σ−i) − ǫ.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 63 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Computation of NE (1/2)

For two-player zero-sum normal-form games

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 64 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Computation of NE (1/2)

For two-player zero-sum normal-form games Exact NE: linear programming

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 64 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Computation of NE (1/2)

For two-player zero-sum normal-form games Exact NE: linear programming Approximate ǫ−NE (iterative algorithms): Regret-matching Regret matching+ Hedge ...

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 64 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Computation of NE (1/2)

For two-player zero-sum normal-form games Exact NE: linear programming Approximate ǫ−NE (iterative algorithms): Regret-matching Regret matching+ Hedge ... Goal of regret minimization: ensure that average regret approaches zero, regardless of the opponents’ strategies.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 64 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Computation of NE (2/2)

Any algorithm that is external regret minimizing, or ǫ-Hannan consistent (HC) converges to 2ǫ-Nash eq.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 65 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Computation of NE (2/2)

Any algorithm that is external regret minimizing, or ǫ-Hannan consistent (HC) converges to 2ǫ-Nash eq. Player’s strategy is ǫ-HC in repeated NFG if against any opponent’s strategy it’s external regret is ≤ ǫ in the limit. I.e. cumulative regret grows sublinearly, so if we take average strategy it’s regret approaches zero.

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 65 / 65

Introduction Background Differentiable Game Mechanics LOGAN Conclusion Extras

Computation of NE (2/2)

Any algorithm that is external regret minimizing, or ǫ-Hannan consistent (HC) converges to 2ǫ-Nash eq. Player’s strategy is ǫ-HC in repeated NFG if against any opponent’s strategy it’s external regret is ≤ ǫ in the limit. I.e. cumulative regret grows sublinearly, so if we take average strategy it’s regret approaches zero. Most of the work we do in our group is based on extensions of regret-matching to sequential games (CFR).

Michal ˇ Sustr Reading group: Latent Optimized GANs January 9th, 2020 65 / 65