Unifying Orthogonal Monte Carlo Methods From Kacs Random Walks To - - PowerPoint PPT Presentation

▶
unifying orthogonal monte carlo methods
SMART_READER_LITE
LIVE PREVIEW

Unifying Orthogonal Monte Carlo Methods From Kacs Random Walks To - - PowerPoint PPT Presentation

Mar 21, 2024 17 likes •251 views

Unifying Orthogonal Monte Carlo Methods From Kacs Random Walks To Hadamard Multi Rademachers Krzysztof Choromanski, Mark Rowland Wenyu Chen, Adrian Weller The Phenomenon of Orthogonal Monte Carlo Estimators Estimation task: Applications:

slide-1
SLIDE 1

Unifying Orthogonal Monte Carlo Methods

Krzysztof Choromanski, Mark Rowland Wenyu Chen, Adrian Weller From Kac’s Random Walks To Hadamard Multi Rademachers

slide-2
SLIDE 2

The Phenomenon of Orthogonal Monte Carlo Estimators

Estimation task:

isotropic distribution (e.g. Gaussian)

Applications:

  • dimensionality reduction

(JLT-mechanisms)

  • scaling kernel methods

(random feature maps)

  • hashing algorithms

(e.g. LSH)

  • (sliced) Wasserstein

distances (WGANs, autoencoders...)

  • reinforcement learning

(ES algorithms)

  • and many, many more...

Standard MC approach:

slide-3
SLIDE 3

The Phenomenon of Orthogonal Monte Carlo Estimators

Estimation task: The Orthogonal Trick: guarantees unbiasedness

  • ften implies better

accuracy Sampling from the Haar measure on the O(d) group # of samples of the MC estimator <= dim Expensive: O(n^3 time) isotropic distribution (e.g. Gaussian)

slide-4
SLIDE 4

Towards Computational Efficiency: The Zoo of Approximate MCs

slide-5
SLIDE 5

Towards Computational Efficiency: The Zoo of Approximate MCs

...

slide-6
SLIDE 6

Towards Computational Efficiency: The Zoo of Approximate MCs

...

slide-7
SLIDE 7

Towards Computational Efficiency: The Zoo of Approximate MCs

... ...

slide-8
SLIDE 8

Towards Computational Efficiency: The Zoo of Approximate MCs

... ...

slide-9
SLIDE 9

Towards Computational Efficiency: The Zoo of Approximate MCs

... ...

slide-10
SLIDE 10

Towards Computational Efficiency: The Zoo of Approximate MCs

... ...

size N x N size N/2 x N/2

Constraints:

slide-11
SLIDE 11

Towards Computational Efficiency: The Zoo of Approximate MCs

... ...

size N x N size N/2 x N/2

  • Constraints:
slide-12
SLIDE 12

Towards Computational Efficiency: The Zoo of Approximate MCs

... ...

size N x N size N/2 x N/2

  • Constraints:
slide-13
SLIDE 13

On the Hunt for the Unifying Theory: The World of Givens Reflections and Rotations

Givens

rotations

Givens

reflections reflection in the jth coordinate

Kac’s random walk matrices Hadamard-Rademacher Chains

slide-14
SLIDE 14

On the Hunt for the Unifying Theory: The World of Givens Reflections and Rotations

Kac’s random walk matrices Hadamard-Rademacher Chains

slide-15
SLIDE 15

On the Hunt for the Unifying Theory: The World of Givens Reflections and Rotations

Hadamard-MultiRademachers Butterfly Matrices

slide-16
SLIDE 16

First Theoretical Results for Free-Lunch Phenomenon in the Nonlinear Regime

slide-17
SLIDE 17

First Theoretical Results for Free-Lunch Phenomenon in the Nonlinear Regime

Still more accurate estimator than unstructured MC baseline

slide-18
SLIDE 18

First Theoretical Results for Free-Lunch Phenomenon in the Nonlinear Regime

Log-Linear Time Complexity (unstructured MC baseline has quadratic)

slide-19
SLIDE 19

First Theoretical Results for Free-Lunch Phenomenon in the Nonlinear Regime

Analysis of the Total Variation Distance between Haar measure on d-sphere and measure induced by standard Kac’s random walk on d-sphere

Pillai, Smith 2016 Kac’s random walk on d-sphere mixes in O(d log d) steps

estimator estimated value

slide-20
SLIDE 20

First Theoretical Results for Free-Lunch Phenomenon in the Nonlinear Regime

Analysis of the Total Variation Distance between Haar measure on d-sphere and measure induced by standard Kac’s random walk on d-sphere

Pillai, Smith 2016 Kac’s random walk on d-sphere mixes in O(d log d) steps

More careful analysis of the LHS estimator estimated value

slide-21
SLIDE 21

How Does It Work In Practice ?

Maximum Mean Discrepancy Experiment Kernel Approximation via Random Features Reinforcement Learning via ES-methods Accuracy Computational Efficiency

slide-22
SLIDE 22

How Does It Work In Practice ?

Maximum Mean Discrepancy Experiment Kernel Approximation via Random Features Reinforcement Learning via ES-methods Accuracy Computational Efficiency

slide-23
SLIDE 23

Thank you for your attention !