New in ML A personal perspective from Academia 5yr PhD Startup in - - PowerPoint PPT Presentation

▶
new in ml a personal perspective from academia
SMART_READER_LITE
LIVE PREVIEW

New in ML A personal perspective from Academia 5yr PhD Startup in - - PowerPoint PPT Presentation

Aug 28, 2023 142 likes •656 views

Joan Bruna Courant Institute and Center for Data Science New in ML A personal perspective from Academia 5yr PhD Startup in Applied experience between Mathematics MsC and PhD One Slide about 5yr Visiting myself experience as positions

slide-1
SLIDE 1

New in ML A personal perspective from Academia

Joan Bruna Courant Institute and Center for Data Science

slide-2
SLIDE 2

One Slide about myself

PhD in Applied Mathematics 5yr Startup experience between MsC and PhD Visiting positions at industrial research lab (Fair) 5yr experience as Assistant Prof. UC Berkeley & NYU

slide-3
SLIDE 3

One Slide about my group

Mathematical Foundations of DL Foundations

  • f RL

Geometric Deep Learning ML and Physics

slide-4
SLIDE 4

One Slide about my group

Mathematical Foundations of DL Foundations

  • f RL

Geometric Deep Learning 10 PhDs 3 postdocs 1 MsC 1 undergrad ML and Physics

slide-5
SLIDE 5

This talk

A PhD Journey

Calibrate expectations and gain independence

You & Peers

Being a good ML citizen and manage peer pressure

Scientific Context

2010s: DL Experimental Revolution

slide-6
SLIDE 6

This talk

A PhD Journey

Calibrate expectations and gain independence

You & Peers

Being a good ML citizen and manage peer pressure

Scientific Context

2010s: DL Experimental Revolution

Aim

Academic Perspective of ML career: a unique moment

slide-7
SLIDE 7

DL golden decade

Machine translation

[Google, ’20]

Computer Vision

[Krizhevsky et al, 12-20]

Games

[AlphaGo, ’16] [AlphaStar, ’19]

Science

[AlphaFold, ’19]

slide-8
SLIDE 8

DL golden decade

Machine translation

[Google, ’20]

Computer Vision

[Krizhevsky et al, 12-20]

Games

[AlphaGo, ’16] [AlphaStar, ’19]

Science

[AlphaFold, ’19]

slide-9
SLIDE 9

DL golden decade

Machine translation

[Google, ’20]

Computer Vision

[Krizhevsky et al, 12-20]

Games

[AlphaGo, ’16] [AlphaStar, ’19]

Science

[AlphaFold, ’19]

None of these problems was thought to be possible!

slide-10
SLIDE 10

Role of Theory so far?

slide-11
SLIDE 11

Role of Theory so far?

Leo Breiman 1995

slide-12
SLIDE 12

Role of Theory so far?

Leo Breiman 1995

None of these questions is fully understood yet!

slide-13
SLIDE 13

Role of Theory so far?

Leo Breiman 1995

None of these questions is fully understood yet!

We need YOUR help!

slide-14
SLIDE 14

Can this go on?

Data Hunger Compute Hunger

slide-15
SLIDE 15

Can this go on?

Data Hunger Compute Hunger Critical Need for Theory

Scaling-up approach unsustainable

slide-16
SLIDE 16

Role of ML Theory?

ML (DL) Theory

slide-17
SLIDE 17

Better Experiments

Guiding principles Robustness Guarantees

Role of ML Theory?

ML (DL) Theory

slide-18
SLIDE 18

Physical Sciences

Scientific Computing Improved Sample Complexity

Better Experiments

Guiding principles Robustness Guarantees

Role of ML Theory?

ML (DL) Theory

slide-19
SLIDE 19

Vignettes

[with B. Menard lab (JHU)]

Cosmology

Build statistical models of early universe that explain expansion and non-Gaussianity

Quantum Mechanics

Parametrise wavefunctions with deep networks having right symmetries

[Pfau et al. ’19]

Distributional Robustness

[Schmidt et al.’19]

slide-20
SLIDE 20

Questions so far?

slide-21
SLIDE 21

A (standard) PhD Journey

[The illustrated guide to a Ph.D,.Matt Might]

Human knowledge

slide-22
SLIDE 22

A (standard) PhD Journey

[The illustrated guide to a Ph.D,.Matt Might]

Human knowledge High- school

slide-23
SLIDE 23

A (standard) PhD Journey

[The illustrated guide to a Ph.D,.Matt Might]

Human knowledge High- school Undergrad

slide-24
SLIDE 24

A (standard) PhD Journey

[The illustrated guide to a Ph.D,.Matt Might]

Human knowledge High- school Undergrad MsC

slide-25
SLIDE 25

A (standard) PhD Journey

[The illustrated guide to a Ph.D,.Matt Might]

Human knowledge High- school Undergrad MsC

Catching up literature

slide-26
SLIDE 26

A (standard) PhD Journey

[The illustrated guide to a Ph.D,.Matt Might]

Human knowledge High- school Undergrad MsC

Catching up literature

slide-27
SLIDE 27

A (standard) PhD Journey

[The illustrated guide to a Ph.D,.Matt Might]

Human knowledge High- school Undergrad MsC

Catching up literature

PhD

Keep pushing!

slide-28
SLIDE 28

Some naive

  • pinions

Startup

slide-29
SLIDE 29

Some naive

  • pinions

No trajectory is better than

  • thers

Startup

slide-30
SLIDE 30

Some naive

  • pinions

No trajectory is better than

  • thers

Non- markovian random process

Startup

slide-31
SLIDE 31

Some naive

  • pinions

No trajectory is better than

  • thers

Non- markovian random process Outcome is important, but path too: human path

Startup

slide-32
SLIDE 32

The ML PhD Journey

ML knowledge Year 0

slide-33
SLIDE 33

D

The ML PhD Journey

ML knowledge Year 1

DL Kernels

slide-34
SLIDE 34

The ML PhD Journey

ML knowledge

GAN IPM

Year 2

slide-35
SLIDE 35

The ML PhD Journey

ML field, year 0 ML field, year 5

slide-36
SLIDE 36

The ML PhD Journey

ML field, year 0 ML field, year 5

Trends are

  • ften

unpredictable

slide-37
SLIDE 37

The ML PhD Journey

ML field, year 0 ML field, year 5

Dent will push knowledge— wherever you land Trends are

  • ften

unpredictable

slide-38
SLIDE 38

The ML PhD Journey

ML field, year 0 ML field, year 5

Dent will push knowledge— wherever you land Trends are

  • ften

unpredictable ML is remarkably broad: profit from it!

slide-39
SLIDE 39

ML Academic Ecosystem

slide-40
SLIDE 40

ML Academic Ecosystem

My initial view

A few stars single- handedly pushing the field forward with breakthroughs

PhD Duties

Follow my advisor Write a few papers

slide-41
SLIDE 41

ML Academic Ecosystem

slide-42
SLIDE 42

ML Academic Ecosystem

My current view

ML research is primarily a team-effort.

slide-43
SLIDE 43

ML Academic Ecosystem

My current view

ML research is primarily a team-effort.

Team can be:

You & advisor Your lab-mates Github org etc.

slide-44
SLIDE 44

ML Academic Ecosystem

My current view

ML research is primarily a team-effort.

Team can be:

You & advisor Your lab-mates Github org etc.

Progress is incremental

Good research should not discriminate.

slide-45
SLIDE 45

ML Academic Ecosystem

My current view

ML research is primarily a team-effort.

Team can be:

You & advisor Your lab-mates Github org etc.

Progress is incremental

Good research should not discriminate.

Research is essentially sustained by students like you!

slide-46
SLIDE 46

Closing Personal Advice

slide-47
SLIDE 47

Closing Personal Advice

Currently much emphasis

  • n designing new

algorithms, models, architectures. Less emphasis

  • n analyzing

current methods that work well in practice.

slide-48
SLIDE 48

Closing Personal Advice

Currently much emphasis

  • n designing new

algorithms, models, architectures. Less emphasis

  • n analyzing

current methods that work well in practice. Current ML competition feels daunting to everyone. Whenever possible, shift focus from papers to ideas. ML Civility A shared responsibility (reviewing, teaching)

slide-49
SLIDE 49

Closing Personal Advice

Feel free to rebuke!

Currently much emphasis

  • n designing new

algorithms, models, architectures. Less emphasis

  • n analyzing

current methods that work well in practice. Current ML competition feels daunting to everyone. Whenever possible, shift focus from papers to ideas. ML Civility A shared responsibility (reviewing, teaching)

slide-50
SLIDE 50

Welcome to the field!