GRADUATE FELLOW FAST FORWARD Bill Dally, Chief Scientist and SVP - - PowerPoint PPT Presentation

▶
graduate fellow fast forward
SMART_READER_LITE
LIVE PREVIEW

GRADUATE FELLOW FAST FORWARD Bill Dally, Chief Scientist and SVP - - PowerPoint PPT Presentation

Sep 09, 2022 101 likes •1.2k views

GRADUATE FELLOW FAST FORWARD Bill Dally, Chief Scientist and SVP Research, NVIDIA Thursday, March 21, 2019 GRADUATE FELLOWSHIP PROGRAM Funding for Ph.D. students revolutionizing disciplines with the GPU Engage: Build mindshare Facilitate

slide-1
SLIDE 1

Bill Dally, Chief Scientist and SVP Research, NVIDIA Thursday, March 21, 2019

GRADUATE FELLOW FAST FORWARD

slide-2
SLIDE 2

2

GRADUATE FELLOWSHIP PROGRAM

Funding for Ph.D. students revolutionizing disciplines with the GPU

Engage:

  • Build mindshare
  • Facilitate recruiting

Learn:

  • Keep a finger on the pulse of leading academic research
  • Keep up with all the applications that are powered by GPUs

Leverage:

  • Track relevant research
  • Help to guide researchers working on relevant problems
slide-3
SLIDE 3

3

GRADUATE FELLOWSHIP PROGRAM

Eligibility/Application Process:

  • Ph.D. candidates in at least their 2nd year
  • Nomination(s) by Professor(s)/Advisor
  • 1-2 page research proposal

Selection Process:

  • Committee of NVIDIA scientists and engineers review applications
  • Applications evaluated for originality, potential, and relevance

165 Graduate Fellowships awarded -- $4.9M since program inception in 2002

slide-4
SLIDE 4

4

CURRENT 2018-2019 GRAD FELLOWS

Adam Stooke, UCB Ana Serrano, Universidad de Zaragoza Aishwarya Agrawal, Georgia Tech Andy Zeng, Princeton Daniel George, UIUC Abhishek Badki, UCSB

slide-5
SLIDE 5

5

CURRENT 2018-2019 GRAD FELLOWS

Philippe Tillet, Harvard Zhilin Yang, CMU Xun Huang, Cornell William Yuan, Harvard NVIDIA Foundation Fellow Huizi Mao, Stanford

slide-6
SLIDE 6

6

CURRENT 2018-2019 GRAD FELLOW FINALISTS

  • Chenxi Liu, Johns Hopkins University
  • Jake Zhao, New York University
  • Mario Drummond, EPFL
  • Mark Buckler, Cornell University
  • Steve Bako, UC Santa Barbara
slide-7
SLIDE 7

7

AGENDA

  • Grad Fellow Fast Forward Talks, 3 mins each:
  • Aishwarya Agrawal, Georgia Tech
  • Abhishek Badki, UC Santa Barbara
  • Daniel George, Univ of Illinois Urbana-Champaign
  • Xun Huang, Cornell
  • Huizi Mao, Stanford
  • Ana Serrano, Univ de Zaragoza
  • Philippe Tillet, Harvard
  • Zhilin Yang, CMU
  • William Yuan, Harvard
  • Certificates/Photographs
  • NVIDIA Foundation Overview
  • Announcement of the 2019-2020 Fellows & Finalists
slide-8
SLIDE 8

8

AISHWARYA AGRAWAL, GEORGIA TECH

slide-9
SLIDE 9

Aishwarya Agrawal, Georgia Tech

GENERATING DIVERSE PROGRAMS WITH INSTRUCTION CONDITIONED REINFORCED ADVERSARIAL LEARNING

March 21, 2019

slide-10
SLIDE 10

10

10

TASK

There is a yellow cube. add object, cube, yellow, small, at (8,14)

Renderer Agent

slide-11
SLIDE 11

11

11

add object, cube, yellow, large, at (12,17) add object, cube, yellow, small, at (22,12) There is a yellow cube. add object, cube, yellow, small, at (8,14)

Agent

TASK

slide-12
SLIDE 12

12

12

Reward Learning Rich Action Space Diverse Outputs

TECHNICAL CHALLENGES

slide-13
SLIDE 13

13

13

Draw 9. Paint five.

DOMAIN 1: MNIST DIGIT PAINTING

slide-14
SLIDE 14

14

14

There is a green cylinder. There is a large sphere.

DOMAIN 2: 3D SCENE CONSTRUCTION

slide-15
SLIDE 15

15

15

Policy Network (Generator) Environment (Renderer)

Instruction Program Final Image Instruction Example Goal Image Reward

Discriminator

Intermediate Image Extending Ganin et al., ICML18

APPROACH

slide-16
SLIDE 16

16

16

Policy Network (Generator) Environment (Renderer)

Instruction Program Final Image Instruction Example Goal Image Reward

Discriminator

Intermediate Image Extending Ganin et al., ICML18

APPROACH

All of the model training uses GPUs!

slide-17
SLIDE 17

17

17

DOMAIN 1: MNIST DIGIT PAINTING

Create zero Put 1 Paint two Draw 3 Add four Draw 5 Paint six Put 7 Create eight Add 9

slide-18
SLIDE 18

18

18

DOMAIN 2: 3D SCENE CONSTRUCTION

There is a small sphere. There is a large cylinder. There is a yellow cube.

slide-19
SLIDE 19

19

19

THANKS! COME TO OUR POSTER!

slide-20
SLIDE 20
slide-21
SLIDE 21

21

ABHISHEK BADKI, UC SANTA BARBARA

slide-22
SLIDE 22

Abhishek Badki, University of California, Santa Barbara

COMPUTATIONAL ZOOM: A FRAMEWORK FOR POST-CAPTURE IMAGE COMPOSITION

March 21, 2019

slide-23
SLIDE 23

23

16 mm, close 35 mm, far 105 mm, farthest

IMAGE COMPOSITION

slide-24
SLIDE 24

24

IMAGE COMPOSITION

slide-25
SLIDE 25

25

OUR GOAL

Post-Capture Image Composition

Input image stack/video

slide-26
SLIDE 26

26

OUR GOAL

Post-Capture Image Composition

Computational zoom results

slide-27
SLIDE 27

27

OUR GOAL

Post-Capture Image Composition

Computational zoom results

slide-28
SLIDE 28

28

OUR GOAL

Post-Capture Image Composition

Computational zoom results

slide-29
SLIDE 29

29

OUR GOAL

Post-Capture Image Composition

Computational zoom results

slide-30
SLIDE 30

30

MULTI-PERSPECTIVE CAMERA MODELS

Allow novel image compositions of the scene

slide-31
SLIDE 31

31

MULTI-PERSPECTIVE IMAGE SYNTHESIS

Multi- perspective rendering Structure from motion 3D reconstruction

slide-32
SLIDE 32

32

MULTI-PERSPECTIVE IMAGE SYNTHESIS

Multi- perspective rendering Structure from motion 3D reconstruction

slide-33
SLIDE 33

33

MULTI-PERSPECTIVE IMAGE SYNTHESIS

Multi- perspective rendering Structure from motion 3D reconstruction

Depth map Normal map

slide-34
SLIDE 34

34

MULTI-PERSPECTIVE IMAGE SYNTHESIS

Multi- perspective rendering Structure from motion 3D reconstruction

Multi-perspective results

Multi-perspective camera model

Images Depth-maps

slide-35
SLIDE 35

35

  • ur result with different image compositions
slide-36
SLIDE 36
slide-37
SLIDE 37

37

DANIEL GEORGE, UIUC

slide-38
SLIDE 38

Daniel George, Google X / University of Illinois at Urbana-Champaign

Deep Learning for Gravitational Wave and Multimessenger Astrophysics

March 21, 2019

Link to full slides: tiny.cc/phd-defense

slide-39
SLIDE 39

39

GRAVITATIONAL WAVES

SXS

Source: ligo.org

slide-40
SLIDE 40

40

slide-41
SLIDE 41

41

slide-42
SLIDE 42

42

slide-43
SLIDE 43

43

slide-44
SLIDE 44

44

slide-45
SLIDE 45

45

slide-46
SLIDE 46

46

slide-47
SLIDE 47

Link to full slides: tiny.cc/phd-defense

slide-48
SLIDE 48

48

XUN HUANG, CORNELL

slide-49
SLIDE 49

Xun Huang, Cornell University

MULTIMODAL UNSUPERVISED IMAGE-TO-IMAGE TRANSLATION

March 21, 2019

slide-50
SLIDE 50

50

50

slide-51
SLIDE 51

51

UNSUPERVISED IMAGE-TO-IMAGE TRANSLATION

slide-52
SLIDE 52

52

UNIMODAL OR MULTIMODAL

Unimodal Multimodal

……

slide-53
SLIDE 53

53

TOWARDS MULTIMODALITY

We assume the image representation space can be disentangled into:

The content space that are shared by both domains. The style space that are specific for each domain.

To sample a diverse set of outputs, we keep the content code of the input and randomly sample style codes from the target style space.

Unsupervised Learning of Disentangled Latent Space

slide-54
SLIDE 54

54

METHODS

We use auto-encoders to encode an image into its latent code and reconstruct the image from the latent code. We employ Generative Adversarial Networks (GANs) to ensure the translated images are realistic. Each model is trained on a NVIDIA Tesla V100 GPU with 16GB memory.

slide-55
SLIDE 55

55

RESULTS (SKETCHES <-> PHOTO)

slide-56
SLIDE 56

56

RESULTS (ANIMALS)

slide-57
SLIDE 57

57

RESULTS (SUMMER <-> WINTER)

slide-58
SLIDE 58
slide-59
SLIDE 59

59

HUIZI MAO, STANFORD

slide-60
SLIDE 60

Huizi Mao, Stanford University

CATDET: AN EFFICIENT VIDEO OBJECT DETECTION SYSTEM

March 21, 2019 To appear on SysML 2019

slide-61
SLIDE 61

61

61

OBJECT DETECTION FROM VIDEO

Goal: to locate and classify objects in a video stream Difficulty: frame-by-frame detection is compute-intensive

slide-62
SLIDE 62

62

62

CATDET: CASCADED TRACKED DETECTOR

CaTDet is a system to save computations of CNN-based detectors Goal: run large CNN models only on selected regions Output Input Detector Network Output Input Refinement Network Proposal Network Tracker

Single-image detector CaTDet Same parameters Smaller workload Little overhead

slide-63
SLIDE 63

63

63

EXAMPLE

Come back to the previous example: We only run the refinement network (the expensive one) on selected regions

Frame N Frame N+1

slide-64
SLIDE 64

64

64

RESULTS

Maintain the same mAP on KITTI dataset Reduce the number of arithmetic operations by 5.2x Reduce GPU time by 3.8x (Maxwell TITAN X)

Method mAP Ops(G) GPU time(s) Faster R-CNN Frame-by-frame 0.740 254.3 0.159 CaTDet 0.740 49.3 (5.2x) 0.042 (3.8x)

More results on the SysML 2019 paper: http://www.sysml.cc/doc/2019/111.pdf

slide-65
SLIDE 65
slide-66
SLIDE 66

66

ANA SERRANO, UNIV DE ZARAGOZA

slide-67
SLIDE 67

Ana Serrano, Universidad de Zaragoza

MOTION PARALLAX FOR VR VIDEOS

March 21, 2019

slide-68
SLIDE 68

68

EXPERIENCES IN VIRTUAL REALITY

SuperHOT VR

SUPERHOT Team

Miyubi

Felix & Paul Studios

Real-world recorded content vs. CG content

slide-69
SLIDE 69

69

69

RECORDING CONTENT FOR VR

Commercially available VR cameras

Kandao Obsidian Yi Halo Facebook Surround360 Nokia Ozo

slide-70
SLIDE 70

70

VIDEO RECORDED FROM A FIXED CAMERA

How to render the scene from different head positions?

Scene recorded from a fixed camera position New camera view to show to the user

slide-71
SLIDE 71

71

Close-up VR view (stereo)

Enabling motion parallax for VR video

OUR APPROACH: LAYERED VIDEO

slide-72
SLIDE 72

72

[Serrano et al. 2019] Motion parallax for 360 RGBD video Optimized for real-time GPU rendering of novel camera views Layered video representation for storing additional scene information Independent of a specific hardware, or camera setup User studies confirm a more compelling viewing experience

OUR APPROACH: LAYERED VIDEO

Enabling motion parallax for VR video

slide-73
SLIDE 73
slide-74
SLIDE 74

74

PHILIPPE TILLET, HARVARD

slide-75
SLIDE 75

Philippe Tillet, Harvard University

Triton: An Imperative Array Language and Compiler for Efficient Tiled Computations in Machine Learning Workloads

March 21, 2019

slide-76
SLIDE 76

76

MOTIVATIONS

slide-77
SLIDE 77

77

EXISTING SOLUTIONS

TensorFlow, PlaidML, Tensor Comprehensions, TVM ...

slide-78
SLIDE 78

78

EXISTING SOLUTIONS

GPU Performance

slide-79
SLIDE 79

79

MY SOLUTION

  • Existing functional languages lack flexibility

Cannot specify how tensors are decomposed into tiles

  • Existing imperative languages lack abstractive power

Cannot specify what the meaning of scalar variables is I developed Triton: a language & compiler which adds the concept of tile to a CUDA-like imperative programs. Best of both worlds.

Triton

slide-80
SLIDE 80

80

MY SOLUTION

Example

slide-81
SLIDE 81

81

MY SOLUTION

GPU Performance

slide-82
SLIDE 82

82

WE CAN DO MORE!

Dense convolution via implicit matrix multiplication

slide-83
SLIDE 83

83

WE CAN DO MORE!

Performance

slide-84
SLIDE 84
slide-85
SLIDE 85

85

ZHILIN YANG, CMU

slide-86
SLIDE 86

Zhilin Yang, CMU

LEARNING BY GENERATIVE MODELING

March 21, 2019

slide-87
SLIDE 87

87

GENERATIVE MODELING

Given data x, model the probability p(x). Generate data by sampling from p(x). Goals:

  • 1. Accurate, realistic generation

➢ match p(x) and true data p*(x).

  • 2. Generation as a scaffold

➢ use p(x) to improve p(y|x).

slide-88
SLIDE 88

88

OUR NEW MODEL: TRANSFORMER-XL

The State-of-the-art Architecture for Language Modeling

Vanilla Transformer Transformer-XL

Recurrence + relative encodings Going beyond fixed-length contexts

slide-89
SLIDE 89

89

BENEFITS OF TRANSFORMER-XL

Learns longer-range dependency (80% longer than RNNs and 450% longer than Transformers) Up to 1,800x faster than Transformers during LM evaluation More accurate at prediction on both long and short sequences Able to generate reasonably coherent, novel text articles with thousands of tokens

slide-90
SLIDE 90

90

STATE-OF-THE-ART LANGUAGE MODELING

Perplexity/bpc (the lower the better) measures how well a model predicts a sample. Part of training runs on GPUs.

20.5 23.5 18.3 21.8 17 18 19 20 21 22 23 24 WikiText-103 One Billion Word Perplexity Previous Best Transformer-XL 1.06 1.13 0.99 1.08 0.95 0.97 0.99 1.01 1.03 1.05 1.07 1.09 1.11 1.13 1.15 enwik8 text8 bpc Previous Best Transformer-XL

slide-91
SLIDE 91

91

TEXT GENERATED BY TRANSFORMER-XL

In July 1805, the French 1st Army entered southern Italy. The army, under the command of Marshal Marmont, were reinforced by a few battalions of infantry under Claude General Auguste de Marmont at the town of Philippsburg and another battalion at Belluno. On 17 September 1805, the army marched from Belluno towards Krems. By 29 September, they had reached… … On 9 October the French Army … on 10 October, he launched his attack … On 25 October, Merveldt left Styria for Tyrol … and defeated the Austrians at the Battle of Hohenlinden on 28 October … The Battle of Warsaw was fought on 23 November 1805 … …

Trained on a small 100M-token dataset.

Long-range dependency: ➢ Able to keep track of time. ➢ Reasonable coherence over thousands of tokens.

slide-92
SLIDE 92

92

BETTER THAN BERT

85.9 92.4 82.9 90.6 91.1 71.7 87.3 94.2 87.9 91.3 92 74.4 70 75 80 85 90 95 MNLI SST-2 MRPC QQP QNLI RTE Accuracy (%) BERT Transformer-XL

Preliminary results. We will release more results and details soon.

slide-93
SLIDE 93
slide-94
SLIDE 94

94

WILLIAM YUAN, HARVARD

slide-95
SLIDE 95

William Yuan, Harvard University

EARLY DETECTION OF NEURODEGENERATION WITH DEEP LEARNING

March 21, 2019

slide-96
SLIDE 96

96

NEURODEGENERATION

Oxford FMRIB Neurodegeneration Group

slide-97
SLIDE 97

97

DATA

Unidentifiable Health Insurance Claims Data Tens of millions of individuals → Tens of billions of individual observations Diagnoses/Procedures/Prescriptions Case/Control Study: 1 Year Prediction

Diag Proc Med Proc Observation window Prediction window AD

slide-98
SLIDE 98

98

METHODS

Word2Vec Style Medical Concept Embedding Temporal Convolutional Nets for Sequence Classification with GPU computing Novel Sequence Representation Counterfactual Event Modeling

Beam, et al, 2018

slide-99
SLIDE 99

99

PREDICTION RESULTS (AUC)

Alzheimer’s Disease Parkinson’s Disease

Baseline 0.724 0.754 Event Sequence-only Prediction 0.706 0.721 Randomly Permuted Events 0.693 0.713 Temporal-only Prediction 0.583 0.599

slide-100
SLIDE 100

100

COUNTERFACTUAL MODELING

Phenotype Relative Effect Size

Memory Loss 1.000 Other Persistent Mental Disorders 0.8495 Mild Cognitive Impairment 0.8222 Alzheimer’s Disease* 0.8000 Parkinson’s Disease* 0.7621 Abnormal Involuntary Movements 0.6975 *unobserved by model

slide-101
SLIDE 101
slide-102
SLIDE 102

102

Certificates and Photos

slide-103
SLIDE 103

103

NVIDIA Foundation Compute the Cure

slide-104
SLIDE 104

104

NVIDIA FOUNDATION

slide-105
SLIDE 105

105

Announcing: The New 2019-2020 Grad Fellows And Finalists

slide-106
SLIDE 106

106

NEW 2019-2020 GRAD FELLOWS

Chen-Hsuan Lin, CMU Daniel Gordon, Univ. Washington Ching-An Cheng, Georgia Tech De-An Huang, Stanford Huaizu Jiang, U. Mass. Amherst Bastian Hagedorn, Univ. Münster

slide-107
SLIDE 107

107

NEW 2019-2020 GRAD FELLOWS

Lifan Wu, UC San Diego Mariya Popova, UNC Chapel Hill Siddharth Reddy, UC Berkeley Jeremy Bernstein, CalTech

slide-108
SLIDE 108

108

NEW 2019-2020 GRAD FELLOW FINALISTS

  • Chao-Yuan Wu, UT Austin
  • Kelvin Xu, UC Berkeley
  • Nathan Otterness, UNC Chapel Hill
  • Wengong Jin, MIT
  • Yunzhu Li, MIT
slide-109
SLIDE 109

THANK YOU