Introduction Lecture slides for Chapter 1 of Deep Learning - - PowerPoint PPT Presentation

▶

May 02, 2023 302 likes •447 views

Introduction Lecture slides for Chapter 1 of Deep Learning www.deeplearningbook.org Ian Goodfellow 2016-09-26 x y Cartesian coordinates Polar coordinates Representations Matter r Figure 1.1 (Goodfellow 2016) Depth: Repeated

SLIDE 1

Introduction

Lecture slides for Chapter 1 of Deep Learning www.deeplearningbook.org Ian Goodfellow 2016-09-26

SLIDE 2

(Goodfellow 2016)

Representations Matter

x y

Cartesian coordinates

r θ

Polar coordinates

Figure 1.1

SLIDE 3

(Goodfellow 2016)

Depth: Repeated Composition

Visible layer (input pixels) 1st hidden layer (edges) 2nd hidden layer (corners and contours) 3rd hidden layer (object parts) CAR PERSON ANIMAL Output (object identity)

Figure 1.2

SLIDE 4

(Goodfellow 2016)

Computational Graphs

x1 x1

σ

w1 w1

×

x2 x2 w2 w2

× +

Element Set

+ × σ

x x w

Element Set Logistic Regression Logistic Regression

Figure 1.3

SLIDE 5

(Goodfellow 2016)

Machine Learning and AI

AI Machine learning Representation learning Deep learning Example: Knowledge bases Example: Logistic regression Example: Shallow autoencoders Example: MLPs

Figure 1.4

SLIDE 6

(Goodfellow 2016)

Learning Multiple Components

Input Hand- designed program Output Input Hand- designed features Mapping from features Output Input Features Mapping from features Output Input Simple features Mapping from features Output Additional layers of more abstract features Rule-based systems Classic machine learning Representation learning Deep learning

Figure 1.5

SLIDE 7

(Goodfellow 2016)

Organization of the Book

1. Introduction

Part I: Applied Math and Machine Learning Basics

2. Linear Algebra
3. Probability and

Information Theory

4. Numerical

Computation

5. Machine Learning

Basics Part II: Deep Networks: Modern Practices

6. Deep Feedforward

Networks

7. Regularization
8. Optimization
9. CNNs
10. RNNs
11. Practical

Methodology

12. Applications

Part III: Deep Learning Research

13. Linear Factor

Models

14. Autoencoders
15. Representation

Learning

16. Structured

Probabilistic Models

17. Monte Carlo

Methods

18. Partition

Function

19. Inference
20. Deep Generative

Models

Figure 1.6

SLIDE 8

(Goodfellow 2016)

Historical Waves

1940 1950 1960 1970 1980 1990 2000 Year 0.000000 0.000050 0.000100 0.000150 0.000200 0.000250 Frequency of Word or Phrase

cybernetics (connectionism + neural networks)

Figure 1.7

SLIDE 9

(Goodfellow 2016)

Historical Trends: Growing Datasets

1900 1950 1985 2000 2015 100 101 102 103 104 105 106 107 108 109 Dataset size (number examples) Iris MNIST Public SVHN ImageNet CIFAR-10 ImageNet10k ILSVRC 2014 Sports-1M Rotated T vs. C T vs. G vs. F Criminals Canadian Hansard WMT

Figure 1.8

SLIDE 10

(Goodfellow 2016)

The MNIST Dataset

Figure 1.9

SLIDE 11

(Goodfellow 2016)

Connections per Neuron

1950 1985 2000 2015 101 102 103 104 Connections per neuron 1 2 3 4 5 6 7 8 9 10 Fruit fly Mouse Cat Human

Figure 1.10

SLIDE 12

(Goodfellow 2016)

Number of Neurons

1950 1985 2000 2015 2056 10−2 10−1 100 101 102 103 104 105 106 107 108 109 1010 1011 Number of neurons (logarithmic scale) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Sponge Roundworm Leech Ant Bee Frog Octopus Human

Figure 1.11

SLIDE 13

(Goodfellow 2016)

Solving Object Recognition

2010 2011 2012 2013 2014 2015 0.00 0.05 0.10 0.15 0.20 0.25 0.30 ILSVRC classification error rate

Figure 1.12