SLIDE 1
2
Introduction to Machine Learning Multilayer Perceptron Barnabs - - PowerPoint PPT Presentation
Introduction to Machine Learning Multilayer Perceptron Barnabs - - PowerPoint PPT Presentation
Introduction to Machine Learning Multilayer Perceptron Barnabs Pczos The Multilayer Perceptron 2 Multilayer Perceptron 3 ALVINN: AN AUTONOMOUS LAND VEHICLE IN A NEURAL NETWORK Dean A. Pomerleau, Carnegie Mellon University, 1989
SLIDE 2
SLIDE 3
3
Multilayer Perceptron
SLIDE 4
4
Dean A. Pomerleau, Carnegie Mellon University, 1989
ALVINN: AN AUTONOMOUS LAND VEHICLE IN A NEURAL NETWORK
Training: using simulated road generator
SLIDE 5
5
Gradient Descent
We want to solve:
SLIDE 6
6
Starting Point
SLIDE 7
7
Starting Point
SLIDE 8
8
Fixed step size can be too big
SLIDE 9
9
Fixed step size can be too small
SLIDE 10
10
SLIDE 11
11
SLIDE 12
12
Character Recognition with MLP
Matlab: appcr1
SLIDE 13
13
The network
Noise-free input: 26 different letters of size 7x5
SLIDE 14
14
Noisy inputs
SLIDE 15
15
% Create MLP hiddenlayers=[10, 25]; net1 = feedforwardnet(hiddenlayers); net1 = configure(net1,X,T); %View view(net1); %Train net1 = train(net1,X,T); %Test Y1 = net1(Xtest);
Matlab MLP Training
SLIDE 16
16
Prediction errors
▪ Network 1 was trained on clean images ▪ Network 2 was trained on noisy images. 30 noisy copies of each letter are created
SLIDE 17
17
The Backpropagation Algorithm
SLIDE 18
18
Multilayer Perceptron
SLIDE 19
19
The gradient of the error
SLIDE 20
20
Notation
SLIDE 21
21
Some observations
SLIDE 22
22
The backpropagated error
SLIDE 23
23
The backpropagated error
Lemma
SLIDE 24
24
The backpropagated error
Therefore,
SLIDE 25
25
The backpropagation algorithm
SLIDE 26
26
SLIDE 27
27
SLIDE 28
28
SLIDE 29
29
SLIDE 30
30
SLIDE 31
31
SLIDE 32
32
SLIDE 33
33
What functions can multilayer perceptrons represent?
SLIDE 34
34
Perceptrons cannot represent the XOR function
f(0,0)=1, f(1,1)=1, f(0,1)=0, f(1,0)=0
What functions can multilayer perceptrons represent?
SLIDE 35
35
“Solve 7-th degree equation using continuous functions of two parameters.”
Related conjecture: Let f be a function of 3 arguments such that Prove that f cannot be rewritten as a composition of finitely many functions of two arguments. Another rewritten form: Prove that there is a nonlinear continuous system of three variables that cannot be decomposed with finitely many functions of two variables.
Hilbert’s 13th Problem
1902: 23 “most important” problems in mathematics The 13th Problem: Conjecture: It can’t be solved…
SLIDE 36
36
f(x,y,z)=Φ1(ψ1(x), ψ2(y))+Φ2(c1ψ3(y)+c2ψ4(z),x)
x z y
Σ Φ1 Φ2 ψ1 ψ2 ψ3 ψ4 Σ
f(x,y,z)
c1 c2
Function decompositions
SLIDE 37
37
1957, Arnold disproves Hilbert’s conjecture.
Function decompositions
SLIDE 38
38
Issues: This statement is not constructive.
Function decompositions
Corollary:
SLIDE 39
39
Kur Hornik, Maxwell Stinchcombe and Halber White: “Multilayer feedforward networks are universal approximators”, Neural Networks, Vol:2(3), 359-366, 1989
Universal Approximators
Definition: ΣN(g) neural network with 1 hidden layer: Theorem: Definition:
SLIDE 40
40
Theorem: (Blum & Li, 1991)
Universal Approximators
Definition: Formal statement:
SLIDE 41
41
Integral approximation in 1-dim:
i j i i
Proof
xi xi xi
Integral approximation in 2-dim: GOAL:
SLIDE 42
42
xi xi xi
The indicator function of Xi polygon can be learned by this neural network: 1 if x is in Xi
- 1 otherwise
Proof
The weighted linear combination of these indicator functions will be a good approximation of the original function f GOAL:
SLIDE 43
43
Proof
This linear equation can also be solved.
SLIDE 44
44