[PPT] - Neural Networks Part 2 Yingyu Liang yliang@cs.wisc.edu Computer PowerPoint Presentation

SLIDE 1

Neural Networks

Part 2

Yingyu Liang yliang@cs.wisc.edu Computer Sciences Department University of Wisconsin, Madison

[Based on slides from Jerry Zhu, Mohit Gupta]

SLIDE 2

slide 2

Limited power of one single neuron

Perceptron: 𝑏 = 𝑕(σ𝑒 𝑥𝑒𝑦𝑒)
Activation function 𝑕: linear, step, sigmoid

… 1 𝑥0 𝑥1 𝑥𝐸 𝑦𝐸 𝑦1 𝑏 𝑕(෍

𝑒

𝑥𝑒𝑦𝑒)

SLIDE 3

slide 3

Limited power of one single neuron

Perceptron: 𝑏 = 𝑕(σ𝑒 𝑥𝑒𝑦𝑒)
Activation function 𝑕: linear, step, sigmoid
Decision boundary linear even for nonlinear 𝑕
XOR problem

… 1 𝑥0 𝑥1 𝑥𝐸 𝑦𝐸 𝑦1 𝑏 𝑕(෍

𝑒

𝑥𝑒𝑦𝑒)

SLIDE 4

slide 4

Limited power of one single neuron

XOR problem
Wait! If one can represent AND, OR, NOT, one can

represent any logic circuit (including XOR), by connecting them

Question: how to?

SLIDE 5

slide 5

Multi-layer neural networks

Standard way to connect Perceptrons
Example: 1 hidden layer, 1 output layer

𝑦2 𝑦1

Layer 3 (output) Layer 2 (hidden) Layer 1 (input)

SLIDE 6

slide 6

Multi-layer neural networks

Standard way to connect Perceptrons
Example: 1 hidden layer, 1 output layer

𝑥11

(2)

𝑏1

2 = 𝑕

෍

𝑒

𝑦𝑒𝑥1𝑒

(2)

𝑦2 𝑦1

𝑥12

(2)

SLIDE 7

slide 7

Multi-layer neural networks

Standard way to connect Perceptrons
Example: 1 hidden layer, 1 output layer

𝑥11

(2)

𝑥21

(2)

𝑥12

(2)

𝑥22

(2)

𝑏1

2 = 𝑕

෍

𝑒

𝑦𝑒𝑥1𝑒

(2)

𝑏2

2 = 𝑕

෍

𝑒

𝑦𝑒𝑥2𝑒

(2)

𝑦2 𝑦1

SLIDE 8

slide 8

Multi-layer neural networks

Standard way to connect Perceptrons
Example: 1 hidden layer, 1 output layer

𝑥11

(2)

𝑥21

(2)

𝑥31

(2)

𝑥12

(2)

𝑥22

(2)

𝑥32

(2)

𝑏1

2 = 𝑕

෍

𝑒

𝑦𝑒𝑥1𝑒

(2)

𝑏2

2 = 𝑕

෍

𝑒

𝑦𝑒𝑥2𝑒

(2)

𝑏3

2 = 𝑕

෍

𝑒

𝑦𝑒𝑥3𝑒

(2)

𝑦2 𝑦1

SLIDE 9

slide 9

Multi-layer neural networks

Standard way to connect Perceptrons
Example: 1 hidden layer, 1 output layer

𝑥11

(2)

𝑥21

(2)

𝑥31

(2)

𝑥12

(2)

𝑥22

(2)

𝑥32

(2)

𝑏1

2 = 𝑕

෍

𝑒

𝑦𝑒𝑥1𝑒

(2)

𝑏2

2 = 𝑕

෍

𝑒

𝑦𝑒𝑥2𝑒

(2)

𝑏3

2 = 𝑕

෍

𝑒

𝑦𝑒𝑥3𝑒

(2)

𝑥1

(3)

𝑥2

(3)

𝑥3

(3)

𝑏 = 𝑕 ෍

𝑗

𝑏𝑗

2 𝑥𝑗 (3)

𝑦2 𝑦1

SLIDE 10

slide 10

Neural net for 𝐿-way classification

Use 𝐿 output units
Training: encode a label 𝑧 by an indicator vector

▪ class1=(1,0,0,…,0), class2=(0,1,0,…,0) etc.

Test: choose the class corresponding to the largest
utput unit

𝑥11

(3)

𝑥12

(3)

𝑥13

(3)

𝑏1 = 𝑕 ෍

𝑗

𝑏𝑗

2 𝑥1𝑗 (3)

𝑦2 𝑦1

…

SLIDE 11

slide 11

Neural net for 𝐿-way classification

Use 𝐿 output units
Training: encode a label 𝑧 by an indicator vector

▪ class1=(1,0,0,…,0), class2=(0,1,0,…,0) etc.

Test: choose the class corresponding to the largest
utput unit

𝑥11

(3)

𝑥12

(3)

𝑥13

(3)

𝑏1 = 𝑕 ෍

𝑗

𝑏𝑗

2 𝑥1𝑗 (3)

𝑦2 𝑦1

𝑏𝐿 = 𝑕 ෍

𝑗

𝑏𝑗

2 𝑥𝐿𝑗 (3)

𝑥𝐿1

(3)

𝑥𝐿2

(3)

𝑥𝐿3

(3)

…

SLIDE 12

slide 14

The (unlimited) power of neural network

In theory

▪ we don’t need too many layers: ▪ 1-hidden-layer net with enough hidden units can represent any continuous function of the inputs with arbitrary accuracy ▪ 2-hidden-layer net can even represent discontinuous functions

SLIDE 13

slide 15

Learning in neural network

Again we will minimize the error (𝐿 outputs):
𝑦: one training point in the training set 𝐸
𝑏𝑑: the 𝑑-th output for the training point 𝑦
𝑧𝑑: the 𝑑-th element of the label indicator vector for 𝑦

𝐹 = 1 2 ෍

𝑦∈𝐸

𝐹𝑦 , 𝐹𝑦 = 𝑧 − 𝑏 2 = ෍

𝑑=1 𝐿

𝑏𝑑 − 𝑧𝑑 2

𝑦2 𝑦1

…

𝑏1 𝑏𝐿 1 … = 𝑧

SLIDE 14

slide 16

Learning in neural network

Again we will minimize the error (𝐿 outputs):
𝑦: one training point in the training set 𝐸
𝑏𝑑: the 𝑑-th output for the training point 𝑦
𝑧𝑑: the 𝑑-th element of the label indicator vector for 𝑦
Our variables are all the weights 𝑥 on all the edges

▪ Apparent difficulty: we don’t know the ‘correct’

utput of hidden units

𝐹 = 1 2 ෍

𝑦∈𝐸

𝐹𝑦 , 𝐹𝑦 = 𝑧 − 𝑏 2 = ෍

𝑑=1 𝐿

𝑏𝑑 − 𝑧𝑑 2

SLIDE 15

slide 17

Learning in neural network

Again we will minimize the error (𝐿 outputs):
𝑦: one training point in the training set 𝐸
𝑏𝑑: the 𝑑-th output for the training point 𝑦
𝑧𝑑: the 𝑑-th element of the label indicator vector for 𝑦
Our variables are all the weights 𝑥 on all the edges

▪ Apparent difficulty: we don’t know the ‘correct’

utput of hidden units

▪ It turns out to be OK: we can still do gradient

descent. The trick you need is the chain rule

▪ The algorithm is known as back-propagation 𝐹 = 1 2 ෍

𝑦∈𝐸

𝐹𝑦 , 𝐹𝑦 = 𝑧 − 𝑏 2 = ෍

𝑑=1 𝐿

𝑏𝑑 − 𝑧𝑑 2

SLIDE 16

slide 18

Gradient (on one data point)

𝑦2 𝑦1 𝑥11

(4)

Layer (4) Layer (3) Layer (2) Layer (1)

𝐹𝑦

want to compute

𝜖𝐹𝑦 𝜖𝑥11

4

SLIDE 17

slide 19

Gradient (on one data point)

𝑦2 𝑦1 = 𝑧 − 𝑏 2 𝑏1 𝑏2 𝑥11

(4)

Layer (4) Layer (3) Layer (2) Layer (1)

𝐹𝑦 𝐹𝑦 𝑏1 𝑧 − 𝑏 2

SLIDE 18

slide 20

Gradient (on one data point)

𝑦2 𝑦1 = 𝑧 − 𝑏 2 𝑏1 𝑏2 𝑥11

(4)

Layer (4) Layer (3) Layer (2) Layer (1)

𝐹𝑦 𝐹𝑦 𝑏1 𝑧 − 𝑏 2 𝑨1

(4) = 𝑥11 (4)𝑏1 (3) + 𝑥12 (4)𝑏2 (3)

𝑕 𝑨1

(4)

𝑨1

(4)

SLIDE 19

slide 21

Gradient (on one data point)

𝑦2 𝑦1 = 𝑧 − 𝑏 2 𝑏1 𝑏2 𝑥11

(4)

Layer (4) Layer (3) Layer (2) Layer (1)

𝐹𝑦 𝐹𝑦 𝑏1 𝑧 − 𝑏 2 𝑨1

(4) = 𝑥11 (4)𝑏1 (3) + 𝑥12 (4)𝑏2 (3)

𝑕 𝑨1

(4)

𝑨1

(4)

𝑥12

4 𝑏2 (3)

𝑥11

4 𝑏1 (3)

𝑥12

(4)

SLIDE 20

slide 22

Gradient (on one data point)

𝑦2 𝑦1 = 𝑧 − 𝑏 2 𝑏1 𝑏2 𝑥11

(4)

Layer (4) Layer (3) Layer (2) Layer (1)

𝐹𝑦 𝐹𝑦 𝑏1 𝑧 − 𝑏 2 𝑨1

(4) = 𝑥11 (4)𝑏1 (3) + 𝑥12 (4)𝑏2 (3)

𝑕 𝑨1

(4)

𝑨1

(4)

𝑥12

4 𝑏2 (3)

𝑥11

4 𝑏1 (3)

𝜖𝐹𝒚 𝜖𝑥11

(4) = 𝜖𝐹𝒚

𝜖𝑏1 𝜖𝑏1 𝜖𝑨1

(4)

𝜖𝑨1

(4)

𝜖𝑥11

(4)

By Chain Rule:

𝜖𝐹𝒚 𝜖𝑏1 = 2(𝑏1 − 𝑧1) 𝜖𝑏1 𝜖𝑨1

(4) = 𝑕′ 𝑨1 (4)

SLIDE 21

slide 23

Gradient (on one data point)

𝑦2 𝑦1 = 𝑧 − 𝑏 2 𝑏1 𝑏2 𝑥11

(4)

Layer (4) Layer (3) Layer (2) Layer (1)

𝐹𝑦 𝐹𝑦 𝑏1 𝑧 − 𝑏 2 𝑨1

(4) = 𝑥11 (4)𝑏1 (3) + 𝑥12 (4)𝑏2 (3)

𝑕 𝑨1

(4)

𝑨1

(4)

𝑥12

4 𝑏2 (3)

𝑥11

4 𝑏1 (3)

𝜖𝐹𝒚 𝜖𝑥11

(4) = 2(𝑏1 − 𝑧1)𝑕′ 𝑨1 (4)

𝜖𝑨1

(4)

𝜖𝑥11

(4)

By Chain Rule:

𝜖𝐹𝒚 𝜖𝑏1 = 2(𝑏1 − 𝑧1) 𝜖𝑏1 𝜖𝑨1

(4) = 𝑕′ 𝑨1 (4)

SLIDE 22

slide 24

Gradient (on one data point)

𝑦2 𝑦1 = 𝑧 − 𝑏 2 𝑏1 𝑏2 𝑥11

(4)

Layer (4) Layer (3) Layer (2) Layer (1)

𝐹𝑦 𝐹𝑦 𝑏1 𝑧 − 𝑏 2 𝑨1

(4) = 𝑥11 (4)𝑏1 (3) + 𝑥12 (4)𝑏2 (3)

𝑕 𝑨1

(4)

𝑨1

(4)

𝑥12

4 𝑏2 (3)

𝑥11

4 𝑏1 (3)

𝜖𝐹𝒚 𝜖𝑥11

(4) = 2(𝑏1 − 𝑧1)𝑕′ 𝑨1 (4) 𝑏1 (3)

By Chain Rule:

𝜖𝐹𝒚 𝜖𝑏1 = 2(𝑏1 − 𝑧1) 𝜖𝑏1 𝜖𝑨1

(4) = 𝑕′ 𝑨1 (4)

SLIDE 23

slide 25

Gradient (on one data point)

𝑦2 𝑦1 = 𝑧 − 𝑏 2 𝑏1 𝑏2 𝑥11

(4)

Layer (4) Layer (3) Layer (2) Layer (1)

𝐹𝑦 𝐹𝑦 𝑏1 𝑧 − 𝑏 2 𝑨1

(4) = 𝑥11 (4)𝑏1 (3) + 𝑥12 (4)𝑏2 (3)

𝑕 𝑨1

(4)

𝑨1

(4)

𝑥12

4 𝑏2 (3)

𝑥11

4 𝑏1 (3)

𝜖𝐹𝒚 𝜖𝑥11

(4) = 2(𝑏1 − 𝑧1)𝑕 𝑨1 (4)

1 − 𝑕 𝑨1

(4)

𝑏1

(3)

By Chain Rule:

𝜖𝐹𝒚 𝜖𝑏1 = 2(𝑏1 − 𝑧1) 𝜖𝑏1 𝜖𝑨1

(4) = 𝑕′ 𝑨1 (4)

SLIDE 24

slide 26

Gradient (on one data point)

𝑦2 𝑦1 = 𝑧 − 𝑏 2 𝑏1 𝑏2 𝑥11

(4)

Layer (4) Layer (3) Layer (2) Layer (1)

𝐹𝑦 𝐹𝑦 𝑏1 𝑧 − 𝑏 2 𝑨1

(4) = 𝑥11 (4)𝑏1 (3) + 𝑥12 (4)𝑏2 (3)

𝑕 𝑨1

(4)

𝑨1

(4)

𝑥12

4 𝑏2 (3)

𝑥11

4 𝑏1 (3)

𝜖𝐹𝒚 𝜖𝑥11

(4) = 2 𝑏1 − 𝑧1 𝑏1 1 − 𝑏1 𝑏1 (3)

By Chain Rule:

𝜖𝐹𝒚 𝜖𝑏1 = 2(𝑏1 − 𝑧1) 𝜖𝑏1 𝜖𝑨1

(4) = 𝑕′ 𝑨1 (4)

SLIDE 25

slide 27

Gradient (on one data point)

𝑦2 𝑦1 = 𝑧 − 𝑏 2 𝑏1 𝑏2 𝑥11

(4)

Layer (4) Layer (3) Layer (2) Layer (1)

𝐹𝑦 𝐹𝑦 𝑏1 𝑧 − 𝑏 2 𝑨1

(4) = 𝑥11 (4)𝑏1 (3) + 𝑥12 (4)𝑏2 (3)

𝑕 𝑨1

(4)

𝑨1

(4)

𝑥12

4 𝑏2 (3)

𝑥11

4 𝑏1 (3)

𝜖𝐹𝒚 𝜖𝑥11

(4) = 2 𝑏1 − 𝑧1 𝑏1 1 − 𝑏1 𝑏1 (3)

By Chain Rule:

𝜖𝐹𝒚 𝜖𝑏1 = 2(𝑏1 − 𝑧1) 𝜖𝑏1 𝜖𝑨1

(4) = 𝑕′ 𝑨1 (4)

Can be computed by network activation

SLIDE 26

slide 28

Backpropagation

𝑦2 𝑦1 = 𝑧 − 𝑏 2 𝑏1 𝑏2 𝑥11

(4)

Layer (4) Layer (3) Layer (2) Layer (1)

𝐹𝑦 𝐹𝑦 𝑨1

(4) = 𝑥11 (4)𝑏1 (3) + 𝑥12 (4)𝑏2 (3)

𝑨1

(4)

𝑥12

4 𝑏2 (3)

𝑥11

4 𝑏1 (3)

𝜖𝐹𝒚 𝜖𝑥11

(4) = 2 𝑏1 − 𝑧1 𝑏1 1 − 𝑏1 𝑏1 (3)

By Chain Rule:

𝜖𝐹𝒚 𝜖𝑨1

(4) = 2(𝑏1 − 𝑧1)𝑕′ 𝑨1 (4)

SLIDE 27

slide 29

Backpropagation

𝑦2 𝑦1 = 𝑧 − 𝑏 2 𝑏1 𝑏2 𝑥11

(4)

Layer (4) Layer (3) Layer (2) Layer (1)

𝐹𝑦 𝐹𝑦 𝑨1

(4) = 𝑥11 (4)𝑏1 (3) + 𝑥12 (4)𝑏2 (3)

𝑨1

(4)

𝑥12

4 𝑏2 (3)

𝑥11

4 𝑏1 (3)

𝜖𝐹𝒚 𝜖𝑥11

(4) = 2 𝑏1 − 𝑧1 𝑏1 1 − 𝑏1 𝑏1 (3)

By Chain Rule:

𝜀1

(4) = 𝜖𝐹𝒚

𝜖𝑨1

(4) = 2(𝑏1 − 𝑧1)𝑕′ 𝑨1 (4)

SLIDE 28

slide 30

Backpropagation

𝑦2 𝑦1 = 𝑧 − 𝑏 2 𝑏1 𝑏2 𝑥11

(4)

Layer (4) Layer (3) Layer (2) Layer (1)

𝐹𝑦 𝐹𝑦 𝜀1

(4)

𝑨1

(4)

𝑥12

4 𝑏2 (3)

𝑥11

4 𝑏1 (3)

𝜖𝐹𝒚 𝜖𝑥11

(4) = 𝜀1 (4)𝑏1 (3)

By Chain Rule:

𝜀1

(4) = 𝜖𝐹𝒚

𝜖𝑨1

(4) = 2(𝑏1 − 𝑧1)𝑕′ 𝑨1 (4)

𝑏1

(3)

SLIDE 29

slide 31

Backpropagation

𝑦2 𝑦1 = 𝑧 − 𝑏 2 𝑏1 𝑏2 𝑥11

(4)

Layer (4) Layer (3) Layer (2) Layer (1)

𝐹𝑦 𝐹𝑦 𝜀1

(4)

𝑨1

(4)

𝑥12

4 𝑏2 (3)

𝑥11

4 𝑏1 (3)

𝜖𝐹𝒚 𝜖𝑥11

(4) = 𝜀1 (4)𝑏1 (3),

𝜖𝐹𝒚 𝜖𝑥12

(4) = 𝜀1 (4)𝑏2 (3)

By Chain Rule:

𝜀1

(4) = 𝜖𝐹𝒚

𝜖𝑨1

(4) = 2(𝑏1 − 𝑧1)𝑕′ 𝑨1 (4)

𝑥12

(4)

𝑏2

(3)

SLIDE 30

slide 32

Backpropagation

𝑦2 𝑦1 = 𝑧 − 𝑏 2 𝑏1 𝑏2 𝑥21

(4)

Layer (4) Layer (3) Layer (2) Layer (1)

𝐹𝑦 𝐹𝑦 𝜀2

(4)

𝑨1

(4)

𝑥12

4 𝑏2 (3)

𝑥11

4 𝑏1 (3)

𝜖𝐹𝒚 𝜖𝑥21

(4) = 𝜀2 (4)𝑏1 (3),

𝜖𝐹𝒚 𝜖𝑥22

(4) = 𝜀2 (4)𝑏2 (3)

By Chain Rule:

𝜀2

(4) = 𝜖𝐹𝒚

𝜖𝑨2

(4) = 2(𝑏2 − 𝑧2)𝑕′ 𝑨2 (4)

𝑥22

(4)

𝑏2

(3)

𝑏1

(3)

SLIDE 31

slide 33

𝜀2

(4)

𝜀1

(3)

𝜀2

(3)

𝜀1

(2)

𝜀2

(2)

𝜀1

(4)

Backpropagation

𝑦2 𝑦1 = 𝑧 − 𝑏 2 𝑏1 𝑏2

Layer (4) Layer (3) Layer (2) Layer (1)

𝐹𝑦

Thus, for any weight in the network: 𝜖𝐹𝑦 𝜖𝑥

𝑘𝑙 (𝑚) = 𝜀 𝑘 (𝑚)𝑏𝑙 (𝑚−1)

𝜀

𝑘 (𝑚)

: 𝜀 of 𝑘𝑢ℎ neuron in Layer 𝑚 𝑏𝑙

(𝑚−1) : Activation of 𝑙𝑢ℎ neuron in Layer 𝑚 − 1

𝑥

𝑘𝑙 (𝑚)

: Weight from 𝑙𝑢ℎ neuron in Layer 𝑚 − 1 to 𝑘𝑢ℎ neuron in Layer 𝑚

SLIDE 32

slide 34

𝜀2

(4)

𝜀1

(3)

𝜀2

(3)

𝜀1

(2)

𝜀2

(2)

𝜀1

(4)

Exercise

𝑦2 𝑦1 = 𝑧 − 𝑏 2 𝑏1 𝑏2

Layer (4) Layer (3) Layer (2) Layer (1)

𝐹𝑦

Show that for any bias in the network: 𝜖𝐹𝑦 𝜖𝑐

𝑘 (𝑚) = 𝜀 𝑘 (𝑚)

𝜀

𝑘 (𝑚)

: 𝜀 of 𝑘𝑢ℎ neuron in Layer 𝑚 𝑐

𝑘 (𝑚)

: bias for the 𝑘𝑢ℎ neuron in Layer 𝑚, i.e., 𝑨𝑘

(𝑚) = σ𝑙 𝑥 𝑘𝑙 (𝑚)𝑏𝑙 (𝑚−1) + 𝑐 𝑘 (𝑚)

SLIDE 33

slide 35

𝜀2

(4)

𝜀1

(3)

𝜀2

(3)

𝜀1

(2)

𝜀2

(2)

𝜀1

(4)

Backpropagation of 𝜀

𝑦2 𝑦1 = 𝑧 − 𝑏 2 𝑏1 𝑏2

Layer (4) Layer (3) Layer (2) Layer (1)

𝐹𝑦

Thus, for any neuron in the network: 𝜀

𝑘 (𝑚) = ෍ 𝑙

𝜀𝑙

𝑚+1 𝑥𝑙𝑘 𝑚+1

𝑕′ 𝑨

𝑘 𝑚

𝜀

𝑘 (𝑚)

: 𝜀 of 𝑘𝑢ℎ Neuron in Layer 𝑚 𝜀𝑙

(𝑚+1)

: 𝜀 of 𝑙𝑢ℎ Neuron in Layer 𝑚 + 1 𝑕′ 𝑨

𝑘 𝑚

: derivative of 𝑘𝑢ℎ Neuron in Layer 𝑚 w.r.t. its linear combination input 𝑥𝑙𝑘

(𝑚+1)

: Weight from 𝑘𝑢ℎ Neuron in Layer 𝑚 to 𝑙𝑢ℎ Neuron in Layer 𝑚 + 1

SLIDE 34

slide 36

Gradient descent with Backpropagation

1. Initialize Network with Random Weights and Biases
2. For each Training Image:

a. Compute Activations for the Entire Network b. Compute 𝜀 for Neurons in the Output Layer using Network Activation and Desired Activation 𝜀

𝑘 (𝑀) = 2 𝑧𝑘 − 𝑏𝑘 𝑏𝑘(1 − 𝑏𝑘)

c. Compute 𝜀 for all Neurons in the previous Layers 𝜀

𝑘 (𝑚) = ෍ 𝑙

𝜀𝑙

𝑚+1 𝑥𝑙𝑘 𝑚+1 𝑏𝑘 𝑚 (1 − 𝑏𝑘 𝑚 )

d. Compute Gradient of Cost w.r.t each Weight and Bias for the Training Image using 𝜀 𝜖𝐹𝑦 𝜖𝑥

𝑘𝑙 (𝑚) = 𝜀 𝑘 (𝑚)𝑏𝑙 (𝑚−1)

𝜖𝐹𝑦 𝜖𝑐

𝑘 (𝑚) = 𝜀 𝑘 (𝑚)

SLIDE 35

slide 37

Gradient descent with Backpropagation

3. Average the Gradient w.r.t. each Weight and Bias over the

Entire Training Set 𝜖𝐹 𝜖𝑥

𝑘𝑙 (𝑚) = 1

𝑜 ෍ 𝜖𝐹𝒚 𝜖𝑥

𝑘𝑙 (𝑚)

𝜖𝐹 𝜖𝑐

𝑘 (𝑚) = 1

𝑜 ෍ 𝜖𝐹𝒚 𝜖𝑐

𝑘 (𝑚)

4. Update the Weights and Biases using Gradient Descent

𝑥

𝑘𝑙 (𝑚)⃪ 𝑥 𝑘𝑙 (𝑚) − 𝜃 𝜖𝐹

𝜖𝑥

𝑘𝑙 𝑚

𝑐

𝑘 𝑚 ⃪ 𝑐 𝑘 𝑚 − 𝜃 𝜖𝐹

𝜖𝑐

𝑘 (𝑚)

5. Repeat Steps 2-4 till Cost reduces below an acceptable level