Perceptrons 2-29-16 What is a neural network? activation - PowerPoint PPT Presentation

Perceptrons 2-29-16

What is a neural network? activation connection functions ● A NN is a directed acyclic graph. weights ● Nodes are organized into layers. -0.5 ● Consecutive layers are fully connected. 1.5 0.2 ● Edges have a weight. 2.7 ● Nodes have activation functions. -0.3 0.8 -1.6 -1.2 Other topologies are possible: 0.4 3.0 ● sparser inter-layer connectivity ● edges within layers -1.0 0.1 hidden layer(s) ● edges jumping layers input layer output layer

What does a neural network compute? Each node computes the weighted sum of its inputs. 1.2 -0.5 -.5 * 1.2 + .8 * -.8 + 3 * .4 = -.04 0.8 This sum is then passed through the node’s -0.8 activation function. 3.0 f(x) = 1 / (1 + e -x ) = 1 / (1 + e .04 ) ≈ .49 0.4 This output is passed on to the next layer.

Exercise: finish feeding values through the network. Sigmoid activation function: 1.2 -0.5 1.5 0.2 2.7 -0.3 0.8 -0.8 Threshold activation function: -1.6 -1.2 0.4 3.0 0.4 -1.0 0.1

What type of learning problem is this? Supervised or unsupervised? ● We’ll be studying neural nets for supervised learning. ● They can also be used for unsupervised learning. Classification or regression? ● Depends on the output units: ○ Discrete-valued output units for classification. ○ Continuous-valued output units for regression.

What is the hypothesis space? Unspecified parameters: ● Network topology ○ Number of hidden layers ○ Size of each hidden layer typically hand-picked ○ Connectivity of hidden layers ● Activation functions ● Edge weights typically learned

What is a perceptron? A perceptron is a 2-layer neural network. 1 ● Only input & output; no hidden units. -0.5 0.2 All activation functions are thresholds. 0.8 ● Threshold at 0. -1.2 One input is a constant 1. 3.0 0.1

Perceptrons represent a decision surface 1 -0.5 0.8 x 1 3.0 x 2

Linear separability Perceptrons can only classify linearly separable data.

Our task: learn perceptron weights from data. Key idea: loop through the training data and update weights when wrong. while any training example is misclassified: for each training example: output = run example through the network for each node i in the output: if output[i] != target[i]: for each input weight w_j to node i: w_j = w_j + Delta(w_j) Delta(w_j) = learning_rate * (target[j] - output[j]) * example[j]

Example: learn AND AND( 0 , 0 ) = 0 AND( 0 , 1 ) = 0 AND( 1 , 0 ) = 0 AND( 1 , 1 ) = 1

Exercise: learn OR OR( 0 , 0 ) = 0 OR( 0 , 1 ) = 1 OR( 1 , 0 ) = 1 OR( 1 , 1 ) = 1

We can compose perceptrons like logic gates. ● We can represent AND, OR, and NOT with perceptrons. ● By composing these, we can make multi-layer networks. ● AND, OR, and NOT constitute a universal gate set, so we can make any boolean function by combining perceptrons. In fact, we can represent any boolean function with a 2-layer perceptron.