Finding Structure in Time Jeffrey L. Elman In Cognitive Science 14, - - PowerPoint PPT Presentation

Finding Structure in Time

Jeffrey L. Elman In Cognitive Science 14, 179 – 211 (1990) presented by Dominic Seyler (dseyler2@illinois.edu)

Outline

• Motivation
• Method
• Experiments
• Exclusive-Or
• Structure in Letter Sequences
• Discovering the Notion “Word”
• Discovering Lexical Classes
• Conclusions

Motivation: The Problem with Time

• Previous methods of representing time
• Associate serial order of temporal pattern with dimensionality of pattern

vector

• [ 0 1 0 0 1 ] <- first, second, third... event in temporal order
• There are several downsides of presenting time this way
• Input buffer is required to represent events all at once
• All input vectors must be the same length and provide for the longest possible

temporal pattern

• Most importantly: Cannot distinguish relative from absolute temporal

position

[ 0 1 1 1 0 0 0 0 0 ] [ 0 0 0 1 1 1 0 0 0 ]

An Alternative Way of Treating Time

• Don’t model time as an explicit

part of the input

• Allow time to be represented by

the effect it has on processing

• Networks allows hidden units to

see previous output

• Recurrent connections are what

give the network memory

Approach: Recurrent Neural Network

• Argument input with additional

units (context units)

• When input is processed

sequentially, the context units contain the exact values of the hidden units of the previous sequence

• The hidden units map the

external input and previous internal state to desired output

Exclusive-OR

• XOR function cannot be learned by a simple two-layer network
• Temporal XOR: One input bit is presented at a time, predict next bit
• Input: 1 0 1 0 0 0
• output: 0 1 0 0 0 ?
• Training: Run 600 passes through a 3,000 bit XOR sequence

Exclusive-OR (cont.)

• It is only sometimes possible

to predict the next bit correctly

• After one bit, there is a 50/50

chance

• After two bits, the third bit will

be the XOR of the first and second

Structure in Letter Sequences

• Idea: Extend prediction from one bit vectors to more complex

predictions (multi-bit)

• Method:
• map six letters to a binary representation (b, d, g, a, i, u)
• Use three consonants to create a random 1,000 letter sequence
• Replace each consonant by adding vowels: b -> ba; d -> dii; g -> guuu
• Example input: dbgbddg -> diibaguuubadiidiiguuu
• Prediction task: given the bit representations of characters in

sequence, predict the character word

Structure in Letter Sequences (cont.)

• Since consonants where ordered

randomly there is high error

• Vowels are not random, therefore

the network can make use of previous information. Thus, error is low.

• Takeaway: Since the input is

structured the network can make partial predictions even where the complete prediction is not possible

Discovering the Notion “Word”

• Learning a language involves learning words
• Can the network automatically learn “words”, when given a

sequential list of concatenated characters?

• Words are represented as concatenated bit vectors of their characters
• These bit vectors are concatenated to form sentences
• Then, each character is inputted sequentially and the network has to

predict the following letter

• input:

manyyearsago

• output:

anyyearsago?

Discovering the Notion “Word” (cont.)

• At the onset of each word error is

high

• As more of the word is received,

error declines

• Error provides good clue as to what

the recurring sequences in the input are and highly correlates with words

• Network can learn boundaries of

linguistic units from input signal

Discovering Lexical Classes from Word Order

• Can network learn the abstract structure that underlies sentences,

when only the surface forms (i.e. words) are presented to it?

• Method
• Define a set of category-to-word mappings (e.g., NOUN-HUMAN -> man,

woman; VERB-PERCEPTION -> smell, see)

• Use templates to create sentences (e.g., NOUN-HUMAN, VERB-EAT, NOUN-

FOOD)

• Words in sentence (e.g., ”woman eat bread”) are mapped to one-hot-

vectors (e.g. 00010 00100 10000)

• Task: Given a word vector (“woman”) predict next word (“eat”).

Discovering Lexical Classes (cont.)

• Since prediction task is

nondeterministic RMS error is not a fitting measurement

• Save the hidden unit vectors for

each word in all possible contexts and average over them

• Perform hierarchical clustering
• Similarity structure of internal

representations is shown in tree

Discovering Lexical Classes (cont.)

• Network has developed internal representations for the input vectors

which reflect facts about possible sequential ordering of inputs

• Hidden unit patterns are not word representations in the

conventional sense, since patterns also reflect prior context.

• Error in predicting the actual next word in a given context is high, but

the network is able to predict the approximate likelihood of

• ccurrence of classes and words
• A given node in hidden layer participates in multiple concepts. Only

the activation pattern in its entirety is meaningful.

Conclusions

• Networks can learn temporal structure implicitly
• Problems change their nature when expressed as temporal events

(XOR could previously not be learned by single-layer network)

• Error signal is a good metric of where structure exists (Error was high

at the beginning of words in sentence)

• Increasing complexity does not necessarily result in worse

performance (Increasing number of bits did not hurt performance)

• Internal representations can be hierarchical in nature (Similarity was

high among words within one class)

Finding Structure in Time

Jeffrey L. Elman In Cognitive Science 14, 179 – 211 (1990) presented by Dominic Seyler (dseyler2@illinois.edu)