Associative memories 9/25/2014 Memorized associations are - - PowerPoint PPT Presentation

Associative memories

• 9/25/2014

Memorized associations are ubiquitous

“Bill”

Stimulus Response

Memorized associations are ubiquitous

“Bill”

Stimulus Response Key properties:

• Noise tolerance (generalization)
• Graceful saturation
• High capacity

First attempts: Holography

van Heerden, 1963 Willshaw, Longuet-higgins, 1960s

r,s(x) = r ? s = Z r(⇣)r(x − ⇣)d⇣ s φr,s(x) = Z s(τ)φr,s + (τ + x)dτ

Mathematically, this is the convolution-correlation scheme from class 4.

Matrix memories

Steinbuch, 1962 Willshaw et al., 1969

Before long, it was realized that better results could be obtained with a simpler, more neurally plausible framework. Let’s explore a simple Hebbian scheme. We have input and output lines, and we strength synapses when they’re

• n together.

Input / Stimulus Output / Response

Storage

Storage

Storage

Storage

Storage

What happens here depends on the specific choice of learning rule.

Storage

Additive Hebb rule M =

n

X

i=1

RiST

i

S1 S2 S3

R3 R2 R1

What happens here depends on the specific choice of learning rule.

Storage

Additive Hebb rule

S1 S2 S3

R3 R2 R1

R M =

n

X

i=1

RiST

i

M =

n

X

i=1

RST M ST

What happens here depends on the specific choice of learning rule.

Storage

Additive Hebb rule

S2

M =

n

X

i=1

RiST

i

M =

n

X

i=1

RST M ST

Storage

Additive Hebb rule

S2

M =

n

X

i=1

RiST

i

M =

n

X

i=1

RST M ST

ˆ R2

ˆ R

Retrieval

Additive Hebb rule

S1

M =

n

X

i=1

RiST

i

M =

n

X

i=1

RST M ST

Retrieval

Additive Hebb rule

S1

M =

n

X

i=1

RiST

i

M ST ˆ R

ˆ R1

ˆ Rj =

n

X

i=1

RiST

i Sj

Retrieval

Additive Hebb rule

S1

M =

n

X

i=1

RiST

i

M ST ˆ R

ˆ R1

ˆ Rj =

n

X

i=1

RiST

i Sj

ˆ Rj =

n

X

i6=j

RiST

i Sj + Rj||Sj||2

Retrieval

Additive Hebb rule

S1

M =

n

X

i=1

RiST

i

M ST ˆ R

ˆ R1

ˆ Rj =

n

X

i=1

RiST

i Sj

ˆ Rj =

n

X

i6=j

RiST

i Sj + Rj||Sj||2

If the Sk are orthonormal, then retrieval is exact. 1

Another perspective

Recall the the optimal memory matrix is M? = RS† If the columns of S are linearly independent, then X† = (XT X)−1XT , giving M? = R(ST S)−1ST . So if the columns of S (the Si) are orthonormal, M? = RST , which is exactly what we got for the simple Hebb rule.

Capacity

How much information can a matrix memory store? Model:

• 1. |S| = |R| = N × P
• 2. Each input pattern (column of S) has mS nonzeros,

and each output pattern (column of R) has mR nonzeros.

• 3. Binary Hebb rule: M = max(RST , 1)

(Each entry is clipped at one.)

• 4. Threshold recall:

ˆ Rjk = ⇢ 1 [MSj]k > τ else P patterns of size N

Capacity

How much information can a matrix memory store? Model:

• 1. |S| = |R| = N × P
• 2. Each input pattern (column of S) has mS nonzeros,

and each output pattern (column of R) has mR nonzeros.

• 3. Binary Hebb rule: M = max(RST , 1)

(Each entry is clipped at one.)

• 4. Threshold recall:

ˆ Rjk = ⇢ 1 [MSj]k > τ else P patterns of size N

• Parameters we can pick

Capacity

M ST

ˆ R2

ˆ R

To choose the threshold τ, note that in the absence of noise, a column

• f M has exactly mS ones.

So in order to recover all the ones in R, we better set τ = mS.

Capacity

Sparsity parameters The chance of given weight in M remaining zero throughout the learning process is (1 − mSmR N 2 )P ≈ e

−P mS mR N2

= (1 − q) NqmS = 1 ⇒ mS = − logN

logq

The probability of a spurious one in ˆ R is the probability of exceeding τ purely by chance. This is NqmS. So the highest value for mS we can choose before make the first error is:

Capacity

Sparsity parameters e

−P mS mR N2

= (1 − q) mS = − logN

logq

From previous slide

Capacity

Sparsity parameters This is quite good! (1 − q) = e

−P mS mR N2

⇒ mSmR N 2 = log(2) N 2

Memory space

The decision to represent memory items as sparse, high- dimensional vectors has some interesting consequences. High-dimensional spaces are counterintuitive.

In 1000 dimensions, 0.001 of all patterns are within 451 bits of a given point, and all but 0.001 are within 549 bits.

• Points tend to be orthogonal - most

point-pairs are “noise-like.”

Kanerva, 1988

Memory space

The decision to represent memory items as sparse, high- dimensional vectors has some interesting consequences. High-dimensional spaces are counterintuitive.

Almost all pairs of points are far apart, but there are multiple“linking” points that are close to both.

• Kanerva, 1988

Linking concepts

Memory space

The decision to represent memory items as sparse, high- dimensional vectors has some interesting consequences. High-dimensional spaces are counterintuitive.

Almost all pairs of points are far apart, but there are multiple“linking” points that are close to both.

• Kanerva, 1988

Linking concepts

Matrix memories in the brain: Marr’s model of the cerebellum

Short, live experiment Marr, 1967 Albus, 1971

Matrix memories in the brain: Marr’s model of the cerebellum

The cerebellum produces smooth, coordinated motor movements. (And may be involved in cognition as well). 24-year old Chinese woman without a cerebellum

Mossy Fiber Contextual Input Purkinje Axons Motor Output

Learn associations straight from context to actions, so you don’t have to “think” before doing.

Mossy Fiber Contextual Input Purkinje Axons Motor Output

Learn associations straight from context to actions, so you don’t have to “think” before doing.

Mossy Fiber Contextual Input Purkinje Axons Motor Output

Learn associations straight from context to actions, so you don’t have to “think” before doing.

Mossy Fiber Contextual Input Purkinje Axons Motor Output

But how does training work? How do the right patterns “appear” on the output (Purkinje) lines?

Climbing Fibers Motor Teaching Input

The rest of the brain

Mossy Fiber Contextual Input Purkinje Axons Motor Output Climbing Fibers Motor Teaching Input

The rest of the brain There’s a remarkable 1-1 correspondence between climbing fibers and Purkinje

• axons. Moreover, each climbing fiber wraps

around and around it’s Purkinje axon, making hundreds of synapses; a single AP can make a Purkinje spike.

Mossy Fiber Contextual Input Purkinje Axons Motor Output

We said that sparsity was a key property. How is that manifested here?

Climbing Fibers Motor Teaching Input

The rest of the brain

Mossy Fiber Contextual Input Purkinje Axons Motor Output Climbing Fibers Motor Teaching Input Granule Cells Sparsification

Mossy Fiber Contextual Input Purkinje Axons Motor Output Climbing Fibers Motor Teaching Input Granule Cells Sparsification

Mossy Fiber Contextual Input Purkinje Axons Motor Output Climbing Fibers Motor Teaching Input Granule Cells Sparsification

Mossy Fiber Contextual Input Purkinje Axons Motor Output Climbing Fibers Motor Teaching Input Granule Cells Sparsification

There are 50 billion granule cells - 3/4 of the brain’s neurons.They’re tiny.

• The idea here is that they “blow up” the

mossy fiber input into a larger space in which the signal can be sparser.

• Granule cells code for sets of mossy fibers

(codons), hypothesized to be primitive input features.

Storing structured information

We’ve discussed how to store S-R pairs, but human cognition goes way beyond this.

Relations

• The kettle is on the table.
• The kettle is to the right of the mug.

Storing structured information

As before, “concepts” are activation vectors. Kettle Jar

Storing structured information

As before, “concepts” are activation vectors. Kettle Jar

How to represent this? Green(Jar) & Gray(Kettle)

Storing structured information

As before, “concepts” are activation vectors. Kettle Jar

How to represent this? Green(Jar) & Gray(Kettle)

Green Gray

Storing structured information

As before, “concepts” are activation vectors. Kettle Jar

How to represent this? Green(Jar) & Gray(Kettle)

Green Gray Now what?

Maybe we should just have all of these patterns fire at once?

• But then how do we know we don’t

have Gray(Jar) & Green(Kettle) ? Or, worse, Jar(Kettle) & Green & Gray?

Storing structured information

As before, “concepts” are activation vectors. Kettle Jar

How to represent this? Green(Jar) & Gray(Kettle)

Green Gray Now what?

Maybe we should just have all of these patterns fire at once?

• But then how do we know we don’t

have Gray(Jar) & Green(Kettle) ? Or, worse, Jar(Kettle) & Green & Gray?

We need a way to bind predicates to arguments.

Storing structured information

As before, “concepts” are activation vectors. Kettle Jar

How to represent this? Green(Jar) & Gray(Kettle)

Green Gray

We need a way to bind predicates to arguments.

⊗ ⊕ ⊗

Binding operator Conjunction operator

Storing structured information

Jar Green How should we choose and ?

⊗ ⊕

⊗

=

Jar Green

Paul Smolensky, “Tensor Product Variable Binding”, 1990

Storing structured information

How should we choose and ?

⊗ ⊕

=

Jar

Paul Smolensky, “Tensor Product Variable Binding”, 1990

Kettle Jar Green Gray

⊗ ⊕ ⊗

Kettle Green Gray

Storing structured information

How should we choose and ?

⊗ ⊕

=

Jar

Paul Smolensky, “Tensor Product Variable Binding”, 1990

Kettle Jar Green Gray

⊗ ⊕ ⊗

Kettle Green Gray

This is just a matrix memory

Storing structured information

We can store some interesting “data structures,” like a stack:

Linearly independent “indexing roles” Some alphabet “fillers”

Or a tree: A B C S = X

i

fi ⊗ ri T = A ⊗ r0 ⊗ +[B ⊗ r0 + C ⊗ r1] ⊗ r1 = A ⊗ r0 ⊗ +B ⊗ r0 ⊗ r1 + C ⊗ r1 ⊗ r1 = A ⊗ r0 ⊗ +B ⊗ r10 + C ⊗ r11

Note the dimensions wrt these sums

Storing structured information

A B C T = A ⊗ r0 ⊗ +[B ⊗ r0 + C ⊗ r1] ⊗ r1 = A ⊗ r0 ⊗ +B ⊗ r0 ⊗ r1 + C ⊗ r1 ⊗ r1 = A ⊗ r0 ⊗ +B ⊗ r10 + C ⊗ r11 Even better, we can do symbolic operations with matrices: A = Wex0T B = Wex0Wex1T Wex01 T = [A, T 0] = Wcons0A + Wcons1T 0 Extraction (car, cdr) Construction (cons) One matrix

Storing structured information

A B C T = A ⊗ r0 ⊗ +[B ⊗ r0 + C ⊗ r1] ⊗ r1 = A ⊗ r0 ⊗ +B ⊗ r0 ⊗ r1 + C ⊗ r1 ⊗ r1 = A ⊗ r0 ⊗ +B ⊗ r10 + C ⊗ r11 Even better, we can do symbolic operations with matrices: A = Wex0T B = Wex0Wex1T Wex01 T = [A, T 0] = Wcons0A + Wcons1T 0 Extraction (car, cdr) Construction (cons) One matrix

• So you can build a whole LISP

in linear algebra!

Storing structured information

One more example: Go from passive-voice sentences to predicate representation

P Aux by V A

[Few movies] are admired by [skeptical critics]

A P V

F F(T) = WT W = Wcons0[Wex1Wex0Wex1] +Wcons1[Wcons0(Wex1Wex1Wex1) + Wcons1Wex0] This is just one matrix multiplication

Admire( [Few movies], [skeptical critics] )

Encoding grammars

Grammaticality can be represented as constraints on which role/filler pairs.

• A singular subject requires a correctly conjugated verb, e.g.
• Such constraints can be encoded as weights in the “binding matrix.” (Note

the departure from matrix memories, in which the entries in the matrix were synapses.) These weights give us an energy function, and evolution dynamics.

• Over time, the network will settle into

low energy (=high grammaticality) states.

Other frameworks: Vector-symbolic architectures Idea: represent role/filler bindings as vectors, rather than tensors; everything is the same size.

Circular covolution (Plate)

Other frameworks Recursive Auto-Associative Memory (RAAM) Learn the compression function. A C B D W WT W[W[W[B, C], D], A] Just like in an ordinary autoencoder, we learn the same weights for encoding and decoding. Pollack (1990)

Modern applications: Socher et al. 2013 Similar to RAAMs, that the model learns a parametrized compression function; details are different.

Bag of words Bigrams

End