The Empirical Data: Do infants generalize identity rules? - - PowerPoint PPT Presentation

Raquel G. Alhama Willem Zuidema

What does a neural network need to learn truly general identity rules?

Pre-Wiring & Pre-Training:

de li de ji we ji...

[Marcus et al. 1999]

PARTICIPANTS: 7 month old infants

The Empirical Data: Do infants generalize identity rules?

The Empirical Data: Do infants generalize identity rules?

[Marcus et al. 1999]

PARTICIPANTS: 7 month old infants FAMILIARIZATION: ABA: “wi-je-wi le-di-le ji-li-ji … ” ABB: “wi-je-je le-di-di ji-li-li ...”

= ji li li = ji li li

The Empirical Data: Do infants generalize identity rules?

[Marcus et al. 1999]

PARTICIPANTS: 7 month old infants FAMILIARIZATION: ABA: “wi-je-wi le-di-le ji-li-ji … ” ABB: “wi-je-je le-di-di ji-li-li ...” TEST: “ba-po-ba ko-ga-ga ba-po-po … ” ABA ABB ABB

= ji li li = ji li li

[Marcus et al. 1999]

The Empirical Data: Do infants generalize identity rules?

The Empirical Data: Do infants generalize identity rules?

RESULT: Difgerential attention between grammars

= ba po ba = ba po po = wi je wi = wi je je

bapoba kogako bapopo kogaga

≠

= A B A = A B B

Modelling the Results

• Symbolic Cognition

“XYZ: X is the same as Z”

[Marcus et al. 1999]

= X Y Z

Modelling the Results

• Symbolic Cognition

“XYZ: X is the same as Z”

• Simple Recurrent Network (SRN)

– Trained to predict next syllable – Fails to predict novel (test) items

[Evaluation: % correct in predicting the third syllable] [Marcus et al. 1999]

= X Y Z

Modelling the Results

• Symbolic Cognition

“XYZ: X is the same as Z”

• Simple Recurrent Network (SRN)

– Trained to predict next syllable – Fails to predict novel (test) items

[Evaluation: % correct in predicting the third syllable] [Marcus et al. 1999]

A generalizing solution is in the hypothesis space

• f the SRN – why doesn't it fjnd it?

= X Y Z

Simulations with a Simple Recurrent Network

Proportion of statistically signifjcant responses to difgerent grammar conditions,

• ut of 400 runs of the model (with difgerent parameter settings)

Simulations with a Simple Recurrent Network

Proportion of statistically signifjcant responses to difgerent grammar conditions,

• ut of 400 runs of the model (with difgerent parameter settings)

What is missing in the SRN simulations?

• The SRN was simulated as a tabula rasa

– It starts learning from a random state

What is missing in the SRN simulations?

• The SRN was simulated as a tabula rasa

– It starts learning from a random state

• Pre-Wiring:

What would be a more cognitively plausible initial state?

• Pre-Training:

What is the role of prior experience?

Implementation: the Echo State Network

ESN, Jaeger (2001)

• Same hypothesis space as SRN
• Reservoir Computing approach:
• nly the weights in the output layer

are trained

(generally with Ridge Regression, but we use Gradient Descent)

• The weights in the reservoir are

randomly initialized

(with spectral radious < 1)

– How can we pre-wire it for this

task?

Pre-Wiring: Delay Line Memory

• DELAY LINE MEMORY: mechanism to preserve the

input by propagating it in a path with a delay

• Implementation:

– “Feed-Forward” structure in the reservoir – Strict or approximated copy

t=0 t=1 t=2

Pre-Wiring: Delay Line Memory

Pre-Wiring: Delay Line Memory

Pre-Wiring: Delay Line Memory

Pre-Wiring: Delay Line Memory

Pre-Wiring: Delay Line Memory

Does the model learn the generalized solution?

Simulations with the Delay Line

Extended Original

Pre-Training

• There are many solutions that fjt the training data

(non-generalizing solutions)

• Where does the pressure to fjnd a general solution

come from?

– Hypothesis: prior experience with environmental data may

have created a domain-general bias for abstract solutions

→ PRE-TRAINING: Incremental Novelty Exposure

T R A I N I N G T E S T

A1 A2 A3 A4 B1 B2 B3 B4

Ai Bj Ai

C1 C2 C3 C4 D1 D2 D3 D4

Ci Dj Ci

Pre-Training: Incremental Novelty Exposure

T R A I N I N G T E S T

A1 A2 A3 A4 B1 B2 B3 B4

A1B1 A5B5 Ai Bj Ai

C1 C2 C3 C4 D1 D2 D3 D4

Ci Dj Ci

C1 C2 C3 C4 D1 D2 D3 D4

Ci Dj Ci

A2 A3 A4 A5 B2 B3 B4 B5

Ai Bj Ai

Pre-Training: Incremental Novelty Exposure

T R A I N I N G T E S T

A1 A2 A3 A4 B1 B2 B3 B4

A1B1 A5B5 ... ... Ai Bj Ai

C1 C2 C3 C4 D1 D2 D3 D4

Ci Dj Ci A2 B2 A6B6

C1 C2 C3 C4 D1 D2 D3 D4

Ci Dj Ci

A2 A3 A4 A5 B2 B3 B4 B5

Ai Bj Ai

Pre-Training: Incremental Novelty Exposure

T R A I N I N G T E S T

A1 A2 A3 A4 B1 B2 B3 B4

A1B1 A5B5 ... ... Ai Bj Ai

C1 C2 C3 C4 D1 D2 D3 D4

Ci Dj Ci A2 B2 A6B6

C1 C2 C3 C4 D1 D2 D3 D4

Ci Dj Ci

A2 A3 A4 A5 B2 B3 B4 B5

Ai Bj Ai

C1 C2 C3 C4 D1 D2 D3 D4

Ci Dj Ci

Ak-3Ak-2Ak-1Ak Bk-3Bk-2Bk-1Bk

Ai Bj Ai Ak-4Bk-4 AkBk

Pre-Training: Incremental Novelty Exposure

Simulations with Incremental Novelty Exposure

Huge increase in % of correct predictions!

Random Random

Conclusions

• Finally, simulations of a recurrent network successfully

solving the task of Marcus et al. 1999

• This simple learning problem might hold lessons for

more complex architectures solving more complex tasks

• Crucial for success are:

(i) Pre-Wiring with a structure that improves memory (cf. LSTM, Memory Networks) (ii) Pre-Training with training regimes that favour generalization (cf. Dropout)

Contact: rgalhama@uva.nl https://staff.fnwi.uva.nl/r.garridoalhama/

Efgect of the Delay Line

Without Delay Line With Delay Line