Topographic Organization of Receptive Fields in RecSOM or RecSOM - PowerPoint PPT Presentation

Topographic Organization of Receptive Fields in RecSOM or RecSOM as nonlinear IFS Peter Tiˇ no University of Birmingham, UK Igor Farkaˇ s Slovak University of Technology, Slovakia Jort van Mourik NCRG, Aston University, UK

Receptive fields in RecSOM Some motivations ◗ Most approaches to topographic map formation operate on the assumption that the data points are members of a finite- dimensional vector space of a fixed dimension. ◗ Recently, there has been an outburst of interest in extending topographic maps to more general data structures, such as se- quences or trees. ◗ Modified versions of SOM that have enjoyed a great deal of interest equip SOM with additional feed-back connections that allow for natural processing of recursive data types. ◗ No prior notion of metric on the structured data space is im- posed, instead, the similarity measure on structures evolves through parameter modification of the feedback mechanism and recursive comparison of constituent parts of the structured data. P. Tiˇ no, I. Farkaˇ s and J. van Mourik 1

Receptive fields in RecSOM Motivations cont’d ◗ Typical examples: Temporal Kohonen Map (Chappell, 1993), recurrent SOM (Koskela, 1998), feedback SOM (Horio, 2001), recursive SOM (Voegtlin, 2002), merge SOM (Strickert, 2003) and SOM for structured data (Hagenbuchner, 2003) ◗ At present, there is no general consensus as to how best to process sequences with SOMs. ◗ Representational capabilities of the models are hardly under- stood. ◗ The internal representation of structures within the models is unclear. ◗ First major theoretical study within a unifying framework in (Hammer, 2004). P. Tiˇ no, I. Farkaˇ s and J. van Mourik 2

Receptive fields in RecSOM Recursive Self-Organizing Map - RecSOM map at time t c i w i s(t) map at time (t−1) P. Tiˇ no, I. Farkaˇ s and J. van Mourik 3

Receptive fields in RecSOM RecSOM - weights Each neuron i ∈ { 1 , 2 , ..., N } in the map has two weight vectors associated with it: • w i ∈ R n – linked with an n -dim input s ( t ) feeding the network at time t • c i ∈ R N – linked with the context y ( t − 1) = ( y 1 ( t − 1) , y 2 ( t − 1) , ..., y N ( t − 1)) containing map activations y i ( t − 1) from the previous time step. P. Tiˇ no, I. Farkaˇ s and J. van Mourik 4

Receptive fields in RecSOM RecSOM - neuron activations The output of a unit i at time t is computed as (1) y i ( t ) = exp( − d i ( t )) , where d i ( t ) = α · � s ( t ) − w i � 2 + β · � y ( t − 1) − c i � 2 (2) α > 0 and β > 0 are model parameters that respectively influence the effect of the input and the context upon neuron’s profile. P. Tiˇ no, I. Farkaˇ s and J. van Mourik 5

Receptive fields in RecSOM RecSOM - learning The weight vectors are updated using the same form of learning rule ∆ w i = γ · h ik · ( s ( t ) − w i ) , (3) ∆ c i = γ · h ik · ( y ( t − 1) − c i ) , (4) k is an index of the best matching unit at time t , k = argmin d i ( t ) = argmax y i ( t ) , i ∈{ 1 , 2 ,...,N } i ∈{ 1 , 2 ,...,N } γ > 0 is the learning rate, h ik is a (Gaussian) neighborhood func- tion. P. Tiˇ no, I. Farkaˇ s and J. van Mourik 6

Receptive fields in RecSOM RecSOM - fixed-input dynamics Under a fixed input vector s ∈ R n , the time evolution becomes d i ( t + 1) = α · � s − w i � 2 + β · � y ( t ) − c i � 2 . (5) After applying a one-to-one coordinate transformation y i = e − d i , y i ( t + 1) = e − α � s − w i � 2 · e − β � y ( t ) − c i � 2 , (6) where � e − d 1 ( t ) , e − d 2 ( t ) , ..., e − d N ( t ) � y ( t ) = ( y 1 ( t ) , y 2 ( t ) , ..., y N ( t )) = . P. Tiˇ no, I. Farkaˇ s and J. van Mourik 7

Receptive fields in RecSOM RecSOM - fixed-input dynamics cont’d Gaussian kernel of inverse variance η > 0 , acting on R N : for any u , v ∈ R N , G η ( u , v ) = e − η � u − v � 2 . (7) The fixed-input dynamics written in a vector form: � � y ( t + 1) = Fs ( y ( t )) = F s , 1 ( y ( t )) , ..., F s ,N ( y ( t )) (8) , where F s ,i ( y ) = G α ( s , w i ) · G β ( y , c i ) , i = 1 , 2 , ..., N. (9) P. Tiˇ no, I. Farkaˇ s and J. van Mourik 8

Receptive fields in RecSOM RecSOM - Contractive IFS Study the conditions under which the map Fs becomes a con- traction. Then, by the Banach Fixed Point theorem, the autonomous Rec- SOM dynamics y ( t + 1) = Fs ( y ( t )) will be dominated by a unique attractive fixed point ys = Fs ( ys ) . A mapping F : R N → R N is said to be a contraction with contrac- tion coefficient ρ ∈ [0 , 1) , if for any y , y ′ ∈ R N , � F ( y ) − F ( y ′ ) � ≤ ρ · � y − y ′ � . (10) F is a contraction if there exists ρ ∈ [0 , 1) so that F is a contraction with contraction coefficient ρ . P. Tiˇ no, I. Farkaˇ s and J. van Mourik 9

Receptive fields in RecSOM Contractive IFS - Theorem Collection of activations coming from the feed-forward part of RecSOM: G α ( s ) = ( G α ( s , w 1 ) , G α ( s , w 2 ) , ..., G α ( s , w N )) . (11) Theorem: Consider an input s ∈ R n . If for some ρ ∈ [0 , 1) , β ≤ ρ 2 e 2 � G α ( s ) � − 2 , (12) then the mapping Fs is a contraction with contraction coefficient ρ . P. Tiˇ no, I. Farkaˇ s and J. van Mourik 10

Receptive fields in RecSOM Theorem - Proof The proof follows the worst case analysis of the distances � Fs ( y ) − Fs ( y ′ ) � under the constraint � y − y ′ � = δ : � Fs ( y ) − Fs ( y ′ ) � . D β ( δ ) = sup y , y ′ ; � y − y ′ � = δ The analysis is quite challenging, because D β ( δ ) can be expressed only implicitly. P. Tiˇ no, I. Farkaˇ s and J. van Mourik 11

Receptive fields in RecSOM Theorem - Proof cont’d It is possible to prove that, for a given β > 0 : 1. lim δ → 0 + D β ( δ ) = 0 , 2. D β is a continuous monotonically increasing concave function of δ . � 3. lim δ → 0 + dD β ( δ ) 2 β = e . dδ 1 0.9 0.8 beta=0.5 0.7 0.6 D(delta) beta=2 0.5 0.4 0.3 0.2 0.1 0 0 0.5 1 1.5 2 2.5 3 3.5 4 delta P. Tiˇ no, I. Farkaˇ s and J. van Mourik 12

Receptive fields in RecSOM Theorem - Proof cont’d Therefore, � 2 β D β ( δ ) ≤ δ (13) e . Using (6) we get that if N δ 2 2 β G 2 α ( s , w i ) ≤ ρ 2 δ 2 , � (14) e i =1 then Fs will be a contraction with contraction coefficient ρ . In- equality (14) is equivalent to 2 β � G α ( s ) � 2 ≤ ρ 2 . (15) e P. Tiˇ no, I. Farkaˇ s and J. van Mourik 13

Receptive fields in RecSOM Experiment Natural language data - ”Brave New World” by Aldous Huxley. Removed punctuation symbols, upper-case letters were switched to lower-case, the space between words was represented by ’-’. Length: 356606 symbols. Letters of the Roman alphabet were binary-encoded using 5 bits. RecSOM with 20 × 20 = 400 neurons was trained for two epochs using the following parameter settings: α = 3 , β = 0 . 7 , γ = 0 . 1 and σ : 10 → 0 . 5 . Radius of the neighborhood function reached its final value at the end of the first epoch and then remained constant to allow for fine-tuning of the weights. P. Tiˇ no, I. Farkaˇ s and J. van Mourik 14

Receptive fields in RecSOM Receptive Fields (RF) RF of a neuron is defined as the common suffix of all sequences for which that neuron becomes the best-matching unit. n− n− h− ad− d− he− he− a− ag . in ig . −th −th −th th ti an− u− − l− nd− e− re− −a− ao an ain in . l t−h th . . y− i− g− ng− ed− f− −to− o− en un −in al −al h wh ty ot− at− p− −a− n− on− m− o− −an n rn ul ll e−l e−h gh x y to t− es− as− er− er− mo o −to −on ion . ol e−m m . ey t− ut− s− is− or− ero t−o o lo ho on on oo . om um im am ai ry ts tw ts− r− r− ro wo io e−o −o e−n on −m t−m si ai ri e−s he−w −w t−w no so tio −o ng−o −o −n −l −h e−i di ei ni ui he−s e−w w nw ong no ak k −k −− −o . −l −h −i t−i −wi −hi −li −thi ns rs ing ng nf e−k j e−c −s −g −m −y −i −i i li hi s us uc e−g g if e−f e−b −c −s −w −w −e . −a −a n−a ia la ha is c nc f of −f −f −b −u −u −d d−a t−a na da . −ha as ac ic ib b . oc −v . −p g−t −t −d −e −q e−a a wa era ra ac ir e−r . os −r −p −t s−t . ow sa ore re ar ar hr r tr or op ov −v t−t d−t −t ot od . u se we ere pe es er her z p e−p p av d−t n−t e−t ot ou au −se be ue me es . her ter ap . mp v st rt −st tt ut out lu tu e e−e ce −he ew ev . q ea . . at t o−t ent ont ind d dd de te e he the− e− e− em ec . at −at ht −it nt −and rd e−d ne −the the he− e− eo . . ee ed ed ad it it id ond nd and ud ld le −the he P. Tiˇ no, I. Farkaˇ s and J. van Mourik 15