Functional Equations & Neural Networks for Time Series - - PowerPoint PPT Presentation

Functional Equations & Neural Networks for Time Series Interpolation

Lars Kindermann, AWI Achim Lewandowski, OEFAI

Drop an object with different speeds and measure the speed at the ground

Question: What’s the speed vm after half the way at xm? v0 v1 x0 x1 v1 f v0   = vm ? = xm v0 v0 v1 free fall friction

Data

An old experiment

Free Fall: Theory: Model: with data fitted With additional Friction: Theory: Model: Integrate numerically and fit and - already a non-trivial Problem! t2

2

  x g =  v1 f v0   v0

2

2g x  + = = g t2

2

  x g k1 t  x – k2 t  x 2 – f t  x     – =  g k

Solving with traditional Physics

x0 x1 v1 v0 x0 x1 vm  vm   = xm v0 v1 f v0   = v1  vm     v0   = =



divide into two equal steps...

  v     f v   =

and solve this functional equation for 

Theory: Assume translation invariance

A Data-based Aproach

A solution of this equation is a kind of square root of the function .

• If

: is a function, we look for another function which composed with itself equals : Because the self-composition of a function is also called “iteration”, the square root of a function is usually called its iterative root. is solved by the fractional iterates of a function :   x     f x   =  f f x   I R

n

I R

n

  x   f   x     f x   = f f x     f2 x   = n x   fm x   = f  x   fm n



x   =

A Functional Equation

A solution of this equation is called a square root of .

• If

: is a function, we look for another function which composed with itself equals : Because the self-composition of a function is also called “iteration”, the square root of a function is usually called its iterative root. is solved by the fractional iterates of a function :   x     f x   =  f f x   I R

n

I R

n

  x   f   x     f x   = f f x     f2 x   = n x   fm x   = f  x   fm n



x   =

=

f   x x y y

=

f  x x y y    

f

3 5

• A Functional Equation

The exponential notation of the iteration of functions can be extended beyond integer exponents:

• means
• for positive integers are the well known iterations of
• denotes the identity function,
• is the inverse funktion of
• is the -th iteration of the inverse of
• is the -th iterative root of
• is the
• th iteration of the -th iterative root or fractional iterate of

The family forms the continuous iteration group of . Within this the translation equation is satisfied. fn x   f 1 f f n n f f0 f0 x   x = f

1 –

f f

n –

n f f 1 n



n f fm n



m n f ft x   f f a

b +

x   f a f b x     =

Generalized Iteration

Map this to a Network

 

x

 x   f x   f

share weights

f1 n



x

 fm n



= f x  

  

1 m n

            

x

f x  

loop n times

• Weight Copy: Train only the last layer and copy the weights continously

backwards

• Weight Sharing: Initialize corresponding weights with equal values and

sum up all delivered by the network learning rule

• Weight Coupling: Start with different values and let the corresponding

weights of the iteration layers approach each other by a term like

• Regularization: Add a penalty term to the error function which assigns an

error to the weight-differences to regularize the network. This allows to uti- lize second order gradient methods like quasi Newton for faster training.

• Exact Gradient: Compute the exact gradients for an iterated Network

wi  wi   wj wi –   =

Training Methods

1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 Start Velocity [m/s] End Velocity [m/s] measurement for v1 (training data) physics for vm (prediction task) fractional iterates (network results)

v1 f v0   = f 0 v0   v0 = vm f 1 2



v0   = f 1 4



f3 4



1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 Start Velocity [m/s] End Velocity [m/s] measurement for v1 (training data) physics for vm (prediction task) fractional iterates (network results)

v1 f v0   = vm f 1 2



v0   = f 0 v0   v0 = f3 4



f 1 4



no friction with friction The Network results are conform to the laws of physics up to a mean error of 10-6

Results for „The Fall“

0.2 0.4 0.6 0.8 1 1.2 1 2 3 4 5 6 7 8 9 10 2 4 6 8 10 12 h [m] v0 [m/s] v [m/s]

Training Data trajectories iterative roots

v0

v(h)

v=f(v0,h)

h

The Embedding Problem

One of the most important functional equations: The Eigenvalue problem of functional calculus. Transform to:

 f x     c x   =

f x    1

–

c x     =

invert

  1

–

x

 x   f x  

c train

            

f

The Schröder Equation

Commuting Functions

 f x     f  x     x x   f f

weight sharing

• utputs

targets

 x  

 f x     f  x     =

• The steel bands are processed by N identical stands in a row
• ,

are known and can be measured

• 

                      x2 x1 xout           f xin p1 ,   f x1 p2 ,   F xin p1pN ,        f xN

1 –

pN ,   xin

Measuring

p1 p2 pN

instrument

=set of parameters like force, heat, strip thikness and width... pi

xin pi xout xout F xin p1...pN    f ...f f xin p1    p2   ... pN ,   = =

Steel Mill Model

Steel Mill Network

For a given autoregressive Box-Jenkins AR(n) timeseries , we define the function : which maps the vector of the last n samples

• ne step into the future

as and can simply write now. The discrete time evolution of the the system can be calculated using the matrix powers of F: . xt akxt

k – k 1 = n



= F Rn Rn  xt

1 –

xt

1 –

 xt

n –

  = xt xt xt

1 –

 xt

n 1 –   –

   = F a1 a2  an 1 0 0 1 0 0 1 = xt F xt

1 –

 = xt

n +

Fn xt

1 –

 =

Timeseries Interpolation

This autoregressive system is called linear embeddable if the matrix power exists also for all real . This is the case if can be decomposed into with being a diagonal matrix consisting of the eigenvalues

• f and being an invertible square matrix which columns are the eigenvec-

tors of . Additionally all must be non-negative to have a linear and real embedding, otherwise we will get a complex embedding. Then we can obtain with Now we have a continuous function and the interpolation of the

• riginal time series

consists of the first element of . Ft t R+  F F S A S 1

–

  = A i F S F i Ft S At S 1

–

  = At 1

t 0

0  0 0 n

t

= x t   Ft x0  = x t   x

Using Generalized Matrix Powers

The Fibonacci series , , is generated by and . By eigenvalue decomposition of we get That is Binet’s formula in the first component x0 = x1 1 = xt xt

1 –

xt

2 –

+ = F 1 1 1 0 = x1 1 0    = F xt

1 +

Ftx1 SAtS 1

– x1

= =

1 5 + 2

• 1

5 – 2

• 1

1

1 5 + 2

• 

  t 1 5 – 2

• 

  t

1

5

• 1

2

• 1

2 5

• –

1

5

• –

1 2

• 1

2 5

• +

1 = xt 1 5

• 1

5 + 2

• 

  

t

1 5 – 2

• 

  

t

– =

Example: A continuous Fibonacci Function

A time series of yearly snapshots from a discrete non linear Lotka-Volterra type predator - prey system (x = hare, y = lynx) is used as training data: and From these samples we calculate the monthly population by use of a neural network based method to compute iterative roots and fractional iterates. The given method provides a natural way to estimate not only the values over a year, but also to extrapolate arbitrarily smooth into the future. xt

1 +

1 a b yt – +   xt = yt

1 +

1 c – d xt +   yt =

Nonlinear Example

2 4 6 8 10 12 1 2 3 4 5 6 7 Prey Predator