Lifting the curse of dimensionality in nonlinear system - - PowerPoint PPT Presentation

▶
lifting the curse of dimensionality in nonlinear system
SMART_READER_LITE
LIVE PREVIEW

Lifting the curse of dimensionality in nonlinear system - - PowerPoint PPT Presentation

Sep 20, 2022 38 likes •573 views

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Kim Batselier, Zhongming Chen, Ching-Yun Ko, Ngai Wong 1 /

slide-1
SLIDE 1

Lifting the curse of dimensionality in nonlinear system identification with tensor networks.

Lifting the curse of dimensionality in nonlinear system identification with tensor networks.

Kim Batselier, Zhongming Chen, Ching-Yun Ko, Ngai Wong

1 / 30

slide-2
SLIDE 2

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Introduction

System identification

System S f(·) Inputs u(t) Outputs y(t) . . . . . .

Main Problem Given measured inputs {u(t) ∈ Rp}N

t=1, outputs {y(t) ∈ Rl}N t=1,

and a parametric input-output mapping f(·) for the system S, estimate the parameters of f(·).

2 / 30

slide-3
SLIDE 3

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Introduction

NFIR black box model y(t) = f(u(t), u(t − 1), . . . , u(t − M + 1)) with nonlinear function f(·)

3 / 30

slide-4
SLIDE 4

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Introduction

NFIR black box model y(t) = f(u(t), u(t − 1), . . . , u(t − M + 1)) with nonlinear function f(·) Taylor expansion of f(·) : NFIR ⇒ Volterra series

3 / 30

slide-5
SLIDE 5

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Introduction

NFIR black box model y(t) = f(u(t), u(t − 1), . . . , u(t − M + 1)) with nonlinear function f(·) Taylor expansion of f(·) : NFIR ⇒ Volterra series SISO truncated Volterra series: y(t) = h0 +

d

  • i=1

M−1

  • k1,...,ki=0

hi(k1, . . . , ki)

i

  • j=1

u(t − kj),

3 / 30

slide-6
SLIDE 6

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Introduction

NFIR black box model y(t) = f(u(t), u(t − 1), . . . , u(t − M + 1)) with nonlinear function f(·) Taylor expansion of f(·) : NFIR ⇒ Volterra series SISO truncated Volterra series: y(t) = h0 +

d

  • i=1

M−1

  • k1,...,ki=0

hi(k1, . . . , ki)

i

  • j=1

u(t − kj), MIMO truncated Volterra series: each of the l outputs in y(t) ∈ Rl is a multivariate polynomial in u(t), u(t − 1), . . . , u(t − M + 1).

3 / 30

slide-7
SLIDE 7

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Introduction

Multivariate polynomials n-variate polynomial of total degree d Number of coefficients is d+n

n

  • ≈ nd, e.g.

7+20

20

  • = 888030

4 / 30

slide-8
SLIDE 8

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Introduction

Multivariate polynomials n-variate polynomial of total degree d Number of coefficients is d+n

n

  • ≈ nd, e.g.

7+20

20

  • = 888030

Curse of dimensionality Truncated Volterra series are described by an exponential amount

  • f unknown coefficients ⇒ need to estimate an exponential

amount of coefficients.

4 / 30

slide-9
SLIDE 9

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Introduction

Multivariate polynomials n-variate polynomial of total degree d Number of coefficients is d+n

n

  • ≈ nd, e.g.

7+20

20

  • = 888030

Curse of dimensionality Truncated Volterra series are described by an exponential amount

  • f unknown coefficients ⇒ need to estimate an exponential

amount of coefficients. Main message of this talk Lifting the curse of dimensionality in this system identification problem with tensor networks.

4 / 30

slide-10
SLIDE 10

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks

Tensor network building blocks A a a A

5 / 30

slide-11
SLIDE 11

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks

Tensor network building blocks A a a A Three tensor network diagram rules

1 Nodes in the network are tensors 2 Each edge of each node is a particular index of the tensor 3 Connected edges refer to summation over that particular index

5 / 30

slide-12
SLIDE 12

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks

Tensor network example 1 C(i, j) =

k A(i, k) B(k, j)

6 / 30

slide-13
SLIDE 13

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks

Tensor network example 1 C(i, j) =

k A(i, k) B(k, j) is visually represented by

A B i k j

6 / 30

slide-14
SLIDE 14

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks

Tensor network example 1 C(i, j) =

k A(i, k) B(k, j) is visually represented by

A B i k j Tensor network example 2 a b k

6 / 30

slide-15
SLIDE 15

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks

Tensor network example 1 C(i, j) =

k A(i, k) B(k, j) is visually represented by

A B i k j Tensor network example 2 a b k

  • k a(k) b(k) = aT b

6 / 30

slide-16
SLIDE 16

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks

Tensor network example 3 A B C D

7 / 30

slide-17
SLIDE 17

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks

Tensor network example 3 A B C D Trace(ABCD) = Trace(BCDA) = · · · = Trace(DABC)

7 / 30

slide-18
SLIDE 18

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks

Tensor network example 4 Singular value decomposition of a rank-r matrix A ∈ Rp×q: A = U S V T with U ∈ Rp×r, V ∈ Rq×r orthogonal and S ∈ Rr×r diagonal.

8 / 30

slide-19
SLIDE 19

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks

Tensor network example 4 Singular value decomposition of a rank-r matrix A ∈ Rp×q: A = U S V T with U ∈ Rp×r, V ∈ Rq×r orthogonal and S ∈ Rr×r diagonal. U S r V T r p q

8 / 30

slide-20
SLIDE 20

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks

Outer product C = a bT = a ◦ b

9 / 30

slide-21
SLIDE 21

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks

Outer product C = a bT = a ◦ b C(i, j) = a(i) b(j)

9 / 30

slide-22
SLIDE 22

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks

Outer product C = a bT = a ◦ b C(i, j) = a(i) b(j) C(i, j) =

1

  • k=1

a(i, k) b(k, j)

9 / 30

slide-23
SLIDE 23

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks

Outer product C = a bT = a ◦ b C(i, j) = a(i) b(j) C(i, j) =

1

  • k=1

a(i, k) b(k, j) a b 1

9 / 30

slide-24
SLIDE 24

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Tensor Networks

Outer product C = a bT = a ◦ b C(i, j) = a(i) b(j) C(i, j) =

1

  • k=1

a(i, k) b(k, j) a b 1 Important take-home message Tensor networks with small interconnection dimensions represent tensors with interrelated entries.

9 / 30

slide-25
SLIDE 25

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network

MIMO Volterra model Each of the l outputs in y(t) ∈ Rl is a multivariate polynomial in u(t), u(t − 1), . . . , u(t − M + 1).

10 / 30

slide-26
SLIDE 26

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network

MIMO Volterra model Each of the l outputs in y(t) ∈ Rl is a multivariate polynomial in u(t), u(t − 1), . . . , u(t − M + 1). Define input and output vectors y(t) :=    y1(t) . . . yl(t)    ∈ Rl ut :=

  • 1

u(t)T · · · u(t − M + 1)T T ∈ R(pM+1) All input monomials are contained in

d

  • ut ◦ ut ◦ · · · ◦ ut ∈ R(pM+1)×···×(pM+1)

10 / 30

slide-27
SLIDE 27

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network

Small example: p = 1, l = 1, M = 2 and d = 2 ut =   1 u(t) u(t − 1)   ut ◦ ut =   1 u(t) u(t − 1)   1 u(t) u(t − 1)

  • =

  1 u(t) u(t − 1) u(t) u(t)2 u(t) u(t − 1) u(t − 1) u(t) u(t − 1) u(t − 1)2   ut ut 1

11 / 30

slide-28
SLIDE 28

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network

Symmetric rank-1 input tensor

d

  • ut ◦ ut ◦ · · · ◦ ut ∈ R(pM+1)×···×(pM+1)

pM + 1 ut ut ut 1 1 1 pM + 1 pM + 1

12 / 30

slide-29
SLIDE 29

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network

Small example: p = 1, l = 1, M = 2 and d = 2 y(t) = f(u(t), u(t − 1)) = vT vec(ut ◦ ut) =

  • v1

v2 v3 v4 · · · v9

        1 u(t) u(t − 1) u(t) . . . u(t − 1)2         

13 / 30

slide-30
SLIDE 30

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network

MIMO Volterra system y(t) = V vec (ut ◦ ut ◦ · · · ◦ ut) V ∈ Rl×(pM+1)d contains coefficients of l polynomials

14 / 30

slide-31
SLIDE 31

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network

MIMO Volterra system y(t) = V vec (ut ◦ ut ◦ · · · ◦ ut) V ∈ Rl×(pM+1)d contains coefficients of l polynomials V matrix as a tensor network V(1) V(2) V(d) l pM + 1 pM + 1 pM + 1 R2 R3 Rd

14 / 30

slide-32
SLIDE 32

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network

MIMO Volterra system y(t) = V vec (ut ◦ ut ◦ · · · ◦ ut) V ∈ Rl×(pM+1)d contains coefficients of l polynomials V matrix as a tensor network V(1) V(2) V(d) l pM + 1 pM + 1 pM + 1 R2 R3 Rd Lifting the curse of dimensionality Storage of tensor network requires ≈ O(d(pM + 1)R2) elements

14 / 30

slide-33
SLIDE 33

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network

MIMO Volterra tensor network V(1) V(2) V(d) y(t) ut ut ut l l pM + 1 pM + 1 pM + 1 R2 R3 Rd

15 / 30

slide-34
SLIDE 34

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network

MIMO Volterra tensor network V(1) V(2) V(d) y(t) ut ut ut l l pM + 1 pM + 1 pM + 1 R2 R3 Rd Tensor Network math

R2

  • r2=1

· · ·

Rd

  • rd=1

pnu+1

  • i1=1

· · ·

pnu+1

  • id=1

1

  • j2=1

· · ·

1

  • jd=1

V(1)(l, i1, r2)V(2)(r2, i2, r3) V(d−1)(rd−1, id, rd) V(d)(rd, id) ut(i1, j2) ut(j2, i2, j3) · · · ut(jd, id).

15 / 30

slide-35
SLIDE 35

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network System identification algorithm

MIMO Volterra system identification problem Given measured inputs {u(t) ∈ Rp}N

t=1, outputs {y(t) ∈ Rl}N t=1,

and an unknown MIMO Volterra model V , estimate the tensor network tensors V(1), V(2), . . . , V(d).

16 / 30

slide-36
SLIDE 36

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network System identification algorithm

MIMO Volterra system identification problem Given measured inputs {u(t) ∈ Rp}N

t=1, outputs {y(t) ∈ Rl}N t=1,

and an unknown MIMO Volterra model V , estimate the tensor network tensors V(1), V(2), . . . , V(d). Alternating Linear Scheme (ALS) initialize unknown tensors V(1), V(2), . . . , V(d) for 1 ≤ k ≤ d

Assume V(1), . . . , V(k−1), V(k+1), . . . , V(d) known and update V(k). Update V(k) ⇒ solve small linear system for vec(V(k))

16 / 30

slide-37
SLIDE 37

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network System identification algorithm

Rk+1 V(1) V(2) ut ut l V(d) ut V(d−1) ut V(k) ut Rk pM + 1

17 / 30

slide-38
SLIDE 38

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network System identification algorithm

Rk+1 V(1) V(2) ut ut l V(d) ut V(d−1) ut V(k) ut Rk pM + 1

18 / 30

slide-39
SLIDE 39

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network System identification algorithm

Rk+1 l V(k) Rk pM + 1

19 / 30

slide-40
SLIDE 40

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network System identification algorithm

Rk+1 l V(k) Rk pM + 1

20 / 30

slide-41
SLIDE 41

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network System identification algorithm

l vec(V(k)) Rk(pM + 1)Rk+1 At

21 / 30

slide-42
SLIDE 42

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network System identification algorithm

l vec(V(k)) Rk(pM + 1)Rk+1 At y(t) = At vec(V(k))

21 / 30

slide-43
SLIDE 43

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network System identification algorithm

Updating V(k)      y(1) y(2) . . . y(N)      =      A1 A2 . . . AN      vec(V(k))

22 / 30

slide-44
SLIDE 44

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network System identification algorithm

MIMO Volterra ALS system identification algorithm Initialize unknown tensors V(1), V(2), . . . , V(d) While stopping criterion not satisfied

For k = 1 : d, Update V(k) For k = d − 1 : −1 : 2, Update V(k)

End while

23 / 30

slide-45
SLIDE 45

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network System identification algorithm

MIMO Volterra ALS system identification algorithm Initialize unknown tensors V(1), V(2), . . . , V(d) While stopping criterion not satisfied

For k = 1 : d, Update V(k) For k = d − 1 : −1 : 2, Update V(k)

End while Remarks Stopping criterion: fixed number of updates, residual on ||Y − ˆ Y ||F /||Y ||F , ... How to choose R2, . . . , Rd? (⇒ Modified ALS updates the ranks automatically) Orthogonalization step after solving the linear system ensures numerical stability.

23 / 30

slide-46
SLIDE 46

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network Experiment

Numerical Experiment Generated Volterra system with d = 10 and M = 7. The ith symmetric Volterra kernel contains entries hi(k1, . . . , ki) = e−.1 i

j=1 k2 j .

white noise input: 700 samples for identification, 2000 samples for validation. relative approximation error ||Y − ˆ Y ||/||Y || ≤ 10−4.

24 / 30

slide-47
SLIDE 47

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network Experiment 25 / 30

slide-48
SLIDE 48

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network Experiment 26 / 30

slide-49
SLIDE 49

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network Experiment

Experiment: double balanced mixer for upconversion 2 inputs: LO = 100Hz sine, IF= 300Hz square-wave.

  • utput= RF.

Sampling frequency: 5kHz for 1 second Five different noise levels added to RF output: SNR=11dB,13dB,16dB,19dB,25dB.

(image from arXiv:physics/0608211) 27 / 30

slide-50
SLIDE 50

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. MIMO Volterra tensor network Experiment

Experiment M = 2, d = 11 ⇒ (pM + 1)d = 511 = 48828125 R2 = · · · = R11 = 5 ⇒ Each TN node has 125 unknowns Identification takes around 2 seconds on standard desktop pc

28 / 30

slide-51
SLIDE 51

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Other system identification applications

Time-varying MIMO Volterra identification V (t + 1)T = V (t)T + ∆V (t)T y(t)T = vec(ut ◦ · · · ◦ ut)T V (t)T + e(t) Time-varying linear state space “State” is a matrix V T ∈ R(pM+1)d×l Tensor network Kalman filter for the recursive estimation of V

29 / 30

slide-52
SLIDE 52

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Other system identification applications

Time-varying MIMO Volterra identification V (t + 1)T = V (t)T + ∆V (t)T y(t)T = vec(ut ◦ · · · ◦ ut)T V (t)T + e(t) Time-varying linear state space “State” is a matrix V T ∈ R(pM+1)d×l Tensor network Kalman filter for the recursive estimation of V “Polynomial in the input” state space x(t + 1) = A x(t) + B vec(ut ◦ · · · ◦ ut) y(t) = C x(t) + D vec(ut ◦ · · · ◦ ut). MOESP subspace identification with Tensor Networks

29 / 30

slide-53
SLIDE 53

Lifting the curse of dimensionality in nonlinear system identification with tensor networks. Conclusion

Open source MATLAB/Octave implementations https://github.com/kbatseli/ References

Batselier K., Chen Z., Wong N., Tensor Network alternating linear scheme for MIMO Volterra system identification, Automatica, vol. 84, 2017, pp. 26-35 Batselier K., Chen Z., Wong N., A Tensor Network Kalman filter with an application in recursive MIMO Volterra system identification, Automatica,

  • vol. 84, 2017, pp. 17-25

Batselier K., Wong N., Matrix output extension of the tensor network Kalman filter with an application in MIMO Volterra system identification, Automatica, vol. 95, 2018, pp. 413-418 Batselier K., Ko C.-Y., Wong N., Tensor network subspace identification

  • f polynomial state space models, Automatica, vol. 95, 2018, pp.

187-196.

30 / 30