The harmonic analysis of kernel functions Mattia Zorzi Department - PowerPoint PPT Presentation

The harmonic analysis of kernel functions Mattia Zorzi Department of Information Engineering University of Padova Cison di Valmarino, September 26th, 2016 Joint work with: A. Chiuso (University of Padova)

Kernels in system identification Model class (e.g. OE models) ∞ � y t = g s u t − s + e t g t impulse response s = 1 Gaussian linear regression model     y 1 g 1 . y N = Φ θ + e N y N := g 2 .  , θ :=     .   .  . . y N θ ∼ N ( 0 , K ) , K kernel function g t is modeled as Gaussian process with zero mean and covariance function Cov [ g t g s ] = K ( t , s ) M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 2 / 15

Kernels in system identification Model class (e.g. OE models) e t u t y t ∞ � y t = g s u t − s + e t g t impulse response s = 1 Gaussian linear regression model     y 1 g 1 . y N = Φ θ + e N y N := g 2 .  , θ :=     .   .  . . y N θ ∼ N ( 0 , K ) , K kernel function g t is modeled as Gaussian process with zero mean and covariance function Cov [ g t g s ] = K ( t , s ) M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 2 / 15

Kernels in system identification Model class (e.g. OE models) e t u t y t ∞ � y t = g s u t − s + e t g t impulse response s = 1 Gaussian linear regression model     y 1 g 1 . y N = Φ θ + e N y N := g 2 .  , θ :=     .   .  . . y N θ ∼ N ( 0 , K ) , K kernel function g t is modeled as Gaussian process with zero mean and covariance function Cov [ g t g s ] = K ( t , s ) M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 2 / 15

Kernels in system identification Model class (e.g. OE models) e t u t y t ∞ � y t = g s u t − s + e t g t impulse response s = 1 Gaussian linear regression model     y 1 g 1 . y N = Φ θ + e N y N := g 2 .  , θ :=     .   .  . . y N θ ∼ N ( 0 , K ) , K kernel function g t is modeled as Gaussian process with zero mean and covariance function Cov [ g t g s ] = K ( t , s ) M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 2 / 15

Kernels in system identification (cont’d) K encodes the a priori information on g t Our a priori information on the impulse response: BIBO stable Frequency content Question How to embed this information in the kernel function? M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 3 / 15

Kernels in system identification (cont’d) K encodes the a priori information on g t Our a priori information on the impulse response: BIBO stable Frequency content Question How to embed this information in the kernel function? M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 3 / 15

Kernels in system identification (cont’d) K encodes the a priori information on g t Our a priori information on the impulse response: Bode Diagram 1 20 0.8 Magnitude (dB) 0.6 0 0.4 BIBO stable −20 0.2 g t −40 0 −180 −0.2 −225 Phase (deg) −0.4 −270 Frequency content −0.6 −315 −0.8 −360 0 10 20 30 40 50 60 70 80 90 100 10 −1 10 0 10 1 t Frequency (rad/s) Question How to embed this information in the kernel function? M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 3 / 15

Kernels in system identification (cont’d) K encodes the a priori information on g t Our a priori information on the impulse response: Bode Diagram 1 20 0.8 Magnitude (dB) 0.6 0 0.4 BIBO stable −20 0.2 g t −40 0 −180 −0.2 −225 Phase (deg) −0.4 −270 Frequency content −0.6 −315 −0.8 −360 0 10 20 30 40 50 60 70 80 90 100 10 −1 10 0 10 1 t Frequency (rad/s) Question How to embed this information in the kernel function? M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 3 / 15

Modeling the impulse response How? g t as a sum of damped sinusoids M 2 t cos ( ω k t + ∠ c k ) | c k | e − α k � g t = k = 1 c k complex Gaussian random variable such that: ◮ c k is zero mean ◮ Cov ( c k , ¯ c j ) = p k δ k − j ◮ Cov ( c k , c j ) = 0 M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 4 / 15

Modeling the impulse response How? 1 0.8 0.6 g t as a sum of damped sinusoids α k 2 t e − 0.4 g t 0.2 0 M ω k 2 t cos ( ω k t + ∠ c k ) | c k | e − α k � g t = −0.2 −0.4 k = 1 −0.6 0 10 20 30 40 50 60 70 80 90 100 t c k complex Gaussian random variable such that: ◮ c k is zero mean ◮ Cov ( c k , ¯ c j ) = p k δ k − j ◮ Cov ( c k , c j ) = 0 M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 4 / 15

Modeling the impulse response How? 1 0.8 0.6 g t as a sum of damped sinusoids α k 2 t e − 0.4 g t 0.2 0 M ω k 2 t cos ( ω k t + ∠ c k ) | c k | e − α k � g t = −0.2 −0.4 k = 1 −0.6 0 10 20 30 40 50 60 70 80 90 100 t c k complex Gaussian random variable such that: ◮ c k is zero mean ◮ Cov ( c k , ¯ c j ) = p k δ k − j ◮ Cov ( c k , c j ) = 0 M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 4 / 15

Modeling the impulse response (cont’d) Sum of damped sinusoids in a grid α j 2 t cos ( ω i t + ∠ c ij ) � � | c ij | e − g t = i j “Infinite dense sum” of damped sinusoids � ∞ � ∞ 2 t cos ( ω t + ∠ c ( α, ω )) d ω d α | c ( α, ω ) | e − α g t = 0 −∞ c ( α, ω ) generalized Fourier transform of g t M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 5 / 15

Modeling the impulse response (cont’d) α 2 α 1 ! 1 Sum of damped sinusoids in a grid k = 1 ! 2 α j 2 t cos ( ω i t + ∠ c ij ) k = 2 � � | c ij | e − g t = i j “Infinite dense sum” of damped sinusoids � ∞ � ∞ 2 t cos ( ω t + ∠ c ( α, ω )) d ω d α | c ( α, ω ) | e − α g t = 0 −∞ c ( α, ω ) generalized Fourier transform of g t M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 5 / 15

Modeling the impulse response (cont’d) α 2 α 1 ! 1 Sum of damped sinusoids in a grid k = 1 ! 2 α j 2 t cos ( ω i t + ∠ c ij ) k = 2 � � | c ij | e − g t = i j “Infinite dense sum” of damped sinusoids � ∞ � ∞ 2 t cos ( ω t + ∠ c ( α, ω )) d ω d α | c ( α, ω ) | e − α g t = 0 −∞ c ( α, ω ) generalized Fourier transform of g t M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 5 / 15

Modeling the impulse response (cont’d) α 2 α 1 ! 1 Sum of damped sinusoids in a grid k = 1 ! 2 α j 2 t cos ( ω i t + ∠ c ij ) k = 2 � � | c ij | e − g t = i j “Infinite dense sum” of damped sinusoids � ∞ � ∞ 2 t cos ( ω t + ∠ c ( α, ω )) d ω d α | c ( α, ω ) | e − α g t = 0 −∞ c ( α, ω ) generalized Fourier transform of g t M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 5 / 15

Harmonic analysis Let � ∞ � ∞ 2 t cos ( ω t + ∠ c ( α, ω )) d ω d α | c ( α, ω ) | e − α g t = 0 −∞ Harmonic representation of the kernel function K � ∞ � ∞ K ( t , s ) = 1 p ( α, ω ) e − α t + s 2 cos ( ω ( t − s )) d ω d α 2 0 −∞ p ( α, ω ) generalized power spectral density M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 6 / 15

Harmonic analysis Let � ∞ � ∞ 2 t cos ( ω t + ∠ c ( α, ω )) d ω d α | c ( α, ω ) | e − α g t = 0 −∞ Harmonic representation of the kernel function K � ∞ � ∞ K ( t , s ) = 1 p ( α, ω ) e − α t + s 2 cos ( ω ( t − s )) d ω d α 2 0 −∞ p ( α, ω ) generalized power spectral density M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 6 / 15

Harmonic analysis Let � ∞ � ∞ 2 t cos ( ω t + ∠ c ( α, ω )) d ω d α | c ( α, ω ) | e − α g t = 0 −∞ Harmonic representation of the kernel function K � ∞ � ∞ K ( t , s ) = 1 p ( α, ω ) e − α t + s 2 cos ( ω ( t − s )) d ω d α 2 0 −∞ p ( α, ω ) generalized power spectral density M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 6 / 15

Harmonic analysis Let � ∞ � ∞ 2 t cos ( ω t + ∠ c ( α, ω )) d ω d α | c ( α, ω ) | e − α g t = 0 −∞ Harmonic representation of the kernel function K � ∞ � ∞ K ( t , s ) = 1 p ( α, ω ) e − α t + s 2 cos ( ω ( t − s )) d ω d α 2 0 −∞ p ( α, ω ) generalized power spectral density M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 6 / 15

Harmonic analysis Let � ∞ � ∞ 2 t cos ( ω t + ∠ c ( α, ω )) d ω d α | c ( α, ω ) | e − α g t = 0 −∞ Harmonic representation of the kernel function K � ∞ � ∞ K ( t , s ) = 1 p ( α, ω ) e − α t + s 2 cos ( ω ( t − s )) d ω d α 2 0 −∞ ! p ( α ; ! ) 0 α p ( α, ω ) generalized power spectral density M. Zorzi The harmonic analysis of kernel functions September 26th, 2016 6 / 15