S p e c t r a l Me t h o d s i n o u r S P i C - PowerPoint PPT Presentation

The UK’s European university S p e c t r a l Me t h o d s i n o u r S P i C e ’ 1 6 S u b mi s s i o n F a r h a n a F e r d o u s i L i z a a n d Ma r e k G r z e s S c h o o l o f C o m p u t i n g , U n i v e r s i t y o f K e n t , U K

Our Team Hidden Markov Models Natural Language Processing Spectral Learning

Our Score The highest score among methods that did not use Neural Networks

Initial Attempts

Spectral learning for HMMs (Hsu et al. 2012) Observable Operator Model for HMMs Empirical moment calculation: U defines an m-dimensional subspace that preserves the state dynamics. Transformed operators for HMMs

The Main Parameters of the Method • The number of hidden states

Main Methods

Weighted Finite Automata and Sequence Prediction Balle et. al. (EMNLP 2014)

Hankel Matrix Balle et al. (EMNLP 2014)

The Basis Balle et al. (EMNLP 2014)

The Main Parameters of the Method • The number of hidden states • The basis • The basis can be chosen from a sub-block of the Hankel matrix where the rows and columns correspond to the substrings and the cells correspond to the frequencies of the substrings in the data. • Therefore, the maximum length of the substrings can be considered as a parameter

Parameter Tuning • A combination of (manual) coordinate ascent and random search • Why random search? (BERGSTRA AND BENGIO (2012))

Other Methods • 3-gram model with smoothing worked better than spectral learning on 3 problems

Experimental results (1) • The Spectral Method did well on problems 1, 2, 3, 9,12 • Presumably, those problems have small numbers of hidden states No of st at e s vs Score ( Sm all num ber of st at es) 0 . 9 5 D a t a s e t 1 D a t a s e t 2 0 . 9 D a t a s e t 9 D a t a s e t 3 D a t a s e t 1 2 0 . 8 5 0 . 8 e r o 0 . 7 5 c S 0 . 7 0 . 6 5 0 . 6 0 . 5 5 0 5 0 1 0 0 1 5 0 2 0 0 N o o f S t a t e s

Experimental result (2) • Score prediction is invariant to changes in the number of states on problems 4, 5, 7, 8,10,11,13 No of st at es vs Score 0 . 6 D a t a s e t 4 D a t a s e t 5 D a t a s e t 7 0 . 5 5 D a t a s e t 8 D a t a s e t 1 0 D a t a s e t 1 3 0 . 5 0 . 4 5 e r o c S 0 . 4 0 . 3 5 0 . 3 0 . 2 5 0 2 0 0 4 0 0 6 0 0 8 0 0 1 0 0 0 N o o f S t a t e s

Experimental result (3) • On problems 5, 8 and 10, 3-gram with smoothing gave slightly batter results than the corresponding spectral approach spectral vs n-gram 0 . 6 5 0 . 6 0 . 5 5 e r o 0 . 5 c S 0 . 4 5 0 . 4 S p e c t r a l 3 - g r a m w i t h K N s m o o t h i n g 0 . 3 5 4 5 6 7 8 9 1 0 P r o b l e m n o

The Final Parameter Values for WFA

T H E U K ’ S E U R O P E A N U N I V E R S I T Y w w w . k e n t . a c . u k