Analyzing pupil dilation data with Generalized Additive Models - PowerPoint PPT Presentation

Analyzing pupil dilation data with Generalized Additive Models (GAMs) Jacolien van Rij & Martijn Wieling Thu, June 27, 2013 Ÿ LOT School 2013, Groningen

Pupil dilation • Primarily determined by light and accommodation re fl exes • Also ‘psychophysiological index of dynamic brain activity in human cognition.’ (Beatty & Lucero-Wagoner, 2000) o slow movements o small scale movements, about .5 mm in pupillary diameter • First reports from 1875 in Germany, rediscovered in 1960s • not precisely clear what it means: attention, e ff ort, processing load, working memory load, emotion, … 2

Characteristics • Hoeks & Levelt (1993): Estimation of pupil dilation response by mathematical function o peak around 930 ms after trigger pupil dilation from baseline pupil dilation from baseline pupil dilation from baseline pupil dilation from baseline pupil dilation from baseline 0.1 0.1 0.1 0.1 0.1 peak latency peak dilation 0.05 0.05 0.05 0.05 0.05 0 0 0 0 0 0 0 0 0 0 1000 1000 1000 1000 1000 2000 2000 2000 2000 2000 3000 3000 3000 3000 3000 4000 4000 4000 4000 4000 5000 5000 5000 5000 5000 Time (ms) Time (ms) Time (ms) Time (ms) Time (ms)

Pupillometry in language processing research • Pupil dilation is a precise and consistent measure of processing activity during on-line language processing • Sensitive to (among others): o grammatical complexity (Just & Carpenter, 1993) o integration of prosodic information and visual context (Engelhardt et al., 2010) o focus prosody (Zellin et al., 2011) 4

Overview • Experimental study • About the data • Analyses using GAMs o random e ff ects structure o testing fi xed e ff ects o model criticism o autocorrelation • Discussion 5

Experimental study Van Rij, 2012; Van Rij, Van Rijn, & Hendriks, in prep. 6

Experimental study • Question: is the processing of object pronouns in fl uenced by context? • Example: The penguin is hitting him with a pan. • Object pronouns are mainly guided by grammatical principles (e.g. Binding Theory; Chomsky, 1981) o In contrast to subject pronouns ( he , she ) 7

Why is this interesting? • Investigates the underlying mechanism of sentence processing • Debate: o Initial- fi lter accounts: grammar fi rst fi lters ungrammatical referents (a.o., Nicol & Swinney, 1989; Clifton et al., 1997; Lewis et al., 2012) § Hypothesis: no early e ff ect of context, late e ff ect may be possible o Competing-constraints accounts: grammar competes with other factors, such as context (a.o., Badecker & Straub, 2002; Clackson et al., 2012) § Hypothesis: early e ff ect of context 8

2x2 within-subjects design • Test sentence: The penguin is hitting him with a pan. • Manipulating linguistic context: o AP (agent-patient): Here you see a penguin and a sheep. o PA (patient-agent): Here you see a sheep and a penguin. • Manipulating visual context: ✓ Congruent: picture matches sentence ✗ Incongruent: picture does not match sentence 9

Experimental study • Question: is the processing of object pronouns in fl uenced by context? • Example: The penguin is hitting him with a pan. • Task: Is the picture congruent with this sentence? • Participants see possible answers on screen 2500 ms after sentence o ff set 10

About the data 11

Raw pupil data ¤ time ¤ x ¤ y ¤ pupil 15871914 480.5 327.1 1201.0 ... 15871918 480.2 327.0 1197.0 ... 15871922 479.9 327.3 1194.0 ... 15871926 479.9 327.8 1194.0 ... 15871930 479.7 328.5 1191.0 ... 15871934 479.5 330.0 1188.0 ... 15871938 479.5 330.1 1188.0 ... • Eye tracker: 15871942 479.3 330.0 1188.0 ... Eye Link 1000, 15871946 479.6 330.8 1187.0 ... 15871950 480.6 331.1 1186.0 ... monocular 250 Hz 15871954 481.5 331.3 1183.0 ... 15871958 481.2 329.9 1180.0 ... MSG MSG 15871960 15871960 >TrialStart TrialStart 50000 50000 3 3 15871962 480.6 329.1 1178.0 ... 15871966 479.5 328.8 1176.0 ... • Markers / messages to 15871970 479.1 329.4 1175.0 ... align trials and stimuli 15871974 480.1 328.4 1178.0 ... 12

x-gaze 1000 Timestamp 750 Raw pupil data 500 250 • Pupil dilation is continuous 0 5233282 5242778 signal y-gaze • Slow changing signal, in 750 Timestamp contrast with gaze data 500 250 0 • Preprocessing 5233282 5242778 o remove blinks pupil area 500 o ( fi lter + down sampling) 400 Timestamp 300 o baseline + normalization 200 100 0 5233282 5242778 Timestamp 13

Linear interpolation of blinks pupil area 500 Timestamp 250 0 5233282 5242778 14

Filtering, down sampling pupil area 500 250 0 5233282 5242778 Timestamp • To avoid aliasing, low pass unfiltered 430 filtered fi lter of 25 Hz 405 • After fi ltering, down sampled to 50 Hz 380 (original frequency 250 Hz) 5235500 5236500 15

Baseline & normalization • Pupil size can be measured in di ff erent units o area, diameter, or arbitrary unit - depends on eye tracker / camera • Converted to pupil dilation: o pupil <- (pupil_size - baseline) / baseline o baseline: average pupil size 50 ms before to 50 ms after onset pronoun • Note: R package for preprocessing of pupil dilation data 16

Align data on pronoun onset • Averaged subject means per condition (±1SE): lot of variation 17

Analyses using GAMs Is object pronoun interpretation in fl uenced by context? 18

Data structure > str(dat) 'data.frame': 78907 obs. of 19 variables: $ Subject : Factor w/ 17 levels • 17 participants, 21-32 pronoun $ Item : Factor w/ 32 levels $ trial : int (3-66) trials each $ Time : int $ pTime : int (-199 - 2983) • 32 items, 4 variants $ trialTime : int $ Context : num (0=PA, 1=AP) (AP-congr, AP-incongr, $ Congruency : num (0=match, 1=mismatch) PA-congr, PA-incongr) $ Interaction : num (Context * Congruency) $ Cond : Factor w/ 4 levels $ ActorRight : num (0=no, 1=yes) $ StartS2 : int • Time window: -200 ms before $ StartVerb : int pronoun onset to 3000 ms after $ Anafoor : int $ EndAnafoor : int pronoun onset $ EndS2 : int $ dilationFiltered: num $ baseline : num $ Pupil : num (-.64 - .64) $ prev : num (first NA!) 19

Baseline model > summary( m0 <- gam(Pupil ~ s(pTime), data=dat) ) ... Parametric coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 0.0136331 0.0003368 40.48 <2e-16 *** --- Approximate significance of smooth terms: edf Ref.df F p-value s(pTime) 8.123 8.793 666.2 <2e-16 *** --- R-sq.(adj) = 0.0691 Deviance explained = 6.92% fREML score = -74097 Scale est. = 0.008945 n = 78907 20

Baseline model > m0 <- gam(Pupil ~ s(pTime), data=dat) § Sanity check: plot model smooths > plot(m0, rug=F) > abline(h=0) 0.04 0.04 > abline(v=c(0,1000), lty=3) > abline(h=-1*coef(m0)[1], s(pTime,8.12) s(pTime,8.12) col='red', lty=2) 0.00 0.00 ¡ intercept adjustment -0.04 -0.04 0 0 500 500 1500 1500 2500 2500 pTime pTime 21

Variation between subjects -0.05 0.00 0.05 0.10 -0.05 0.00 0.05 0.10 -0.05 0.00 0.05 0.10 0 0 0 500 500 500 1500 1500 1500 2500 2500 2500 -0.05 0.00 0.05 0.10 -0.05 0.00 0.05 0.10 -0.05 0.00 0.05 0.10 0 0 0 500 500 500 1500 1500 1500 2500 2500 2500 -0.05 0.00 0.05 0.10 -0.05 0.00 0.05 0.10 -0.05 0.00 0.05 0.10 0 0 0 500 500 500 1500 1500 1500 2500 2500 2500 -0.05 0.00 0.05 0.10 -0.05 0.00 0.05 0.10 -0.05 0.00 0.05 0.10 0 0 0 500 500 500 1500 1500 1500 2500 2500 2500 -0.05 0.00 0.05 0.10 -0.05 0.00 0.05 0.10 -0.05 0.00 0.05 0.10 0 0 0 500 500 500 1500 1500 1500 2500 2500 2500 -0.05 0.00 0.05 0.10 -0.05 0.00 0.05 0.10 0 0 500 500 1500 1500 2500 2500 22

Di ff erent types of random e ff ects with GAMs 1. Random intercept: s(Item, bs="re") 2. Random intercept + random slope: s(Item, pTime,bs="re") 3. Random wiggly curve: s(pTime, Subject, bs="fs", m=1) s(pTime) s(pTime,Subject) s(pTime,Subject) • Additive e ff ects: 0.15 0.05 0.02 0.1 0 0.05 0 0 -0.05 -0.04 -0.05 0 500 1500 2500 0 500 1500 2500 0 500 1500 2500 pTime pTime pTime Pupil ~ s(Time) + s(Time, Subject, Pupil ~ s(Time) + s(Time, Subject, bs bs=" ="fs fs", m=1) ", m=1) Pupil ~ s(Time, Subject, Pupil ~ s(Time, Subject, bs bs=" ="fs fs", m=1) ", m=1) 23

Random effects – use AIC Test random e ff ects > m0 <- bam(Pupil ~ s(pTime,k=10), data=dat) > m0a <- bam(Pupil ~ s(pTime,k=10) + s(pTime, Subject, bs='fs', m=1), data=dat) > AIC(m0)-AIC(m0a) [1] 5483.468 > m0b <- bam(Pupil ~ s(pTime, k=10) + s(pTime, Subject, bs='fs', m=1) + s(Item, bs='re'), data=dat) > AIC(m0a)-AIC(m0b) [1] 2756.84 24