Really? Using the nullabor package to learn if what we see is - - PowerPoint PPT Presentation

▶
really using the nullabor package to learn if what we see
SMART_READER_LITE
LIVE PREVIEW

Really? Using the nullabor package to learn if what we see is - - PowerPoint PPT Presentation

May 20, 2023 475 likes •686 views

Really? Using the nullabor package to learn if what we see is really there Di Cook, Monash University Joint with Hadley Wickham, Heike Hofmann, Niladri Roy Chowdhury, Mahbub Majumder Photo by Lyn Cook Outline Why? lineup, rorschach

slide-1
SLIDE 1

Photo by Lyn Cook

Really? Using the nullabor package to learn if what we see is really there

Di Cook, Monash University

Joint with Hadley Wickham, Heike Hofmann, Niladri Roy Chowdhury, Mahbub Majumder

slide-2
SLIDE 2

−20 WOMBAT 2016, Melbourne, Australia

Outline

Why? lineup, rorschach functions null generating mechanisms p-values metrics

2

slide-3
SLIDE 3

200 400 600 800

9 12 3 6 9 12 3

N deposition g m-1 y-1

200 400 600 800

biomass g m-1

12 3 6 9 12

“Biomass really looks related to nitrogen deposition, but none of my tests show a significant relationship!” Ecologist colleague

slide-4
SLIDE 4

“These four species of wasps have very different gene expression patterns” Published paper 2010

slide-5
SLIDE 5

“Is it possible that the pollsters are systematically biased?” Our US election monitoring

slide-6
SLIDE 6

−20 WOMBAT 2016, Melbourne, Australia

Why inference?

Plots of data allow us to uncover the unexpected, but it needs to be calibrated against what might be seen by chance, if there really is no underlying pattern Classical statistical inference allows computing probabilities of this being a likely value of a statistic if there really is no structure

6

slide-7
SLIDE 7

−20 WOMBAT 2016, Melbourne, Australia

Inference

Once you see it, its too late You cannot legitimately test for significance of structure

7

slide-8
SLIDE 8

−20 WOMBAT 2016, Melbourne, Australia

nullabor

Lineup protocol: Plots your data among a field of “null” plots Puts it in the context of what it might look like if there is really no structure Encrypts the location of the data plot

8

slide-9
SLIDE 9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 200 400 600 800 200 400 600 800 200 400 600 800 200 400 600 800 3 6 9 12 3 6 9 12 3 6 9 12 3 6 9 12 3 6 9 12

N deposition g m-1 y-1 biomass g m-1

> decrypt("fg0t DARA up iYzuRuYp Fl") [1] "True data in position 10"

slide-10
SLIDE 10

LD1 LD2

−6 −4 −2 2 4 −6 −4 −2 2 4 −6 −4 −2 2 4 −6 −4 −2 2 4 1

  • ●
  • 6
  • ●
  • ●
  • 11
  • ●
  • ●
  • 16
  • ●
  • −10

−5 0 5 2

  • ●
  • 7
  • ●
  • 12
  • 17
  • ●
  • ●
  • ●
  • −10

−5 0 5 3

  • ●
  • 8
  • ●
  • ●
  • ●
  • 13
  • ●
  • ●
  • 18
  • ●
  • ●
  • ●
  • −10

−5 0 5 4

  • ●
  • ●
  • 9
  • ●
  • 14
  • ●
  • ●
  • 19
  • ●
  • −10

−5 0 5 5

  • 10
  • ●
  • 15
  • ●
  • 20
  • ●
  • −10

−5 0 5 Group

  • F
  • G
  • Q
  • W
slide-11
SLIDE 11
  • 1

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 −5 5 10 −5 5 10 −5 5 10 −5 5 10 RasmussenGallup Other Fox RasmussenGallup Other Fox RasmussenGallup Other Fox RasmussenGallup Other Fox RasmussenGallup Other Fox

Difference in %

> decrypt("fg0t DARA up iYzuRuYp Q") [1] "True data in position 5"

slide-12
SLIDE 12

−20 WOMBAT 2016, Melbourne, Australia

nullabor functions

lineup: Generates a lineup using one of the given null generating mechanisms pvisuals: Compute p-values after showing to impartial jurers distmet: empirical distribution of distance between data plot and null plots

12

slide-13
SLIDE 13

−20 WOMBAT 2016, Melbourne, Australia

Demo

13

slide-14
SLIDE 14

Table 1: Comparison of visual inference with traditional hypothesis testing. Mathematical Inference Visual Inference Hypothesis H0 : µ1 = µ2 vs Ha : µ1 = µ2 H0 : µ1 = µ2 vs Ha : µ1 = µ2 Test Statistic T(y) =

¯ y1−¯ y2 s q

1 n1 + 1 n2

T(y) =

50 100 150 200 site A site B

Site Conc (mg/kg)

label site A site B

Sampling Distribution fT (y)(t);

!tn!1(! 2) 0 tn!1(! 2)

fT (y)(t);

Site Conc (mg/kg) 50 100 150 200 50 100 150 200 50 100 150 200 50 100 150 200 1 6 11 16 site A site B 2 7 12 17 site A site B 3 8 13 18 site A site B 4 9 14 19 site A site B 5 10 15 20 site A site B label site A site B

Reject H0 if

  • bserved T is extreme
  • bserved plot is identifiable
slide-15
SLIDE 15

−20 WOMBAT 2016, Melbourne, Australia

Visual p-values

For one observer, the probability of randomly selecting the data plot is 1/m, where m is the number of plots in the lineup. With multiple observers, the p-value is estimated by

15

Number of independent observers Number of observers choosing data plot

P(X ≥ x) = 1 − BinomK,1/m(x − 1) =

K

⇤

i=x

K i ⇥ 1 m ⇥i m − 1 m ⇥K−i

slide-16
SLIDE 16

−20 WOMBAT 2016, Melbourne, Australia

Null generators

null_dist: Null hypothesis: variable has specified distribution null_lm: Null hypothesis: variable is linear combination of predictors, comes with different residual generators null_permute: Null hypothesis: variable is independent of others

16

slide-17
SLIDE 17

−20 WOMBAT 2016, Melbourne, Australia

Distance metrics

Can we measure how different the data plot is from the null plots?

17

(a) Dataset X with two variables X1 and X2

  • 3
6 5.0 7.5 10.0 12.5 15.0 X1 X2 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 p q 1 2 3 4 5 Count

1 1 1 1 1 1 5 2 2 1 1 5 1 5 2 4 1 1 1 2 2 1 1 3 2 1 1

1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 p q

(b) Dataset Y with permuted X1 and original X2

  • 3
6 5.0 7.5 10.0 12.5 15.0 Permuted X1 X2 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 p q 1 2 3 4 Count

1 1 1 1 1 1 1 4 1 2 1 1 1 3 1 1 1 1 3 2 3 2 1 1 2 1 1 1 1 1 2 1 1 1 1 1

1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 p q

Difference

slide-18
SLIDE 18
slide-19
SLIDE 19
slide-20
SLIDE 20

−20 WOMBAT 2016, Melbourne, Australia

Summary

Really useful package Helps to adjust our expectations, dampen surprise, support surprise Calibrate your eyes on what randomness looks like

20