Sequential Implementation of Monte Carlo Tests with Uniformly - PowerPoint PPT Presentation

Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk Axel Gandy Department of Mathematics Imperial College London a.gandy@imperial.ac.uk useR! 2009, Rennes July 8-10, 2009

Introduction ◮ Test statistic T , reject for large values. ◮ Observation: t . ◮ p -value: p = P( T ≥ t ) Often not available in closed form. ◮ Monte Carlo Test: n p naive = 1 � ˆ I ( T i ≥ t ) , n i =1 where T , T 1 , . . . T n i.i.d. ◮ Examples: ◮ Bootstrap, ◮ Permutation tests. ◮ Goal: Estimate p using few X i Mainly interested in deciding if p ≤ α for some α . Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 2

Introduction ◮ Test statistic T , reject for large values. ◮ Observation: t . ◮ p -value: p = P( T ≥ t ) Often not available in closed form. ◮ Monte Carlo Test: n p naive = 1 � ˆ I ( T i ≥ t ) , n � �� � i =1 =: X i ∼ B (1 , p ) where T , T 1 , . . . T n i.i.d. ◮ Examples: ◮ Bootstrap, ◮ Permutation tests. ◮ Goal: Estimate p using few X i Mainly interested in deciding if p ≤ α for some α . Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 2

Introduction ◮ Test statistic T , reject for large values. ◮ Observation: t . ◮ p -value: p = P( T ≥ t ) Often not available in closed form. ◮ Monte Carlo Test: n p naive = 1 � ˆ I ( T i ≥ t ) , n � �� � i =1 =: X i ∼ B (1 , p ) where T , T 1 , . . . T n i.i.d. ◮ Examples: ◮ Bootstrap, ◮ Permutation tests. ◮ Goal: Estimate p using few X i Mainly interested in deciding if p ≤ α for some α . Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 2

Introduction ◮ Test statistic T , reject for large values. ◮ Observation: t . ◮ p -value: p = P( T ≥ t ) Often not available in closed form. ◮ Monte Carlo Test: n p naive = 1 � ˆ I ( T i ≥ t ) , n � �� � i =1 =: X i ∼ B (1 , p ) where T , T 1 , . . . T n i.i.d. ◮ Examples: ◮ Bootstrap, ◮ Permutation tests. ◮ Goal: Estimate p using few X i Mainly interested in deciding if p ≤ α for some α . Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 2

Sequential approaches based on S n = � n i =1 X i 10 9 8 ◮ Stop once S n ≥ U n or 7 S n ≤ L n 6 S n ◮ τ : hitting time 5 ◮ Compute ˆ 4 p based on S τ and τ . 3 2 ◮ Hit B U : decide p > α , 1 ◮ Hit B L : decide p ≤ α , 0 0 1 2 3 4 5 6 7 8 9 10 n Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 3

Sequential approaches based on S n = � n i =1 X i 10 9 8 ◮ Stop once S n ≥ U n or 7 S n ≤ L n 6 S n ◮ τ : hitting time 5 ◮ Compute ˆ 4 p based on S τ and τ . 3 2 ◮ Hit B U : decide p > α , 1 ◮ Hit B L : decide p ≤ α , 0 0 1 2 3 4 5 6 7 8 9 10 n Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 3

Sequential approaches based on S n = � n i =1 X i 10 9 8 ◮ Stop once S n ≥ U n or 7 S n ≤ L n 6 S n ◮ τ : hitting time 5 ◮ Compute ˆ 4 p based on S τ and τ . 3 2 ◮ Hit B U : decide p > α , 1 ◮ Hit B L : decide p ≤ α , 0 0 1 2 3 4 5 6 7 8 9 10 n Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 3

Previous Approaches ◮ Besag & Clifford (1991): S n h 0 n m 0 ◮ (Truncated) Sequential Probability Ratio Test, Fay et al. (2007) S n h 0 n m 0 ◮ R-package MChtest. Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 4

What do we really want? Is p ≤ α ? Two individuals using the same statistical method on the same data should arrive at the same conclusion. First law of applied statistics, Gleser (1996) Consider the resampling risk � P p (ˆ p > α ) if p ≤ α, RR p (ˆ p ) ≡ P p (ˆ p ≤ α ) if p > α. Want: sup RR p (ˆ p ) ≤ ǫ p ∈ [0 , 1] for some (small) ǫ > 0. For Besag & Clifford (1991), SPRT: sup p RR P ≥ 0 . 5 Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 5

What do we really want? Is p ≤ α ? Two individuals using the same statistical method on the same data should arrive at the same conclusion. First law of applied statistics, Gleser (1996) Consider the resampling risk � P p (ˆ p > α ) if p ≤ α, RR p (ˆ p ) ≡ P p (ˆ p ≤ α ) if p > α. Want: sup RR p (ˆ p ) ≤ ǫ p ∈ [0 , 1] for some (small) ǫ > 0. For Besag & Clifford (1991), SPRT: sup p RR P ≥ 0 . 5 Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 5

What do we really want? Is p ≤ α ? Two individuals using the same statistical method on the same data should arrive at the same conclusion. First law of applied statistics, Gleser (1996) Consider the resampling risk � P p (ˆ p > α ) if p ≤ α, RR p (ˆ p ) ≡ P p (ˆ p ≤ α ) if p > α. Want: sup RR p (ˆ p ) ≤ ǫ p ∈ [0 , 1] for some (small) ǫ > 0. For Besag & Clifford (1991), SPRT: sup p RR P ≥ 0 . 5 Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 5

What do we really want? Is p ≤ α ? Two individuals using the same statistical method on the same data should arrive at the same conclusion. First law of applied statistics, Gleser (1996) Consider the resampling risk � P p (ˆ p > α ) if p ≤ α, RR p (ˆ p ) ≡ P p (ˆ p ≤ α ) if p > α. Want: sup RR p (ˆ p ) ≤ ǫ p ∈ [0 , 1] for some (small) ǫ > 0. For Besag & Clifford (1991), SPRT: sup p RR P ≥ 0 . 5 Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 5

What do we really want? Is p ≤ α ? Two individuals using the same statistical method on the same data should arrive at the same conclusion. First law of applied statistics, Gleser (1996) Consider the resampling risk � P p (ˆ p > α ) if p ≤ α, RR p (ˆ p ) ≡ P p (ˆ p ≤ α ) if p > α. Want: sup RR p (ˆ p ) ≤ ǫ p ∈ [0 , 1] for some (small) ǫ > 0. For Besag & Clifford (1991), SPRT: sup p RR P ≥ 0 . 5 Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 5

What do we really want? Is p ≤ α ? Two individuals using the same statistical method on the same data should arrive at the same conclusion. First law of applied statistics, Gleser (1996) Consider the resampling risk � P p (ˆ p > α ) if p ≤ α, RR p (ˆ p ) ≡ P p (ˆ p ≤ α ) if p > α. Want: sup RR p (ˆ p ) ≤ ǫ p ∈ [0 , 1] for some (small) ǫ > 0. For Besag & Clifford (1991), SPRT: sup p RR P ≥ 0 . 5 Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 5

Recursive Definition of the Boundaries Want: sup p RR p (ˆ p ) ≤ ǫ Suffices to ensure P α (hit B U ) ≤ ǫ P α (hit B L ) ≤ ǫ Recursive definition: Given U 1 , . . . , U n − 1 and L 1 , . . . , L n − 1 , define ◮ U n as the minimal value such that P α (hit B U until n ) ≤ ǫ n ◮ and L n as the maximal value such that P α (hit B L until n ) ≤ ǫ n where ǫ n ≥ 0 with ǫ n ր ǫ (spending sequence). Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 6

Recursive Definition of the Boundaries Want: sup p RR p (ˆ p ) ≤ ǫ Suffices to ensure P α (hit B U ) ≤ ǫ P α (hit B L ) ≤ ǫ Recursive definition: Given U 1 , . . . , U n − 1 and L 1 , . . . , L n − 1 , define ◮ U n as the minimal value such that P α (hit B U until n ) ≤ ǫ n ◮ and L n as the maximal value such that P α (hit B L until n ) ≤ ǫ n where ǫ n ≥ 0 with ǫ n ր ǫ (spending sequence). Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 6

Recursive Definition of the Boundaries Want: sup p RR p (ˆ p ) ≤ ǫ Suffices to ensure P α (hit B U ) ≤ ǫ P α (hit B L ) ≤ ǫ Recursive definition: Given U 1 , . . . , U n − 1 and L 1 , . . . , L n − 1 , define ◮ U n as the minimal value such that P α (hit B U until n ) ≤ ǫ n ◮ and L n as the maximal value such that P α (hit B L until n ) ≤ ǫ n where ǫ n ≥ 0 with ǫ n ր ǫ (spending sequence). Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 6

Recursive Definition of the Boundaries Want: sup p RR p (ˆ p ) ≤ ǫ Suffices to ensure P α (hit B U ) ≤ ǫ P α (hit B L ) ≤ ǫ Recursive definition: Given U 1 , . . . , U n − 1 and L 1 , . . . , L n − 1 , define ◮ U n as the minimal value such that P α (hit B U until n ) ≤ ǫ n ◮ and L n as the maximal value such that P α (hit B L until n ) ≤ ǫ n where ǫ n ≥ 0 with ǫ n ր ǫ (spending sequence). Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 6

Recursive Definition - Example n ◮ α = 0 . 2, ǫ n = 0 . 4 5+ n . ◮ U n =the minimal value such that P α (hit B U until n ) ≤ ǫ n ◮ L n = maximal value such that P α (hit B L until n ) ≤ ǫ n n = 0 P α ( S n = k , τ ≥ n ) k= 3 k= 2 k= 1 k= 0 1 0 ǫ n U n 1 L n -1 Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 7