Common hypotheses - X and Y Suppose we have two univariate samples: x 1 , x 2 , . . . , x m and y 1 , y 2 , . . . y n . We model these as each being independent realizations of some univariate random variable, say X and Y respectively. Suppose we are interested in the hypothesis H 0 : E ( X ) = E ( Y ). ◮ we need a discrepancy measure; how about d = | x − y | ? This may or may not be the best discrepancy measure but unusually large values would be evidence against the hypothesis. ◮ we also need to be able to generate new realizations x i and y i when H 0 holds. This requires a little more thinking. How to generate the data under H 0 is is not obvious.
Common hypotheses - X and Y Suppose we have two univariate samples: x 1 , x 2 , . . . , x m and y 1 , y 2 , . . . y n . We model these as each being independent realizations of some univariate random variable, say X and Y respectively. Suppose we are interested in the hypothesis H 0 : E ( X ) = E ( Y ). ◮ we need a discrepancy measure; how about d = | x − y | ? This may or may not be the best discrepancy measure but unusually large values would be evidence against the hypothesis. ◮ we also need to be able to generate new realizations x i and y i when H 0 holds. This requires a little more thinking. How to generate the data under H 0 is is not obvious. Suppose we assume that both X and Y are continuous random variables.
Common hypotheses - X and Y Suppose we have two univariate samples: x 1 , x 2 , . . . , x m and y 1 , y 2 , . . . y n . We model these as each being independent realizations of some univariate random variable, say X and Y respectively. Suppose we are interested in the hypothesis H 0 : E ( X ) = E ( Y ). ◮ we need a discrepancy measure; how about d = | x − y | ? This may or may not be the best discrepancy measure but unusually large values would be evidence against the hypothesis. ◮ we also need to be able to generate new realizations x i and y i when H 0 holds. This requires a little more thinking. How to generate the data under H 0 is is not obvious. Suppose we assume that both X and Y are continuous random variables. ◮ If we had Q X ( u ), say for all u ∈ (0 , 1), then
Common hypotheses - X and Y Suppose we have two univariate samples: x 1 , x 2 , . . . , x m and y 1 , y 2 , . . . y n . We model these as each being independent realizations of some univariate random variable, say X and Y respectively. Suppose we are interested in the hypothesis H 0 : E ( X ) = E ( Y ). ◮ we need a discrepancy measure; how about d = | x − y | ? This may or may not be the best discrepancy measure but unusually large values would be evidence against the hypothesis. ◮ we also need to be able to generate new realizations x i and y i when H 0 holds. This requires a little more thinking. How to generate the data under H 0 is is not obvious. Suppose we assume that both X and Y are continuous random variables. ◮ If we had Q X ( u ), say for all u ∈ (0 , 1), then we could generate new x i s by generating uniform U (0 , 1) realizations and letting x i = Q X ( u i ); similarly for Y .
Common hypotheses - X and Y Suppose we have two univariate samples: x 1 , x 2 , . . . , x m and y 1 , y 2 , . . . y n . We model these as each being independent realizations of some univariate random variable, say X and Y respectively. Suppose we are interested in the hypothesis H 0 : E ( X ) = E ( Y ). ◮ we need a discrepancy measure; how about d = | x − y | ? This may or may not be the best discrepancy measure but unusually large values would be evidence against the hypothesis. ◮ we also need to be able to generate new realizations x i and y i when H 0 holds. This requires a little more thinking. How to generate the data under H 0 is is not obvious. Suppose we assume that both X and Y are continuous random variables. ◮ If we had Q X ( u ), say for all u ∈ (0 , 1), then we could generate new x i s by generating uniform U (0 , 1) realizations and letting x i = Q X ( u i ); similarly for Y . We could use the observed values to produce estimates � Q X ( u ) and use these instead.
Common hypotheses - X and Y Suppose we have two univariate samples: x 1 , x 2 , . . . , x m and y 1 , y 2 , . . . y n . We model these as each being independent realizations of some univariate random variable, say X and Y respectively. Suppose we are interested in the hypothesis H 0 : E ( X ) = E ( Y ). ◮ we need a discrepancy measure; how about d = | x − y | ? This may or may not be the best discrepancy measure but unusually large values would be evidence against the hypothesis. ◮ we also need to be able to generate new realizations x i and y i when H 0 holds. This requires a little more thinking. How to generate the data under H 0 is is not obvious. Suppose we assume that both X and Y are continuous random variables. ◮ If we had Q X ( u ), say for all u ∈ (0 , 1), then we could generate new x i s by generating uniform U (0 , 1) realizations and letting x i = Q X ( u i ); similarly for Y . We could use the observed values to produce estimates � Q X ( u ) and use these instead. ◮ But we still need to ensure that E ( X ) = E ( Y ) for the generated data!
Common hypotheses - X and Y We might generate the new x i s from an estimated quantile function � Q X ( u ), but one whose mean we have adjusted to be the same as that of the estimated quantile function � Q Y ( u ).
Common hypotheses - X and Y We might generate the new x i s from an estimated quantile function � Q X ( u ), but one whose mean we have adjusted to be the same as that of the estimated quantile function � Q Y ( u ). How about generateFromMeanShiftData <- function (data, size = length (data), mu = 0) { dataWithMeanMu <- data - mean (data) + mu quantile (dataWithMeanMu, probs = runif (size)) }
Common hypotheses - X and Y We might generate the new x i s from an estimated quantile function � Q X ( u ), but one whose mean we have adjusted to be the same as that of the estimated quantile function � Q Y ( u ). How about generateFromMeanShiftData <- function (data, size = length (data), mu = 0) { dataWithMeanMu <- data - mean (data) + mu quantile (dataWithMeanMu, probs = runif (size)) } Together with discrepancyMeans <- function (data) { # assume data is a list of named components x and y x <- data $ x y <- data $ y abs ( mean (x) - mean (y)) } we should be able to get a test of the hypothesis that could at least guide our exploration.
Common hypotheses - X and Y First a function to get new x and y realizations simultaneously: # First we need a function to generate new x and y at once generateXYcommonMean <- function (data){ # assume data is a list containing x and y x <- data $ x y <- data $ y # return new data as a list of x and y newx <- generateFromMeanShiftData (x) newy <- generateFromMeanShiftData (y) list (x = newx, y = newy) }
Common hypotheses - X and Y We put these pieces together to test our hypothesis H 0 : E ( X ) = E ( Y ).
Common hypotheses - X and Y We put these pieces together to test our hypothesis H 0 : E ( X ) = E ( Y ). Assuming all of the previous functions, we can write a test function more generally as numericalTestXY <- function (x, y = NULL, nReps = 2000, discrepancyFn = discrepancyMeans, generateXY = generateXYcommonMean, returnStats = FALSE){ data <- if ( is.null (y)) { list (x = x[[1]], y = x[[2]]) } else { list (x = x, y = y)} dObserved <- discrepancyFn (data) discrepancies <- sapply (1 : nReps, FUN = function (i){ newData <- generateXY (data) discrepancyFn (newData) } ) result <- mean (discrepancies >= dObserved) if (returnStats){ result <- list (p = result, dobs = dObserved, d = discrepancies)} result }
Common hypotheses - X and Y We put these pieces together to test our hypothesis H 0 : E ( X ) = E ( Y ). Assuming all of the previous functions, we can write a test function more generally as numericalTestXY <- function (x, y = NULL, nReps = 2000, discrepancyFn = discrepancyMeans, generateXY = generateXYcommonMean, returnStats = FALSE){ data <- if ( is.null (y)) { list (x = x[[1]], y = x[[2]]) } else { list (x = x, y = y)} dObserved <- discrepancyFn (data) discrepancies <- sapply (1 : nReps, FUN = function (i){ newData <- generateXY (data) discrepancyFn (newData) } ) result <- mean (discrepancies >= dObserved) if (returnStats){ result <- list (p = result, dobs = dObserved, d = discrepancies)} result } Note that, as written, this is not peculiar to the hypothesis H 0 : E ( X ) = E ( Y ).
Common hypotheses - X and Y We put these pieces together to test our hypothesis H 0 : E ( X ) = E ( Y ). Assuming all of the previous functions, we can write a test function more generally as numericalTestXY <- function (x, y = NULL, nReps = 2000, discrepancyFn = discrepancyMeans, generateXY = generateXYcommonMean, returnStats = FALSE){ data <- if ( is.null (y)) { list (x = x[[1]], y = x[[2]]) } else { list (x = x, y = y)} dObserved <- discrepancyFn (data) discrepancies <- sapply (1 : nReps, FUN = function (i){ newData <- generateXY (data) discrepancyFn (newData) } ) result <- mean (discrepancies >= dObserved) if (returnStats){ result <- list (p = result, dobs = dObserved, d = discrepancies)} result } Note that, as written, this is not peculiar to the hypothesis H 0 : E ( X ) = E ( Y ). It should work for other numerical tests provided it was handed the correct functions.
Common hypotheses - X and Y We can try it out on a few examples.
Common hypotheses - X and Y We can try it out on a few examples. x <- rnorm (35) y <- rnorm (45, mean = 0.1) numericalTestXY (x,y) ## [1] 0.1775 y <- rnorm (45, mean = 0.5) numericalTestXY (x,y) ## [1] 0.002 y <- y + 0.01 numericalTestXY (x,y) ## [1] 5e-04 x <- rnorm (35, sd = 2) y <- rnorm (45, mean = 0.1) numericalTestXY (x,y) ## [1] 0.616
Common hypotheses - X and Y We can try it out on some real data.
Common hypotheses - X and Y We can try it out on some real data. First on comparing average weights of cars. # Compare average weights boxplot (wt ~ am, data = mtcars, col = "grey") 5 4 wt 3 2 0 1 am automatic <- mtcars $ am == 0 manual <- ! automatic numericalTestXY (mtcars $ wt[automatic], mtcars $ wt[manual]) ## [1] 0
Common hypotheses - X and Y Now on comparing average time to reach a quarter mile. # Compare average time to a quarter mile boxplot (qsec ~ am, data = mtcars, col = "grey") 22 20 qsec 18 16 0 1 am result <- numericalTestXY (mtcars $ qsec[automatic], mtcars $ qsec[manual], returnStats = TRUE) result $ p ## [1] 0.1575
Common hypotheses - X and Y Can get a histogram of all the discrepancies # Compare average weights hist (result $ d, col = "grey", xlab = "discrepancy", main = "Testing mean qsec by am") abline (v = result $ dobs, lty =2, lwd = 2, col = "red") legend ("topright", legend = c ( paste0 ("discrepancy is ", round (result $ dobs,3)), paste0 ("p-value is ", round (result $ p,3))))
Common hypotheses - X and Y Can get a histogram of all the discrepancies # Compare average weights hist (result $ d, col = "grey", xlab = "discrepancy", main = "Testing mean qsec by am") abline (v = result $ dobs, lty =2, lwd = 2, col = "red") legend ("topright", legend = c ( paste0 ("discrepancy is ", round (result $ dobs,3)), paste0 ("p-value is ", round (result $ p,3)))) Testing mean qsec by am discrepancy is 0.823 500 p−value is 0.158 400 Frequency 300 200 100 0 0.0 0.5 1.0 1.5 2.0 discrepancy
Same with pictures Could the same hypothesis be tested using a graphical discrepancy measure and a line up test?
Same with pictures Could the same hypothesis be tested using a graphical discrepancy measure and a line up test? Here’s one graphical discrepancy measure: compareBoxplots <- function (data, subjectNo) { y <- data $ y # assume data is a list of x and y x <- data $ x group <- factor ( c ( rep ("x", length (x)), rep ("y", length (y)))) # create data for boxplots bp_data <- data.frame (vals = c (x,y), group = group) boxplot (vals ~ group, data = bp_data, main = paste (subjectNo), cex.main = 4, xlab = "", ylab = "", col = adjustcolor ( c ("firebrick", "steelblue"), 0.5), xaxt = "n", yaxt = "n" # turn off axes ) } Note: ◮ ‘data‘ is now a list of two named vectors “x‘ and ‘y‘, and ◮ boxplots are to be compared only by difference in locations, nothing else.
The lineup tests with boxplots To test H 0 : E ( X ) = E ( Y ) with boxplots, we need only generate the new x s and ys as before. data <- list (x = mtcars $ wt[automatic], y = mtcars $ wt[manual]) lineup (data, generateSubject = generateXYcommonMean, showSubject = compareBoxplots, layout= c (4, 5))
The lineup tests with boxplots To test H 0 : E ( X ) = E ( Y ) with boxplots, we need only generate the new x s and ys as before. data <- list (x = mtcars $ wt[automatic], y = mtcars $ wt[manual]) lineup (data, generateSubject = generateXYcommonMean, showSubject = compareBoxplots, layout= c (4, 5)) Notes: ◮ The lineup() function was not changed.
The lineup tests with boxplots To test H 0 : E ( X ) = E ( Y ) with boxplots, we need only generate the new x s and ys as before. data <- list (x = mtcars $ wt[automatic], y = mtcars $ wt[manual]) lineup (data, generateSubject = generateXYcommonMean, showSubject = compareBoxplots, layout= c (4, 5)) Notes: ◮ The lineup() function was not changed. ◮ The lineup test looks very much like its numerical counterpart,
The lineup tests with boxplots To test H 0 : E ( X ) = E ( Y ) with boxplots, we need only generate the new x s and ys as before. data <- list (x = mtcars $ wt[automatic], y = mtcars $ wt[manual]) lineup (data, generateSubject = generateXYcommonMean, showSubject = compareBoxplots, layout= c (4, 5)) Notes: ◮ The lineup() function was not changed. ◮ The lineup test looks very much like its numerical counterpart, ◮ instead of a discrepancyFn function it has a showSubject function. The logic is identical.
A line up test. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 True Location: log(2.7810636828402e+77, base=33) - 39
A line up test. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 True Location: log(2.7810636828402e+77, base=33) - 39 = 12
The lineup tests with boxplots - qsec Test H 0 : E ( X ) = E ( Y ) with qsec . data <- list (x = mtcars $ qsec[automatic], y = mtcars $ qsec[manual]) lineup (data, generateSubject = generateXYcommonMean, showSubject = compareBoxplots, layout= c (4, 5))
A line up test. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 True Location: log(1.27173474825649e+90, base=30) - 43
A line up test. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 True Location: log(1.27173474825649e+90, base=30) - 43 = 18
Overlaid quantile plots as discrepancy Or, we could test H 0 : E ( X ) = E ( Y ) using a different graphical discrepancy measure, say overlaid quantile plots. compareQuantiles <- function (data, subjectNo) { y <- sort (data $ y) # assume data is a list of x and y x <- sort (data $ x) ylim <- extendrange ( c (x,y)) n <- length (y) m <- length (x) py <- ppoints (n) px <- ppoints (m) plot (px, x, ylim = ylim, xlim = c (0,1), main = paste (subjectNo), cex.main = 4, xlab = "", ylab = "", col = adjustcolor ("firebrick", 0.75), pch = 17, cex = 4, xaxt = "n", yaxt = "n" # turn off axes ) points (py, y, col = adjustcolor ("steelblue", 0.75), pch = 19, cex = 4) } Note: ◮ quantiles are to be compared only by difference in locations, that is height.
A line up test. Test H 0 : E ( X ) = E ( Y ), for wt data <- list (x = mtcars $ wt[automatic], y = mtcars $ wt[manual]) lineup (data, generateSubject = generateXYcommonMean, showSubject = compareQuantiles, layout= c (4, 5))
A line up test. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 True Location: log(8.59504455717143e+63, base=9) - 64
A line up test. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 True Location: log(8.59504455717143e+63, base=9) - 64 = 3
A line up test. Test H 0 : E ( X ) = E ( Y ), for qsec data <- list (x = mtcars $ qsec[automatic], y = mtcars $ qsec[manual]) lineup (data, generateSubject = generateXYcommonMean, showSubject = compareQuantiles, layout= c (4, 5))
A line up test. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 True Location: log(4.74284397516047e+80, base=16) - 60
A line up test. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 True Location: log(4.74284397516047e+80, base=16) - 60 = 7
A more general hypothesis What about the more general hypothesis involving two samples?
A more general hypothesis What about the more general hypothesis involving two samples? Namely, H 0 : F Y ( y ) = F X ( x ) for unknown F X ( x ) and F Y ( y ).
A more general hypothesis What about the more general hypothesis involving two samples? Namely, H 0 : F Y ( y ) = F X ( x ) for unknown F X ( x ) and F Y ( y ). The graphical discrepancies previously used would still work.
A more general hypothesis What about the more general hypothesis involving two samples? Namely, H 0 : F Y ( y ) = F X ( x ) for unknown F X ( x ) and F Y ( y ). The graphical discrepancies previously used would still work. But what about generating x s and y s when H 0 holds?
A more general hypothesis What about the more general hypothesis involving two samples? Namely, H 0 : F Y ( y ) = F X ( x ) for unknown F X ( x ) and F Y ( y ). The graphical discrepancies previously used would still work. But what about generating x s and y s when H 0 holds? If H 0 holds, then F Y ( y ) = F X ( x ) and the x s and y s are simply two different samples from the same distribution.
A more general hypothesis What about the more general hypothesis involving two samples? Namely, H 0 : F Y ( y ) = F X ( x ) for unknown F X ( x ) and F Y ( y ). The graphical discrepancies previously used would still work. But what about generating x s and y s when H 0 holds? If H 0 holds, then F Y ( y ) = F X ( x ) and the x s and y s are simply two different samples from the same distribution. That means we could combine the values x 1 , . . . , x m and y 1 , . . . , y n into a single sample of size m + n .
A more general hypothesis What about the more general hypothesis involving two samples? Namely, H 0 : F Y ( y ) = F X ( x ) for unknown F X ( x ) and F Y ( y ). The graphical discrepancies previously used would still work. But what about generating x s and y s when H 0 holds? If H 0 holds, then F Y ( y ) = F X ( x ) and the x s and y s are simply two different samples from the same distribution. That means we could combine the values x 1 , . . . , x m and y 1 , . . . , y n into a single sample of size m + n . Label the new sample as z 1 , . . . , z m , z m +1 , . . . , z m + n .
A more general hypothesis What about the more general hypothesis involving two samples? Namely, H 0 : F Y ( y ) = F X ( x ) for unknown F X ( x ) and F Y ( y ). The graphical discrepancies previously used would still work. But what about generating x s and y s when H 0 holds? If H 0 holds, then F Y ( y ) = F X ( x ) and the x s and y s are simply two different samples from the same distribution. That means we could combine the values x 1 , . . . , x m and y 1 , . . . , y n into a single sample of size m + n . Label the new sample as z 1 , . . . , z m , z m +1 , . . . , z m + n . The observed x s are simply a sample of m values (without replacement) from the z s; the y s are the remaining unsampled z s.
A more general hypothesis What about the more general hypothesis involving two samples? Namely, H 0 : F Y ( y ) = F X ( x ) for unknown F X ( x ) and F Y ( y ). The graphical discrepancies previously used would still work. But what about generating x s and y s when H 0 holds? If H 0 holds, then F Y ( y ) = F X ( x ) and the x s and y s are simply two different samples from the same distribution. That means we could combine the values x 1 , . . . , x m and y 1 , . . . , y n into a single sample of size m + n . Label the new sample as z 1 , . . . , z m , z m +1 , . . . , z m + n . The observed x s are simply a sample of m values (without replacement) from the z s; the y s are the remaining unsampled z s. We could generate new x s and new y s in exactly the same way.
A more general hypothesis What about the more general hypothesis involving two samples? Namely, H 0 : F Y ( y ) = F X ( x ) for unknown F X ( x ) and F Y ( y ). The graphical discrepancies previously used would still work. But what about generating x s and y s when H 0 holds? If H 0 holds, then F Y ( y ) = F X ( x ) and the x s and y s are simply two different samples from the same distribution. That means we could combine the values x 1 , . . . , x m and y 1 , . . . , y n into a single sample of size m + n . Label the new sample as z 1 , . . . , z m , z m +1 , . . . , z m + n . The observed x s are simply a sample of m values (without replacement) from the z s; the y s are the remaining unsampled z s. We could generate new x s and new y s in exactly the same way. Note: there are many other ways that we might use the z s to generate new x s and y s as well.
A more general hypothesis - generating new data Following, the proposed method, we write a new function to generate the data under H 0 : # First we need a function to generate new x and y at once mixRandomly <- function (data){ # assume data is a list containing x and y x <- data $ x y <- data $ y m <- length (x) n <- length (y) z <- c (x,y) xIndices <- sample (1 : (m + n), m, replace = FALSE) newx <- z[xIndices] newy <- z[ - xIndices] # the remainder list (x = newx, y = newy) }
A more general hypothesis - lineup test H 0 : F X ( x ) = F Y ( y ) can now be tested using either graphical discrepancy. With boxplots as lineup (data, generateSubject = mixRandomly, showSubject = compareBoxplots, layout= c (4, 5))
A more general hypothesis - lineup test H 0 : F X ( x ) = F Y ( y ) can now be tested using either graphical discrepancy. With boxplots as lineup (data, generateSubject = mixRandomly, showSubject = compareBoxplots, layout= c (4, 5)) or with quantile plots as lineup (data, generateSubject = mixRandomly, showSubject = compareQuantiles, layout= c (4, 5)) Only the graphical discrepancy changes.
A line up test H 0 : F X ( x ) = F Y ( y ) - wt 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 True Location: log(7.00340620250571e+124, base=22) - 83
A line up test H 0 : F X ( x ) = F Y ( y ) - wt 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 True Location: log(7.00340620250571e+124, base=22) - 83 = 10
A line up test H 0 : F X ( x ) = F Y ( y ) - wt 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 True Location: log(4.05362155971444e+67, base=7) - 63
A line up test H 0 : F X ( x ) = F Y ( y ) - wt 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 True Location: log(4.05362155971444e+67, base=7) - 63 = 17
A line up test H 0 : F X ( x ) = F Y ( y ) - qsec 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 True Location: log(2.13598703592091e+96, base=32) - 44
A line up test H 0 : F X ( x ) = F Y ( y ) - qsec 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 True Location: log(2.13598703592091e+96, base=32) - 44 = 20
A line up test H 0 : F X ( x ) = F Y ( y ) - wt 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 True Location: log(2.15356351058832e+79, base=21) - 49
A line up test H 0 : F X ( x ) = F Y ( y ) - wt 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 True Location: log(2.15356351058832e+79, base=21) - 49 = 11
A numerical test H 0 : F X ( x ) = F Y ( y ) A popular numerical test of this hypothesis is the Kolmogorov-Smirnov test which uses the maximum absolute difference between F X ( t ) and F Y ( t ) over all t .
A numerical test H 0 : F X ( x ) = F Y ( y ) A popular numerical test of this hypothesis is the Kolmogorov-Smirnov test which uses the maximum absolute difference between F X ( t ) and F Y ( t ) over all t . In R the function ks.test() computes the test and provides the discrepancy measure as its “statistic” and also a p -value usually based on some asymptotic approximation.
A numerical test H 0 : F X ( x ) = F Y ( y ) A popular numerical test of this hypothesis is the Kolmogorov-Smirnov test which uses the maximum absolute difference between F X ( t ) and F Y ( t ) over all t . In R the function ks.test() computes the test and provides the discrepancy measure as its “statistic” and also a p -value usually based on some asymptotic approximation. We could write a corresponding discrepancy measure for numericalTestXY() as ks.discrep <- function (data) { ks.test (data $ x,data $ y) $ statistic }
A numerical test H 0 : F X ( x ) = F Y ( y ) A popular numerical test of this hypothesis is the Kolmogorov-Smirnov test which uses the maximum absolute difference between F X ( t ) and F Y ( t ) over all t . In R the function ks.test() computes the test and provides the discrepancy measure as its “statistic” and also a p -value usually based on some asymptotic approximation. We could write a corresponding discrepancy measure for numericalTestXY() as ks.discrep <- function (data) { ks.test (data $ x,data $ y) $ statistic } And use this to estimate significance levels by simulation: numericalTestXY (x = mtcars $ qsec[automatic], y = mtcars $ qsec[manual], discrepancyFn = ks.discrep, generateXY = mixRandomly) ## [1] 0.2855
Different tests for H 0 : E ( X ) = E ( Y ) How about using? discrepancyMedians <- function (data){ abs ( median (data $ x) - median (data $ y))}
Different tests for H 0 : E ( X ) = E ( Y ) How about using? discrepancyMedians <- function (data){ abs ( median (data $ x) - median (data $ y))} Not the best discrepancy measure for testing differences in means !
Different tests for H 0 : E ( X ) = E ( Y ) How about using? discrepancyMedians <- function (data){ abs ( median (data $ x) - median (data $ y))} Not the best discrepancy measure for testing differences in means ! Or maybe the usual t-statistic?
Different tests for H 0 : E ( X ) = E ( Y ) How about using? discrepancyMedians <- function (data){ abs ( median (data $ x) - median (data $ y))} Not the best discrepancy measure for testing differences in means ! Or maybe the usual t-statistic? The discrepancy measure is d = | t | with x − y � t = σ 2 σ 2 m + y x n
Different tests for H 0 : E ( X ) = E ( Y ) How about using? discrepancyMedians <- function (data){ abs ( median (data $ x) - median (data $ y))} Not the best discrepancy measure for testing differences in means ! Or maybe the usual t-statistic? The discrepancy measure is d = | t | with x − y � t = σ 2 σ 2 m + y x n when σ x = σ y and both X and Y are normally distributed, we even know the distribution when H 0 holds! Even if σ x ! = σ y , it can be approximated.
Different tests for H 0 : E ( X ) = E ( Y ) How about using? discrepancyMedians <- function (data){ abs ( median (data $ x) - median (data $ y))} Not the best discrepancy measure for testing differences in means ! Or maybe the usual t-statistic? The discrepancy measure is d = | t | with x − y � t = σ 2 σ 2 m + y x n when σ x = σ y and both X and Y are normally distributed, we even know the distribution when H 0 holds! Even if σ x ! = σ y , it can be approximated. So we could just use the built-in t.test() function t.test (x, y) # for different variances, or t.test (x, y, var.equal = TRUE) # for equal variances.
Different tests for H 0 : E ( X ) = E ( Y ) How about using? discrepancyMedians <- function (data){ abs ( median (data $ x) - median (data $ y))} Not the best discrepancy measure for testing differences in means ! Or maybe the usual t-statistic? The discrepancy measure is d = | t | with x − y � t = σ 2 σ 2 m + y x n when σ x = σ y and both X and Y are normally distributed, we even know the distribution when H 0 holds! Even if σ x ! = σ y , it can be approximated. So we could just use the built-in t.test() function t.test (x, y) # for different variances, or t.test (x, y, var.equal = TRUE) # for equal variances. Alternatively, if assuming normality seems unjustified, perhaps we might still simulate using the t-statistic? t.stat <- function (data){ abs ( t.test (x = data $ x, y = data $ y) $ statistic)} Note that this statistic was constructed based on normal theory though.
Different tests for H 0 : E ( X ) = E ( Y ) We could try these out! Here on the qsec variable of mtcars " x <- mtcars $ qsec[automatic] ; y <- mtcars $ qsec[manual] In order of increasing assumptions about X and Y # Using the absolute difference in means numericalTestXY (x, y, discrepancyFn = discrepancyMeans) ## [1] 0.1505
Different tests for H 0 : E ( X ) = E ( Y ) We could try these out! Here on the qsec variable of mtcars " x <- mtcars $ qsec[automatic] ; y <- mtcars $ qsec[manual] In order of increasing assumptions about X and Y # Using the absolute difference in means numericalTestXY (x, y, discrepancyFn = discrepancyMeans) ## [1] 0.1505 # Using the two-sample t-statistic (unequal variances) numericalTestXY (x, y, discrepancyFn = t.stat) ## [1] 0.213
Different tests for H 0 : E ( X ) = E ( Y ) We could try these out! Here on the qsec variable of mtcars " x <- mtcars $ qsec[automatic] ; y <- mtcars $ qsec[manual] In order of increasing assumptions about X and Y # Using the absolute difference in means numericalTestXY (x, y, discrepancyFn = discrepancyMeans) ## [1] 0.1505 # Using the two-sample t-statistic (unequal variances) numericalTestXY (x, y, discrepancyFn = t.stat) ## [1] 0.213 # Using the normal theory two-sample t-statistic (unequal variances) t.test (x, y) $ p.value ## [1] 0.2093498
Different tests for H 0 : E ( X ) = E ( Y ) We could try these out! Here on the qsec variable of mtcars " x <- mtcars $ qsec[automatic] ; y <- mtcars $ qsec[manual] In order of increasing assumptions about X and Y # Using the absolute difference in means numericalTestXY (x, y, discrepancyFn = discrepancyMeans) ## [1] 0.1505 # Using the two-sample t-statistic (unequal variances) numericalTestXY (x, y, discrepancyFn = t.stat) ## [1] 0.213 # Using the normal theory two-sample t-statistic (unequal variances) t.test (x, y) $ p.value ## [1] 0.2093498 # Using the normal theory two-sample t-statistic (pooled variances) t.test (x, y, var.equal = TRUE) $ p.value ## [1] 0.2056621
Different tests for H 0 : E ( X ) = E ( Y ) We could try these out! Here on the qsec variable of mtcars " x <- mtcars $ qsec[automatic] ; y <- mtcars $ qsec[manual] In order of increasing assumptions about X and Y # Using the absolute difference in means numericalTestXY (x, y, discrepancyFn = discrepancyMeans) ## [1] 0.1505 # Using the two-sample t-statistic (unequal variances) numericalTestXY (x, y, discrepancyFn = t.stat) ## [1] 0.213 # Using the normal theory two-sample t-statistic (unequal variances) t.test (x, y) $ p.value ## [1] 0.2093498 # Using the normal theory two-sample t-statistic (pooled variances) t.test (x, y, var.equal = TRUE) $ p.value ## [1] 0.2056621 # And finally, using the absolute difference MEDIANS numericalTestXY (x, y, discrepancyFn = discrepancyMedians) ## [1] 0.4255
Different tests for H 0 : E ( X ) = E ( Y ) We could try these out! Here on the qsec variable of mtcars " x <- mtcars $ qsec[automatic] ; y <- mtcars $ qsec[manual] In order of increasing assumptions about X and Y # Using the absolute difference in means numericalTestXY (x, y, discrepancyFn = discrepancyMeans) ## [1] 0.1505 # Using the two-sample t-statistic (unequal variances) numericalTestXY (x, y, discrepancyFn = t.stat) ## [1] 0.213 # Using the normal theory two-sample t-statistic (unequal variances) t.test (x, y) $ p.value ## [1] 0.2093498 # Using the normal theory two-sample t-statistic (pooled variances) t.test (x, y, var.equal = TRUE) $ p.value ## [1] 0.2056621 # And finally, using the absolute difference MEDIANS numericalTestXY (x, y, discrepancyFn = discrepancyMedians) ## [1] 0.4255 Be careful . Recommendation here is discrepancyMeans (based on fewest assumptions).
On numerical tests When we constucted the function numericalTestXY() we were imagining the two sample problem. As with the lineup() test function, it might be better to write a more general numericalTest() function that would work more broadly.
On numerical tests When we constucted the function numericalTestXY() we were imagining the two sample problem. As with the lineup() test function, it might be better to write a more general numericalTest() function that would work more broadly. For example, numericalTest <- function (data, nReps = 2000, discrepancyFn, generateFn, returnStats = FALSE){ dObserved <- discrepancyFn (data) discrepancies <- sapply (1 : nReps, FUN = function (i){ newData <- generateFn (data) discrepancyFn (newData) } ) result <- mean (discrepancies >= dObserved) if (returnStats){ result <- list (p = result, dobs = dObserved, d = discrepancies)} result }
On numerical tests The advantage of writing the general numericalTest() function is that the code exactly mirrors the structure of a test of significance.
On numerical tests The advantage of writing the general numericalTest() function is that the code exactly mirrors the structure of a test of significance. Notes: ◮ the structure of numericalTest() makes no assumption about
On numerical tests The advantage of writing the general numericalTest() function is that the code exactly mirrors the structure of a test of significance. Notes: ◮ the structure of numericalTest() makes no assumption about ◮ the nature of the data,
On numerical tests The advantage of writing the general numericalTest() function is that the code exactly mirrors the structure of a test of significance. Notes: ◮ the structure of numericalTest() makes no assumption about ◮ the nature of the data, ◮ the hypothesis being tested,
Recommend
More recommend
