Performing randomisation tests using nonparametric methods
Randomisation tests (see for example Edgington (2007)) are tests where a particular test statistic, like a sum or a correlation, for observed data is compared to all possible values of this statistic that is calculated from all possible permutations of the data. This generates a one or two-tailed probability (p-value) which is the probability of observing an effect at least as extreme as that observed in the data. This may be approximated by taking a large enough number of samples (eg 10000) from the data (also available using the Monte-Carlo option in certain SPSS nonparametric procedures).
This test has the advantage of not assuming a particular distribution in the data and can be performed with relatively small amounts of data.
Some of these tests can be performed using the exact option in nonparametric tests in SPSS.
An example of its use is given below which may be obtained using the exact option for 2 independent (unrelated) samples in SPSS.
Score |
Group |
|
1 |
1 |
|
2 |
1 |
|
3 |
2 |
|
4 |
2 |
|
5 |
2 |
|
6 |
2 |
|
7 |
2 |
|
We are interested in seeing if the two groups differ on their ranked scores. To do this the Mann-Whitney test uses the sum of the observation ranks in one of the groups and compares this sum to all possible sums that could result from two groups of sizes 2 and 5.
In the above example we have the most extreme scenario where the two smallest observed scores are both in the two observation group. There are 21 possible ways of obtaining two ranks from 7 observations (assuming no ties) and none of these have rank sum less than that observed (1+2=3). The 21 possible pairs are given below.
Group Ranks (N=2) |
Rank sum |
|
1,2 |
3 |
|
1,3 |
4 |
|
1,4 |
5 |
|
1,5 |
6 |
|
1,6 |
7 |
|
1,7 |
8 |
|
2,3 |
5 |
|
2,4 |
6 |
|
2,5 |
7 |
|
2,6 |
8 |
|
2,7 |
9 |
|
3,4 |
5 |
|
3,5 |
8 |
|
3,6 |
9 |
|
3,7 |
10 |
|
4,5 |
9 |
|
4,6 |
10 |
|
4,7 |
11 |
|
5,6 |
11 |
|
5,7 |
12 |
|
6,7 |
13 |
|
So the (one-tailed) p-value or probability of observing at least as low a rank sum as that observed given two groups of sizes 2 and 5 is 1/21=0.048. Try inputting this data into SPSS, choosing exact under analyze:nonparametric tests:exact and you should get this result. Just double to get a two-tailed p-value if you are not sure of the direction of difference (=0.096).
In R defining (unpaired) groups, x and y:
x <- c(1,2) y <- c(3,4,5,6,7)
then running
wilcox.test(x,y,exact=TRUE,alternative="less")
gives the same result.
Wilcoxon rank sum test data: x and y W = 0, p-value = 0.04762 alternative hypothesis: true mu is less than 0
Reference
Edgington ES, Onghena P (2007) Randomisation tests:fourth edition. CRC Press:London.