FAQ/npexact - CBU statistics Wiki

Revision 4 as of 2007-11-06 12:44:42

Clear message
location: FAQ / npexact

Performing randomisation tests using nonparametric methods

Randomisation tests (see for example Edgington (2007)) are tests where a particular test statistic, like a sum or a correlation, for observed data is compared to all possible values of this statistics that can result from all possible permutations of the data. This generates a one or two-tailed probability (p-value) which is the probability of observing an effect at least as extreme as that observed in the data. This may be approximated by taking a large enough number of samples (eg 10000) from the data (also available using the Monte-Carlo option in certain SPSS nonparametric procedures).

This test has the advantage of not assuming a particular distribution in the data and can be performed with relatively small amounts of data.

Some of these tests can be performed using the exact option in nonparametric tests in SPSS.

An example of its use is given below which may be obtained using the Mann-Whitney exact option for 2 unrelated samples in SPSS.

Score

Group

1

1

2

1

3

2

4

2

5

2

6

2

7

2

We are interested in seeing if the two groups differ on their ranked scores. To do this the Mann-Whitnesy test uses the sum of the observation ranks in each group and compares the sum in the smaller group (1) with all possible sums that could result from two groups of sizes 2 and 5.

In the above example we have the most extreme case where the two smallest observed scores are both in the first group. There are 7x6/2 = 21 possible pairs from 7 observations and none of these have rank sum less than that observed (1+2=3). So the (one-tailed) p-value or probability of observing at least as low a rank sum as that observed given two groups of sizes 2 and 5 is 1/21=0.048.

Reference

Edgington ES, Onghena P (2007) Randomisation tests:fourth edition. CRC Press:LOndon.