A note about using ranked outcomes in t-tests and ANOVAs

This note aims to show the state-of-play on the use of ranked responses in analysis of variance particularly with regard to testing for interactions.

Conover and Iman (1981) showed that the Mann-Whitney test is equivalent to a t-test on ranked responses. Huitema (1980) in Chapter 12 and Shirley (1981) illustrate the use and interpretation of an analysis of covariance on ranked outcomes with worked examples. Shirley (1987) extends Conover and Iman's work to investigating interactions by presenting a worked example involving a full factorial between subjects analysis of variance. Blair, Sawilowsky and Higgins (1987) and Wilcox (2003, p.604) do not advocate using ranked outcomes to test for interactions 'in more complex designs'. Thompson (1991) takes a conservative stance, stating that ANOVAs on ranked data can be used to test for interactions only "when there are exactly two levels of both main effects or when there is only one main effect" (p. 697). Akritas (1991) shows that rank transformed responses are not applicable to repeated measures ANOVA.

Leys and Schumann (2010) present a step-by-step worked example which tests for an interaction in a pair of between subjects factor ANOVA (see here) using an adjusted rank transform with a working input spreadsheet. This spreadsheet will work out a nonparametric two-way interaction for 2 between subjects factors (BB) each having two levels. The response and group raw data and number of cases are entered into Sheet1 and the results displayed in Sheet2. The calculations follow the method illustrated by the simple example in Leys and Schumann (2010). This approach involves testing the two-way interaction tests on ranked data when the main effects have been removed and is known as the adjusted rank transformation test (ART). A .exe file for use on windows to perform more general ART tests for nonparametric interactions for factorial ANOVAs is downloadable from here.

Mansouri (1998) also presents some examples using SAS for looking at interactions between two between subjects factors (see here). Mansouri and Chang (1995) note down problems using rank tranforms when responses follow a Cauchy distribution and advocate using Normal scores to downweight the inflated Type I error.

More recently Zimmerman (2011) has shown that using t-tests on ranked outcomes, even when there are a large proportion of equal scores (having tied ranks), have greater power and smaller type I error than the conventional t-test on raw scores when the outcome is not normally distributed. He also claims using ranked outcomes works well with one-way ANOVAs.

An alternative strategy ignoring ranks altogether is to fit a generalized linear mixed model (GLMM) assuming responses are counts and follow a poisson distribution. In this way interactions for repeated measures data may be fitted e.g. to 20 people in one of four diagnostic groups each being assessed in three positions (lying, sitting and standing). One inputs data in 'long' format consisting of a single response with columns indicating subject, response, group and position. The GLMM can then be fitted in R reading a SPSS data file and using the syntax below.

library(foreign)
x <- read.spss("U://My Documents//WHIMINT.sav")
x1 <- attach(x)
library(foreign)
library(lme4)
int <-glmer( resp ~ position + grp + position*grp + (1|sub), data = x, family = poisson)

Edgington (1995) suggests using randomization (bootstrap) tests when sample sizes are small to obtain more robust p-values from parametric procedures such as ANOVAs using model fit statistics. This is succinctly done in R for the repeated measures using GLMMs. The syntax below uses a logged response to handle any skew in the response with the difference in the Akaike Information Criterion (AIC) comparing model fits of the ANOVA (fitted as a mixed model) with and without the interaction term as the fit statistic of interest.

library(foreign)
x <- read.spss("U://My Documents//WHIMINT.sav")
x1 <- attach(x)
library(lme4)
int <-lmer( log(resp) ~ as.factor(position) + as.factor(poolgrp) + as.factor(position)*as.factor(poolgrp) + (1|sub), data = x)
int2 <-lmer( log(resp) ~ as.factor(position) + as.factor(poolgrp) + (1|sub), data = x)
diff <- AIC(int) - AIC(int2)

nb<-1000
n <- length(resp)
boot<-matrix(NA,nb,6)
attributes(boot)
for (i in 1:nb)
{
bs<-sample(resp,n,replace =F)
int <-lmer( log(bs) ~ as.factor(position) + as.factor(poolgrp) + as.factor(position)*as.factor(poolgrp) + (1|sub), data = x)
boot[i,1] <- AIC(int)
int2 <-lmer( log(bs) ~ as.factor(position) + as.factor(poolgrp) + (1|sub), data = x)
boot[i,2] <- AIC(int2)
boot[i,3] <- AIC(int2) - AIC(int)
}

sum(boot[,3]<diff)/nb # GIVES randomization test p-value

One could alternatively use differences in AICs from fitting a Poisson GLMM although this takes much longer to run than using the above lmer() procedure which assumes a continuous response.

library(foreign)
x <- read.spss("U://My Documents//WHIMINT.sav") #move the interaction SPSS file to the My Documents folder
x1 <- attach(x)
library(lme4)
int <-glmer( resp ~ as.factor(position) + as.factor(poolgrp) + as.factor(position)*as.factor(poolgrp) + (1|sub), data = x, family = poisson)
int2 <-glmer( resp ~ as.factor(position) + as.factor(poolgrp) + (1|sub), data = x, family = poisson)
diff <- AIC(int) - AIC(int2)

nb<-1000
n <- length(resp)
boot<-matrix(NA,nb,6)
attributes(boot)
for (i in 1:nb)
{
bs<-sample(resp,n,replace =F)
int <-glmer( bs ~ as.factor(position) + as.factor(poolgrp) + as.factor(position)*as.factor(poolgrp) + (1|sub), data = x, family = poisson)
boot[i,1] <- AIC(int)
int2 <-glmer( bs ~ as.factor(position) + as.factor(poolgrp) + (1|sub), data = x, family = poisson)
boot[i,2] <- AIC(int2)
boot[i,3] <- AIC(int2) - AIC(int)
}

sum(boot[,3]<diff)/nb

References

Akritas, MG (1991) Limitations of the rank transform procedure: a study of repeated measures designs, part 1. Journal of the Americal Statistical Association 86(414) 457-460.

Blair, RC, Sawilowsky, SS and Higgins, JJ (1987). Limitations of the rank transform statistic in tests for interactions. Communications in Statistics - Simulation and Computation 16 1133-1145.

Conover, WJ and Iman, RL (1981). Rank transformations as a bridge between parametric and nonparametric statistics. American Statistician 35 124-129. doi:10.2307/2683975

Edgington, ES (1995). Randomization tests. Third Edition. Marcel Dekker Inc:New York.

Huitema, BE (1980). The analysis of covariance and alternatives. 2nd Edition. Wiley: New York.

Leys, C. and Schumann, S (2010). A nonparametric method to analysze interactions: the adjusted rank transform test. Journal of Experimental Psychology 46 684-688.

Mansouri, H. (1998). Multifactor analysis of variance based on the aligned rank transform technique. Computational Statistics & Data Analysis 29(2) 177-189.

Mansouri, H. and Chang, G-H. (1995). A comparative study of some rank tests for interaction. Computational Statistics & Data Analysis 19(1), 85-96.

Quade, D (1967). Rank analysis of covariance. Journal of the American Statistical Association 62 1187-1200.

Shirley, EAC (1981). A distribution-free method for analysis of covariance based on ranked data. Applied Statistics 30 158-162.

Shirley, EAC (1987). Applications of ranking methods to multiple comparison procedures and factorial experiments. Applied Statistics 36(2) 205-213.

Thompson, GL (1991). A note on the rank transform for interactions. Biometrika 78 697–701. This is quite a mathematical paper but it suggests under certain conditions the sums of squares for a two-way interaction from a ANOVA on a ranked response can be compared to a chi-square distribution.

Wilcox, RR (2003, p.604). Applying contemporary statistical techniques. Academic Press:New York.

Zimmerman, DW (2011). Power comparisons of significance tests of location using scores, ranks, and modular ranks. British Journal of Mathematical and Statistical Psychology 64 233-243.