R code for computing the ad agreement measure and its 95% critical value

The code below can be pasted at a R prompt in R.

Firstly read in the fields and foreign libraries.

Install.packages(c(“fields”,"foreign"))
library(fields)
library(foreign)

Next read in the raw data which can be read in from a SPSS spreadsheet with items as rows and raters as columns. An example input for 3 raters (r1, r2 and r3) rating each of 5 items (upto a maximum rating of 7) taken from section 5.2 of Kreuzpointner et al. (2010) is given here.

score <- read.spss("U:\\R_Work\\items.sav")

Just paste this function text below into R.

adval <- function(b, perc) {
score <- data.frame(score)
score <- as.matrix(score)
score <- t(score)
nrate <- nrow(score)
nitem <- ncol(score)
a <- 1

rate <- matrix(score,nrow=nrate,ncol=nitem)
adboot <- matrix(0,1000,1)
out <- 0

for (i in 1:nitem) {
out <- out + sum(rdist(rate[,i])^2)/2 
}

ifelse (nrate-floor(nrate/2)*2 > 0, dmax <- nitem * (b-a)*(b-a)*0.25*(nrate*nrate-1), dmax <- nitem * (b-a)*(b-a)*0.25*(nrate*nrate))

ad <- 1 - (out/dmax)

# bootstrap to obtain 95 percentile for the null distribution of ad based on the 
# binomial distribution using 10000 samples as sugegsted by Kreuzpointner et al.

p <- (mean(rate)-1)/(b-1)
n <- b-1
outboo <- 0

for (ict in 1:10000) {
outboo <- 0
rb <- rbinom(nitem*nrate,n,p)+1
rboo <- matrix(rb,nrow=nrate,ncol=nitem)

for (i in 1:nitem) {
outboo <- outboo + sum(rdist(rboo[,i])^2)/2 
}

adboot[ict] <- 1 - (outboo/dmax)

 }

cat("ad = ", ad, "\n") 
cat(100*perc,"percentile for ad = ",quantile(adboot,probs=perc), "\n")
}

Then you are ready to run the function above using the maximum number of ratings and 1- the significance level as inputs.

adval(7,0.95)

which should output something like the below

ad =  0.9722222 
95 percentile for ad =  0.9777778 

So there is no evidence that there is an agreement between the raters. Note: we get a slightly different result to Kruezpointner et al. because we use a binomial probability of 0.15 instead of 0.2, as in their paper, to estimate the critical 5% threshold for statistical significance.

Reference

Kreuzpointner L, Simon P and Theis FJ (2010) The ad coefficient as a descriptive measure of the within-group agreement of ratings. British Journal of Mathematical and Statistical Psychology 63 341-360.