# R code for computing the ad agreement measure and its 95% critical value

The code below can be pasted at a R prompt in R.

Firstly read in the fields and foreign libraries.

Install.packages(c(“fields”,"foreign")) library(fields) library(foreign)

Next read in the raw data which can be read in from a SPSS spreadsheet with items as rows and raters as columns. An example input for 3 raters (r1, r2 and r3) rating each of 5 items (upto a maximum rating of 7) taken from section 5.2 of Kreuzpointner et al. (2010) is given here.

score <- read.spss("U:\\R_Work\\items.sav")

Just paste this function text below into R.

adval <- function(b, perc) { score <- data.frame(score) score <- as.matrix(score) score <- t(score) nrate <- nrow(score) nitem <- ncol(score) a <- 1 rate <- matrix(score,nrow=nrate,ncol=nitem) adboot <- matrix(0,1000,1) out <- 0 for (i in 1:nitem) { out <- out + sum(rdist(rate[,i])^2)/2 } ifelse (nrate-floor(nrate/2)*2 > 0, dmax <- nitem * (b-a)*(b-a)*0.25*(nrate*nrate-1), dmax <- nitem * (b-a)*(b-a)*0.25*(nrate*nrate)) ad <- 1 - (out/dmax) # bootstrap to obtain 95 percentile for the null distribution of ad based on the # binomial distribution using 10000 samples as sugegsted by Kreuzpointner et al. p <- (mean(rate)-1)/(b-1) n <- b-1 outboo <- 0 for (ict in 1:10000) { outboo <- 0 rb <- rbinom(nitem*nrate,n,p)+1 rboo <- matrix(rb,nrow=nrate,ncol=nitem) for (i in 1:nitem) { outboo <- outboo + sum(rdist(rboo[,i])^2)/2 } adboot[ict] <- 1 - (outboo/dmax) } cat("ad = ", ad, "\n") cat(100*perc,"percentile for ad = ",quantile(adboot,probs=perc), "\n") }

Then you are ready to run the function above using the maximum number of ratings and 1- the significance level as inputs.

adval(7,0.95)

which should output something like the below

ad = 0.9722222 95 percentile for ad = 0.9777778

So there is no evidence that there is an agreement between the raters. Note: we get a slightly different result to Kruezpointner et al. because we use a binomial probability of 0.15 instead of 0.2, as in their paper, to estimate the critical 5% threshold for statistical significance.

Reference

Kreuzpointner L, Simon P and Theis FJ (2010) The ad coefficient as a descriptive measure of the within-group agreement of ratings. *British Journal of Mathematical and Statistical Psychology* **63** 341-360.