Intraclass correlations

An alternative, to the kappa statistic, which uses an analysis of variance output to estimate rater reliability is the intraclass correlation coefficient (ICC).

For a repeated measures anova involving k raters it follows assuming both subjects and raters are fixed effects that

$$\mbox{MS(total w)} = \frac{\mbox{SS(raters)–MS(subjects x raters)}}{\mbox{df(raters) + df(subjects x raters)}}$$

$$\mbox{ICC1} = \frac{\mbox{MS(subjects)–MS(total w)}}{\mbox{MS(subjects) + (k-1)MS(total w)}}$$

where MS is the Mean square from the repeated measures analysis of variance.

This corresponds to the (single measures) one-way random approach in SPSS (see [http://www.uvm.edu/~dhowell/StatPages/More_Stuff/icc/icc.html here).]

It follows that the intra-class correlation (ICC), unlike the Pearson correlation, is useful for pooling paired data each having three or more observations. Einfield and Tonge (1992, p 12) prefer using the ICC to the Pearson as it is more conservative owing to that fact it "takes account of the absolute as well as the relative difference between the scores of two raters".

Howell (1997) also recommends an alternative, more widely used ICC which assumes that the raters are a random sample from a larger population which is of form below:

$$\mbox{ICC2} = \frac{\mbox{MS(subjects)–MS(subjects x raters)}}{\mbox{MS(subjects) + (k-1)MS(subjects x raters) + k[MS(raters) - MS(subjects x raters)]/n}}$$

where n is the number of subjects being rated. ICC2 agrees with the (single measures) two-way random (and two-way mixed) ICCs with a type of absolute agreement.

ICC1 and ICC2 give similar, correctly low, results when testing absolute inter-rater agreement. For example if two raters rate three subjects giving ratings 1,2; 2,4 and 3,6 respectively then ICC1 = 0.44 and ICC2 = 0.52.

ICC may be computed in SPSS using analyze>scale>reliability analysis>statistics and choosing one of the two ICCs which allow a type of absolute agreement and looking at single measures. The consistency option is to be avoided as it does not compare the differences in ratings between raters.

Examples of ICC computation in SPSS [http://www.ats.ucla.edu/stat/spss/library/whichicc.htm are available here] and [attachment:ICC.doc here.] ICC1 above corresponds to sfsingle (ICC(1,1)) and ICC2 to sfrandom (ICC(2,1)) in Shrout and Fleiss (1979).

$$\frac{\mbox{true inter-rater variance}}{\mbox{true inter-rater variance + common error in rating variance}}$$

as mentioned as a reliability correlation in the two rater case, for example, in [http://www-users.york.ac.uk/%7Emb55/talks/oxtalk.htm a paper by Martin Bland and Doug Altman.]

An overview of approaches to inter rater reliability including the ICC is given by Darroch and McCloud (1986).

ICCs below 0.40 are regarded as poor, 0.40 to 0.59 as fair and above 0.60 as good (Cicchetti and Sparrow, 1981). Kramer and Feinstein (1981) also give rules of thumb for sizes of ICCs.

• [:FAQ/iccpr: Inferiority of using a Pearson correlation to assess absolute agreement amongst raters as opposed to an ICC]

References

Cicchetti, DV and Sparrow SA (1981) Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior. Am J Ment Defic 86 127-37.

Darroch JN, McCloud PI (1986) Category distinguishability and observer agreement Australian Journal of Statistics 28 371-88.

Howell DC (1997) Statistical methods for psychologists. Fourth edition. Wadsworth:Belmont,CA. (pages 490-493).

Einfield, SL and Tonge, BJ (1992) Manual for the developmental hebaviour checklist (DBC)(Primary Carer version). Melbourne:School of Psychiatry, University of new South Wales, and Centre for Developmental Psychiatry, Monash University, Clayton, Victoria.

Kramer, MS and Feinstein, AR (1981) The biostatistics of concordance. Clinical Phamacology and Therapeutics 29 111-123.

Shrout, PE and Fleiss, JL (1979). Intraclass Correlations: Uses in Assessing Rater Reliability, Psychological Bulletin, 86 (2) 420-428. (A good primer showing how anova output can be used to compute ICCs).