Adjusted p-values in SPSS and R

Howell DC (1992, 1997, 2002) describes adjustments to uncorrected p-values based on the total number of pairwise group comparisons being performed. The macro in the box below implements some of these. This is an adaptation of a macro (rmpost.sps) first written by David Nichols of SPSS. In particular Holm-Bonferroni method is recommended for multiple testing of several correlations from the same matrix by Larzelere and Mulaik (1977) and Howell (2002;pages 388-390). Other work (see [attachment:hb.pdf here]) suggests the Holm-Bonferroni may be used for correlation matrices less than 15 by 15 in size. There are, however, problems [:FAQ/SpssBonferroni:using Bonferroni methods] and so a Holm-Sidak approach is available outputted as downsidak in the macro below. Both Holm-Bonferroni and Holm-Sidak methods, and the [:FAQ/FDR: FDR method], may also be performed [attachment:hierps.xls using a spreadsheet] or [:FAQ/Rpvs: with R.]

Reference

Howell (2002) Statistical methods for psychology. Fifth Edition. Wadsworth:Pacific Grove:CA

Larzelere RE and Mulaik SA (1977) Single-sample tests for many correlations. Psychological Bulletin 84 557-569.

[COPY AND PASTE THE BOX SYNTAX INTO A SPSS SYNTAX WINDOW; SELECT ALL AND RUN. EDIT THE INPUT DATA AS REQUIRED]

* enter a column of pvalues and this macro will
* adjust for the number
* in the column. The Ryan and Einot and Gabriel
* methods are for pairwise
* comparisons of group locations (e.g. means, 
* mean ranks) with a step size of abs(j - i)+ 1 
* where the higher of the two means has an 
* overall rank of j and the lower overall 
* rank, i.

* SPSS uses REGWQ to compute this for pairwise
* comparison of group means in univariate 
* for between subs factors
* Could be applied to p-values from ANY
* procedure e.g. nonparametrics as just uses 
* p-value and number of comparisons

* Create a dataset with all uncorrected 
* p-values and 
* step = abs(difference in ranks of group 
* locations) + 1.
* adjust data input below as required.
* If interested ONLY in Holm and Sidak methods 
* put step = 1 for all inputted p-values.

* The program creates a file called temp.sps
* in My Documents folder which may be deleted
* after running the macro

* -99 in the output for Holm and Sidak 
* procedures indicates the pairwise comparison
* is not tested and deemed nonsignificant
* because the previous comparison was 
* nonsignificant (p=0.05, by default)
* this may be changed by changing last line
* in this box

DATA LIST list / PVAL(f9.3) STEP (f2.0).
BEGIN DATA
0.266 2
0.139 3
0.016 2
END DATA.

set errors=none.
set mprint=off.

DEFINE PV(PVALUE=!TOKENS(1) /STEP=!TOKENS(1) /ALP=!TOKENS(1)).
SORT CASES BY  !PVALUE (A) .
COMPUTE pos=$CASENUM.

* Calculate the number of p values.
RANK !PVALUE /n into N.
* N contains the number of cases in the file.
* make a submacro to be invoked from the syntax.

DO IF $CASENUM=1.
WRITE OUTFILE 'C:\Documents and Settings\user\My Documents\temp.sps' /"DEFINE !nbcases()"n"!ENDDEFINE.".
END IF.
EXE.

INCLUDE FILE='C:\Documents and Settings\user\My Documents\temp.sps'.
/* The number of cases in the file is now accessible using !nbcases */.

COMPUTE bonferr=!PVALUE*!nbcases.
IF (bonferr>1) bonferr=1.
COMPUTE sidak=1-(1-!PVALUE)**!nbcases.
COMPUTE holm=(!nbcases-pos+1)*!PVALUE.
IF (LAG(holm,1)>!ALP | LAG(holm,1)=-99) holm=-99.
COMPUTE downsidk=1-(1-!PVALUE)**(!nbcases-pos+1).
IF (LAG(downsidk,1)>!ALP | LAG(downsidk,1)=-99) downsidk=-99.
COMPUTE ryan=!PVALUE*!nbcases/!STEP.
IF (ryan>1) ryan=1.
COMPUTE eingab=1-(1-!PVALUE)**(!nbcases/!STEP).
IF (eingab>1) eingab=1.
FORMAT bonferr to downsidk ryan eingab (f7.3).
VARIABLE LABELS PVALUE 'Original' /bonferr '1-step Bonferroni'
 /sidak '1-step Sidak' /holm 'Step-down Holm`s'/ downsidk 'Step-down Sidak' 
/ryan 'Ryan'  /eingab 'Einot & Gabriel' /STEP 'Step'.
EXECUTE.

REPORT FORMAT=LIST AUTOMATIC ALIGN(CENTER)
  /VARIABLES=!PVALUE bonferr sidak holm downsidk !STEP ryan eingab 
  /TITLE "Original and adjusted p-values".

!ENDDEFINE.
* changing the value of alp to re-specify
* significance level
PV PVALUE=PVAL STEP=STEP ALP=0.05.