Adjusted p-values in SPSS and R
Howell DC (1992, 1997, 2002) describes adjustments to uncorrected p-values based on the total number of pairwise group comparisons being performed. The macro in the box below implements some of these. This is an adaptation of a macro (rmpost.sps) first written by David Nichols of SPSS for controlling type I error when performing multiple t-tests between groups in repeated measures. In particular Holm-Bonferroni method is recommended for multiple testing of several correlations from the same matrix by Larzelere and Mulaik (1977) and Howell (2002;pages 388-390). Other work (see [attachment:hb.pdf here]) suggests the Holm-Bonferroni may be used for correlation matrices less than 15 by 15 in size. There are, however, problems [:FAQ/SpssBonferroni:using Bonferroni methods] and so a Holm-Sidak approach is available outputted as downsidak in the macro below. Both Holm-Bonferroni (recommended [http://www.uky.edu/ComputingCenter/SSTARS/www/documentation/MultipleComparisons_3.htm#b7 here]) and Holm-Sidak stepdown methods, and a method gaining popularity in imaging studies, the [:FAQ/FDR: FDR method], may also be performed [attachment:hierps.xls using a spreadsheet] or [:FAQ/Rpvs: with R.]
Lix and Sajobi (2010) say that the above FDR approach (Benjamini and Hochberg, 1995) is more powerful than both Bonferroni and Hochberg (1988) with the latter having good power in comparing post-hoc repeated measures contrasts.
References
Benjamini Y & Hochberg Y (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society 57 289-300.
Howell (2002) Statistical methods for psychology. Fifth Edition. Wadsworth:Pacific Grove:CA
Larzelere RE and Mulaik SA (1977) Single-sample tests for many correlations. Psychological Bulletin 84 557-569.
Lix LM and Sajobi T (2010) Testing multiple outcomes in repeated measures designs. Psychological Methods 15(3) 268-280.
[COPY AND PASTE THE BOX SYNTAX INTO A SPSS SYNTAX WINDOW; SELECT ALL AND RUN. EDIT THE INPUT DATA AS REQUIRED]
* enter a column of pvalues and this macro will * adjust for the number * in the column. The Ryan and Einot and Gabriel * methods are for pairwise * comparisons of group locations (e.g. means, * mean ranks) with a step size of abs(j - i)+ 1 * where the higher of the two means has an * overall rank of j and the lower overall * rank, i. * SPSS uses REGWQ to compute this for pairwise * comparison of group means in univariate * for between subs factors * Could be applied to p-values from ANY * procedure e.g. nonparametrics as just uses * p-value and number of comparisons * Create a dataset with all uncorrected * p-values and * step = abs(difference in ranks of group * locations) + 1. * adjust data input below as required. * If interested ONLY in Holm and Sidak methods * put step = 1 for all inputted p-values. * The program creates a file called temp.sps * in My Documents folder which may be deleted * after running the macro * -99 in the output for Holm and Sidak * procedures indicates the pairwise comparison * is not tested and deemed nonsignificant * because the previous comparison was * nonsignificant (p=0.05, by default) * this may be changed by changing last line * in this box DATA LIST list / PVAL(f9.3) STEP (f2.0). BEGIN DATA 0.266 2 0.139 3 0.016 2 END DATA. set errors=none. set mprint=off. DEFINE PV(PVALUE=!TOKENS(1) /STEP=!TOKENS(1) /ALP=!TOKENS(1)). SORT CASES BY !PVALUE (A) . COMPUTE pos=$CASENUM. * Calculate the number of p values. RANK !PVALUE /n into N. * N contains the number of cases in the file. * make a submacro to be invoked from the syntax. DO IF $CASENUM=1. WRITE OUTFILE 'C:\Documents and Settings\peterw\My Documents\temp.sps' /"DEFINE !nbcases()"n"!ENDDEFINE.". END IF. EXE. INCLUDE FILE='C:\Documents and Settings\peterw\My Documents\temp.sps'. /* The number of cases in the file is now accessible using !nbcases */. COMPUTE bonferr=!PVALUE*!nbcases. IF (bonferr>1) bonferr=1. COMPUTE sidak=1-(1-!PVALUE)**!nbcases. COMPUTE holm=(!nbcases-pos+1)*!PVALUE. IF (LAG(holm,1)>!ALP | LAG(holm,1)=-99) holm=-99. COMPUTE downsidk=1-(1-!PVALUE)**(!nbcases-pos+1). IF (LAG(downsidk,1)>!ALP | LAG(downsidk,1)=-99) downsidk=-99. COMPUTE ryan=!PVALUE*!nbcases/!STEP. IF (ryan>1) ryan=1. COMPUTE eingab=1-(1-!PVALUE)**(!nbcases/!STEP). IF (eingab>1) eingab=1. FORMAT bonferr to downsidk ryan eingab (f7.3). VARIABLE LABELS !PVALUE 'Original' /bonferr '1-step Bonferroni' /sidak '1-step Sidak' /holm 'Step-down Holm`s'/ downsidk 'Step-down Sidak' /ryan 'Ryan' /eingab 'Einot & Gabriel' /STEP 'Step'. EXECUTE. REPORT FORMAT=LIST AUTOMATIC ALIGN(CENTER) /VARIABLES=!PVALUE bonferr sidak holm downsidk !STEP ryan eingab /TITLE "Original and adjusted p-values". !ENDDEFINE. * changing the value of alp to re-specify * significance level PV PVALUE=PVAL STEP=STEP ALP=0.05.