Multiple Comparison Procedures—Cutting the Gordian Knot

Published in Agron. J. 107:730–735 (2015) doi:10.2134/agronj2012.0394 Available freely online through the author-supported open access option. Copyright © 2015 by the American Society of Agronomy, 5585 Guilford Road, Madison, WI 53711. All rights reserved. No part of this periodical may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher. Multiple comparison procedures are commonly used to test for “signifi cant” diff erences between treatment means in experiments, even in cases where the set of treatments has clear structure and has been derived with obvious questions in mind. In these cases, the use of an MCP is inappropriate, as has been pointed out a multitude of times (e.g., Swallow, 1984; Little, 1978). To quote Swallow (1984), MCPs “were developed for cases where the treatment set lacked structure, that is, where the treatments were just a collection of varieties or perhaps chemicals with no particular inter-relationships. Most treatment designs are not of this type. Usually, the treatment set has a structure, and the statistical analysis should recognize that structure.” Th is can be achieved by specifying appropriate contrasts between the treatments, with each contrast addressing a particular question of interest to the researcher; in many cases, these contrasts can be chosen to be “orthogonal” (mutually independent) but this is not essential. Journals oft en encourage researchers to thus tailor their statistical analysis to the objectives of the research, but the specifi cation of appropriate contrasts is not a skill easily acquired by researchers and help from a biometrician is not always available, so this encouragement is oft en to no avail. As an example of such advice, in the instructions to authors of Agronomy Journal (September, 2012), the statistical methods section is largely devoted to warning of the limitations of MCPs, with the closing statement, “[w]hen treatments have a logical structure, orthogonal contrasts among treatments should be used.” Th us, the long-running debate on the relative merits of the many diff erent MCPs is relevant only to the minority of studies in which such a procedure is appropriate. I fi rst introduce some necessary statistical terminology, then discuss the general topic of MCPs in relation to various types of error rate and in relation to the levels of conservatism of some of the better known MCPs. Th e idea of inconsistency is then introduced and discussed, with particular attention being paid to Fisher’s restricted least signifi cant diff erence (LSD) procedure, the MCP most commonly used in Agronomy Journal. Next, the implication of the choice of MCP for required sample size (estimated using a power analysis) is discussed. Finally, a practical solution to the problem of best choice of MCP is described. Th is consists of using the simplest of procedures, the unrestricted LSD procedure, with the proviso that it be regarded as an hypothesis formulation tool, with any interesting pairwise hypotheses thus formulated requiring testing in a second, independent experiment.