Statistical Power Analysis A Simple and General Model for Traditional and Modern Hypothesis Tests THIRD EDITION Statistical Power Analysis A Simple and General Model for Traditional and Modern Hypothesis Tests THIRD EDITION Kevin R Murphy Pennsylvania State University Brett Myors Griffith University Allen Wolach Illinois Institute of Technology Routledge Taylor & Francis Group 270 Madison Avenue New York, NY 10016 Routledge Taylor & Francis Group Park Square Milton Park, Abingdon Oxon OX14 4RN © 2009 by Taylor & Francis Group, LLC Routledge is an imprint of Taylor & Francis Group, an Informa business This edition published in the Taylor & Francis e-Library, 2010 To purchase your own copy of this or any of Taylor & Francis or Routledge’s collection of thousands of eBooks please go to www.eBookstore.tandf.co.uk International Standard Book Number-13: 978-1-84169-774-1 (Softcover) 978-1-84169-775-8 (Hardcover) Except as permitted under U.S Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe Library of Congress Cataloging-in-Publication Data Murphy, Kevin R., 1952Statistical power analysis : a simple and general model for traditional and modern hypothesis tests / Kevin R Murphy, Brett Myron, Allen Wolach 3rd ed p cm ISBN 978-1-84169-774-1 (pbk.) ISBN 978-0-415-96555-2 (hardback) Statistical hypothesis testing Statistical power analysis I Myron, Brett II Wolach, Allen H III Title QA277.M87 2008 519.5’6 dc22 Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the Psychology Press Web site at http://www.psypress.com ISBN 0-203-84309-6 Master e-book ISBN 2008037988 Contents Preface ix The Power of Statistical Tests The Structure of Statistical Tests The Mechanics of Power Analysis Statistical Power of Research in the Social and Behavioral Sciences 17 Using Power Analysis 19 Hypothesis Tests Versus Confidence Intervals 23 Summary 24 A Simple and General Model for Power Analysis The General Linear Model, the F Statistic, and Effect Size The F Distribution and Power Using the Noncentral F Distribution to Assess Power Translating Common Statistics and ES Measures Into F Defining Large, Medium, and Small Effects Nonparametric and Robust Statistics From F to Power Analysis Analytic and Tabular Methods of Power Analysis Using the One-Stop F Table The One-Stop F Calculator Summary Power Analyses for Minimum-Effect Tests 49 Implications of Believing That the Nil Hypothesis Is Almost Always Wrong 53 Minimum-Effect Tests as Alternatives to Traditional Null Hypothesis Tests 56 v 25 27 29 32 33 38 39 40 41 42 45 47 viâ•… â•… Contents Testing the Hypothesis That Treatment Effects Are Negligible Using the One-Stop Tables to Assess Power to Test Minimum-Effect Hypotheses Using the One-Stop F Calculator for Minimum-Effect Tests Summary 64 67 68 Using Power Analyses Estimating the Effect Size Four Applications of Statistical Power Analysis Calculating Power Determining Sample Sizes Determining the Sensitivity of Studies Determining Appropriate Decision Criteria Summary 71 72 77 78 79 81 82 87 Correlation and Regression 89 The Perils of Working With Large Samples 90 Multiple Regression 92 Power in Testing for Moderators 96 Why Are Most Moderator Effects Small? 97 Implications of Low Power in Tests for Moderators 99 Summary 100 t-Tests and the Analysis of Variance 101 The t-Test 101 Independent Groups t-Test 103 Traditional Versus Minimum-Effect Tests 105 One-Tailed Versus Two-Tailed Tests 107 Repeated Measures or Dependent t-Test 108 The Analysis of Variance 110 Which Means Differ? 113 Summary 116 Multifactor ANOVA Designs The Factorial Analysis of Variance Factorial ANOVA Example Fixed, Mixed, and Random Models Randomized Block ANOVA: An Introduction to Repeated-Measures Designs Independent Groups Versus Repeated Measures Complexities in Estimating Power in Repeated-Measures Designs Summary 59 117 118 124 126 128 129 134 135 Split-Plot Factorial and Multivariate Analyses 137 Split-Plot Factorial ANOVA 137 Contentsâ•… â•… viiâ•… Power for Within-Subject Versus Between-Subject Factors Split-Plot Designs With Multiple Repeated-Measures Factors The Multivariate Analysis of Variance Summary 140 141 141 144 The Implications of Power Analyses Tests of the Traditional Null Hypothesis Tests of Minimum-Effect Hypotheses Power Analysis: Benefits, Costs, and Implications for Hypothesis Testing Direct Benefits of Power Analysis Indirect Benefits of Power Analysis Costs Associated With Power Analysis Implications of Power Analysis: Can Power Be Too High? Summary 145 146 147 151 151 153 154 155 157 References 159 Appendices 163 Author Index 209 Subject Index 211 Preface One of the most common statistical procedures in the behavioral and social sciences is to test the hypothesis that treatments or interventions have no effect, or that the correlation between two variables is equal to zero, etc.— i.e., tests of the null hypothesis Researchers have long been concerned with the possibility that they will reject the null hypothesis when it is in fact correct (i.e., make a Type I error), and an extensive body of research and data-analytic methods exists to help understand and control these errors Less attention has been devoted to the possibility that researchers will fail to reject the null hypothesis, when in fact treatments, interventions, etc., have some real effect (i.e., make a Type II error) Statistical tests that fail to detect the real effects of treatments or interventions might substantially impede the progress of scientific research The statistical power of a test is the probability that it will lead you to reject the null hypothesis when that hypothesis is in fact wrong Because most statistical tests are conducted in contexts where treatments have at least some effect (although it might be minuscule), power often translates into the probability that your test will you lead to a correct conclusion about the null hypothesis Viewed in this light, it is obvious why researchers have become interested in the topic of statistical power and in methods of assessing and increasing the power of their tests This book presents a simple and general model for statistical power analysis that is based on the widely used F statistic A wide variety of statistics used in the social and behavioral sciences can be thought of as special applications of the “general linear model” (e.g., t-tests, analysis of variance and covariance, correlation, multiple regression), and the F statistic can be used in testing hypotheses about virtually any of these specialized applications The model for power analysis laid out here is quite simple, and it illustrates how these ix .. .Statistical Power Analysis A Simple and General Model for Traditional and Modern Hypothesis Tests THIRD EDITION Statistical Power Analysis A Simple and General Model for Traditional and Modern. .. for multifactor analysis of variance (ANOVA), including split-plot and randomized block factorial designs Although conceptual issues for power analysis are similar in factorial ANOVA and other... significance tests, power analyses, and assessments of N and alpha needed for traditional and minimum-effects tests • New chapters (Chapters and 8) demonstrating the application of POWER in complex ANOVA