Position
Talk Title
Beyond Power: Assessing Research Designs in Large Samples
Abstract
One of the biggest threats to the validity of statistical conclusions in null hypothesis significance testing (NHST) is low sample sizes, leading to low power and thus unreliable findings. Indeed, many examples of unreliable findings highlighted by the "reproducibility crisis" are arguably examples of underpowered studies. However, there are many other threats to validity even when large samples are present and it seems that the research is adequately powered. We will discuss these pitfalls in working with large datasets, including issues such as researcher degrees of freedom, causal bias, and multiple comparisons. We'll also explore some emerging methods for assessing these validity threats while planning studies, such as performing a comprehensive design analysis that goes beyond a traditional power analysis.
Biography
Ethan C. Brown is Associate Director of the Research Methodology Consulting Center, in the College of Education and Human Development at the University of Minnesota. His interests lie at the intersection of improving methodological practice and studying people's understanding of statistics. As a methodological consultant and practical statistician, he works with educational researchers to improve their design and analysis; as a software developer, he develops packages that help to make advanced methods and analysis more accessible to applied researchers; as a researcher, he studies how people understand uncertainty, particularly with regard to sample size and power. He holds a Ph.D. in Educational Psychology from the University of Minnesota.
