Abstract
When conducting a statistical test one of the initial risks that must be considered is a Type I error, also known as a false positive. The Type I error rate is set by nominal alpha, assuming all underlying conditions of the statistic are met. Experiment-wise Type I error inflation occurs when multiple tests are conducted overall for a single experiment. There is a growing trend in the social and behavioral sciences utilizing nested designs. A Monte Carlo study was conducted using a two-layer design. Five theoretical distributions and four real datasets taken from Micceri (1989) were used, each with five different sample sizes and conducted with nominal alpha set to 0.05 and 0.01. These were conducted both unconditionally and conditionally. All permutations were performed for 1,000,000 repetitions. It was found that when conducted unconditionally, the experiment-wise Type I error rate increases from alpha = 0.05 to 0.10 and 0.01 increases to 0.02. Conditionally, it is extremely unlikely to ever find results for the factor, as it requires a statistically significant nest as a precursor, which leads to extremely reduced power. Hence, caution should be used when interpreting nested designs.
DOI
10.22237/jmasm/1493596980
Recommended Citation
Sawilowsky, J. & Markman, B. (2017). Experiment-wise Type I error rates in nested (hierarchical) study designs. Journal of Modern Applied Statistical Methods, 16(1), 52-68. doi: 10.22237/jmasm/1493596980
Included in
Applied Statistics Commons, Social and Behavioral Sciences Commons, Statistical Theory Commons
