Using Bayesian Multilevel Models to Control for Multiplicity among Means

Authors

Michael J. Zweifel, University of Nebraska-LincolnFollow

First Advisor

Rafael De Ayala

Date of this Version

Fall 11-30-2018

Document Type

Article

Comments

A DISSERTATION Presented to the Faculty of The Graduate College at the University of Nebraska In Partial Fulfillment of Requirements For the Degree of Doctor of Philosophy, Major: Psychological Studies in Education (Quantitative, Qualitative, and Mixed Methods), Under the Supervision of Professor Rafael J. De Ayala. Lincoln, Nebraska: November, 2018

Copyright (c) 2018 Michael Zweifel

Abstract

It is well known that the Type I error rate will exceed α when multiple hypothesis tests are conducted simultaneously. This is known as Type I error inflation. The probability of committing a Type I error grows monotonically as the number as the number of hypothesis being tested increases. A class of methods, known as multiple comparison procedures, has been developed to combat this issue. However, in turn for maintaining the Type I error rate below α, multiple comparison procedures sacrifice power to correctly reject false hypotheses. The loss of power is exacerbated when variance heterogeneity is present.

In the case of making multiple comparisons among means, a possible alternative to multiple comparison procedures is to use Bayesian multilevel models to control for Type I error inflation. Bayesian multilevel models reduce the risk of committing a Type I error by shrinking all means towards the grand mean, in turn, making it more difficult to declare any mean significantly different from one another.

To compare the performance of multiple comparison procedures and Bayesian multilevel models, a Monte Carlo simulation study, in which the number of hypotheses and variance heterogeneity was manipulated, was conducted. The results indicated that the Bayesian multilevel models maintain the Type I error rate at α and display greater power than the traditional methods when a large number of hypotheses are tested. When the number of hypotheses tested were small, the Bayesian models were not able to maintain strong control of the Type I error rate.

Adviser: Rafael J. De Ayala