Logical contradictions in the One-way ANOVA and Tukey-Kramer multiple comparisons tests with more than two groups of observations
share
We show that the One-way ANOVA and Tukey-Kramer (TK) tests agree on any sample with two groups. This result is based on a simple identity connecting the Fisher-Snedecor and studentized probabilistic distributions and is proven without any additional assumptions; in particular, the standard ANOVA assumptions (independence, normality, and homoscedasticity (INAH)) are not needed. In contrast, it is known that for a sample with k > 2 groups of observations, even under the INAH assumptions, with the same significance level α, the above two tests may give opposite results: (i) ANOVA rejects its null hypothesis H_0^A: μ_1 = … = μ_k, while the TK one, H_0^TK(i,j): μ_i = μ_j, is not rejected for any pair i, j ∈{1, …, k}; (ii) the TK test rejects H_0^TK(i,j) for a pair (i, j) (with i ≠ j) while ANOVA does not reject H_0^A. We construct two large infinite pseudo-random families of samples of both types satisfying INAH: in case (i) for any k ≥ 3 and in case (ii) for some larger k. Furthermore, in case (ii) ANOVA, being restricted to the pair of groups (i,j), may reject equality μ_i = μ_j with the same α. This is an obvious contradiction, since μ_1 = … = μ_k implies μ_i = μ_j for all i, j ∈{1, …, k}. Similar contradictory examples are constructed for the Multivariable Linear Regression (MLR). However, for these constructions it seems difficult to verify the Gauss-Markov assumptions, which are standardly required for MLR.
READ FULL TEXT