Investigation of the behavior of classical criteria for multiple comparisons, on abnormal heterogeneous distributions, by the Monte-Carlo method

Abstract

Traditional analytical methods for studying the applicability of multiple comparison methods are effective only under very strict restrictions on the corresponding general populations. At the same time, computer simulations and the Monte Carlo method can be successfully applied to solve this issue. Using the Monte Carlo method, we simulate the performance of tests performed in multiple comparisons on small samples from distorted (compared to normal) distributions. We investigate the possibility of using the classical ANOVA tests and the nonparametric Kruskal — Wallis test for small samples with a non-normal distribution and / or heterogeneous in variance. The Levene test is used as a criterion for the homogeneity of samples in terms of variances. Normality (Gaussianity) of samples is checked using the Shapiro — Wilk test. To distort the normality of samples, general populations are used distributed over the chi-square and Student’s t-distribution with a small number of degrees of freedom. Also, the non-normality of the distributions is tracked using the parameters: kurtosis (peakedness coefficient) and skewness. The so-called Mersenne Twister implemented within the Wolfram Mathematica software package is used as a pseudo-random number generator. The number of tests for each set of parameters has been brought to a million. The effective probabilities of errors of the 1st kind are calculated and conclusions are drawn about the influence of inhomogeneity of variances, non-normality of kurtosis and asymmetry on the effectiveness of the studied criteria. As a result, we can say that there is often no reason to use nonparametric methods instead of parametric ones, to the detriment of the power of the corresponding criteria.