Why does the experiment wise error rate of a multiplecomparisons procedure differ from the significance level for each comparison (assuming the experiment has more than two treatments)?

In this question, the author requests an explanation of the difference between the significance level for each comparison and the experiment-wise error rate of a multiple comparisons procedure.

The likelihood of making at least one Type I error throughout an entire research study in a test involving multiple comparisons. The probability of making a type1 error when conducting a particular test or comparison is known as the test error rate, which is distinct from the experiment-wise error rate.

The simple definition of Bonferroni's inequality is that the experiment-wise error rate equals the sum of the comparison-wise error rates. As a result, the possibility of a type I error occurring anywhere in a list of 10 hypotheses that have all been tested at a level of α=0.01 is no greater than 10*0.01=0.1.

With this approach, inequality is applied backward. For our list of g hypotheses, we set a desired experiment-wise rate (often α_E = 0.1 or 0.5), each test is executed as usual.

The actual rate is probably lower because the Bonferroni Inequality provides the maximum error rate. Because of its simplicity, this method is frequently used to control experiment-wise error rates even though it is pretty conservative. It is incredibly flexible in its applications and is not confined to ANOVA situations.

For instance, we might use t-tests to compare consumption levels to national norms for 100 different foods in a study of dietary habits among lung cancer patients. Even if diet has nothing to do with lung cancer, you would expect 100*.05="false significances" in the results. If you run each of these g=100 tests at a significance level of α_E = 0.05, you could manage this by setting α_E = 0.1; each t-test would then use α= 0.1/100 to control it.

It serves to illustrate two main ideas behind the multiple comparison problem. First, we must be more cautious with each hypothesis the longer the list of possible explanations. Second, whenever it is practical, we should employ a technique specifically tailored to the analysis's design and more effective than Bonferroni's. For example, methods such as Tukey's HSD discussed earlier are designed for pairwise comparisons within the one-way ANOVA. They are more effective than the incredibly cautious Bonferroni method.