Null hypothesis significance testing has successfully reduced the complexity of scientific inference to a dichotomous decision (i.e., ‘reject’ versus ‘not reject’). As a consequence,
p values and their associated statistical significance play an important role in the social and medical sciences. But do we truly understand what statistical significance testing and
p values entail? Judging by the vast literature on controversies regarding their application and interpretation, this seems questionable. It has even been argued that significance testing should be abandoned all together [
2]. We seek to extend Fayer’s [
3] paper on statistically significant correlations and to clarify some of the controversies regarding statistical significance testing by explaining that (1) the
p value is not the probability of the null hypothesis; (2) rejecting the null hypothesis does not prove that the alternative hypothesis is true; (3) not rejecting the null hypothesis does not prove that the alternative hypothesis is false; (4) statistical significance testing is not necessarily an objective evaluation of results; and (5) the
p value does not give an indication of the size of the effect. …
