The Difference Between Statistically Significant and Not Statistically Significant is Not Statistically Significant

By Andrew Gelman and Hal Stern.

The difference between 25+-10 (statistically significantly different from zero) and 10+-10 (not statistically significantly different from zero) is 15+-14 -- not statistically significant.

A good example was the research linking homosexuality to the number of older siblings -- number of older brothers was statistically significant, number of older sisters was not, but the difference between the two effects was not statistically significant!

Gerd Gigerenzer has an article "Mindless Statistics" criticizing the use of statistics in the field of psychology. Psychologists use the "null ritual" to publish papers and advance their careers, but don't take statistics seriously. Practicioners suffer from cognitive illusions which emphasize the "usefulness" of finding a statistically significant result.

If an effect with p<0.01 is found, the probability of the null hypothesis being true is not known, nor is the probability that the effect will be found in future experiments.