Andrew Gelman and John Carlin have a fantastic new paper on why we should move beyond power calculations to concentrating on Type S (sign) and Type M (magnitude) errors: http://www.stat.columbia.edu/~gelman/research/published/PPS551642_REV2.pdf

It’s very readable and explains intuitively why these errors occur, how to calculate their probability, and how to reduce their probability. They advocate considering Type S and M errors both when designing a study, but also later when interpreting statistical analyses so we have a better idea of how confident (or not) we should be in our results.

A couple of quotes:

It is quite possible for a result to be significant at the 5% level—with a 95% confidence interval that entirely excludes zero—and for there to be a high chance, sometimes 40% or more, that this interval is on the wrong side of zero.

Using statistical significance as a screener can lead researchers to drastically overestimate the magnitude of an effect.

They include R code to calculate the probability of Type S and Type M errors and use two studies as examples. One of them is a UBC psychology study where they show that there’s a 24% chance that the authors found an effect in the wrong direction and the effect they did find is expected to be almost 10 times too high. The data in these psychology studies are not that different from the data we often work with in ecology, so take note! Small expected effects and lots of noise require a lot of data, there’s no way around it.

A figure on Gelman’s blog that hits home what 6% power looks like in terms of Type S and Type M errors. This shows the expected distribution of measured effect sizes. At low power, if an effect is significant it must be in the wrong direction or an exaggeration of the effect. http://andrewgelman.com/2014/11/17/power-06-looks-like-get-used/