I believe that there is at least one moment (to say the least) in the life of a scientist in which he thought to his data as a sort of superposition between “statistically significant” and something more like “well… there’s definitelly a trend there”, the latter definition often used when the difference observed in the data sets didn’t reach the statistical threshold to be deemed “non-casual”. When the second case occurs, the only conclusion is that the two (or more) mean values belong to samples of the same population that happened to be different only by sheer chance, and not due to our experimental hypothesis. Usually, in bio-medical sciences, this threshold is represented by alpha=0.05 and, quite unfortunately it has gradually become the magical door that divides undignified results from striking discoveries -some sort of holy grail of scientific literature, and has transformed several scientifical endeavors in trivial search for a p-value that was lower of 0.05. Things, however, might be mature enough for a “surgical strike on thoughtless testing of statistical significance”.
However, according to a recent Comment appeared last week on Nature website, the dichotomous use of p values might represent a tremendously limited approach when it comes to discuss scientific data. The paper also tested a sample of 791 papers from 5 journals finding a 51% ratio of wrong interpretations from incorrect use of p-value. When it comes to data sets – and the conclusions to which they must lead – strict and absolute categorization might not really be the best approach in that all statistics can naturally vary, even in an unexpected measure, between different studies. As a matter of fact we should stop using “non significant” as a synonym for “not-true” and start embracing a little degree of uncertainty in our data, discussing data in terms of compatibility with an assumption, rather than define as “irrelevant” any result that diverts even a little bit from significancy boundaries. The paper gives four advices to understand the caveats of statistical categorizing approaches and avoid them.