What is a p-value?

The 2016 statement of the American Statistical Association (ASA) offers an informal definition: the p-value is the chance, computed while a specified statistical model is assumed to hold, of getting a data summary (for example the difference between two sample means) at least as extreme as the one observed (Wasserstein and Lazar, 2016). The statement adds that, provided the model’s assumptions hold, a smaller p-value signals poorer agreement between the data and the null hypothesis. Greenland et al. (2016) stress that a p-value puts to the test every assumption behind the data-generating model, not only the null hypothesis.

The statement says the p-value is commonly misused and misinterpreted. Among its principles: a p-value is neither the probability that the hypothesis under study is true nor the probability that chance alone generated the data; it does not indicate how large an effect is or how much a result matters, since with a large enough sample even a tiny effect can give a small p-value; and a value near 0.05, taken alone, is only weak evidence against the null hypothesis.

The statement cautions against basing conclusions solely on whether a p-value crosses a cut-off such as 0.05, and asks for all analyses and p-values to be reported. The Cochrane Handbook calls the 0.05 threshold arbitrary and asks for exact P values with confidence intervals.