The use of p-values in statistical analysis and decision making has received increased criticism in academic writings over the past couple of years. This controversy was brought to the attention of the general public in two recent articles in The New Yorker [3] and The New York Times [1]. Furthermore, earlier this year, the Supreme Court [5] had to concern itself with the validity of statistical evidence established by p values in the context of drug trials.

In an influential paper, Sellke, Bayarri, and Berger [4] show that the probability of observing a given small p-value (of, for instance, 0.05) is about as high if the null hypothesis is true as when it is not. Consider the following experiment of a coin toss. Assume the coin is tossed 200 times and heads comes up 115 times. In a standard test on statistical significance, one would reject the null hypothesis of the coin being fair based on a p value of 0.04 in a two-tailed test. Yet, the probability of observing 115 heads is about the same under the assumption of the coin being fair (p=0.005956) as under the assumption of the coin not being fair (p=0.004975).

In the presentation, the speakers will make the case against p values but also show how to calibrate traditional p values as odds ratios based on Sellke, Bayarri, and Berger [4]. Further, they will show how to calculate the probability of the null hypothesis being true using methods of variable selection. Finally, based on Gelman and Weakliem [2], they will demonstrate how to evaluate the relevance of covariates that turn out statistically insignificant. All examples are demonstrated live and the associated R code is made available to the public.

Literature:
[1] Carey, Benedict, “You Might Already Know This...,” The New York Times, January 10, 2011.
[2] Gelman, Andrew, and David Weakliem (2008), “Of Beauty, Sex, and Power: Statistical Challenges in Estimating Small Effects,” http://www.stat.columbia.edu/~gelman/research/unpublished/power4r.pdf.
[3] Lehrer, Jonah, “The Truth Wears Off: Is There Something Wrong With the Scientific Method?” The New Yorker, December 13, 2010, http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer#ixzz1HTmXk2ni.
[4] Sellke, Thomas, M.J. Bayarri, and James O. Berger, “Calibration of P-values for Testing Precise Null Hypotheses,” Duke University, Institute of Statistics and Decision Sciences, 1999, http://www.stat.duke.edu/~berger/papers/99-13.html.
[5] United States Supreme Court, Syllabus: Matrixx Initiatives, Inc., et al. v. Siracusano et al., Certiorari to the United States Court of Appeals for the Ninth Circuit, No. 09-1156. Argued January 10, 2011-Decided March 22, 2011, http://www.supremecourt.gov/opinions/10pdf/09-1156.pdf.