17.9 Testing hypotheses by estimation, comparison & model checking

Each of the “three pillars of data analysis” discussed in the previous section can be used to test a statistical hypothesis. This is where we see a further difference between Bayesian and frequentist approaches.

Bayesian hypothesis testing uses either parameter estimation or model comparison (e.g., Bayes factors), as discussed extensively in Chapter 11.

Frequentist hypothesis testing in terms of \(p\)-values is based on the third pillar “prediction”. To see this, recall that \(p\)-values are derived from the assumption that the point-valued null hypothesis is true. We then ask: based on a model which assumes that the null-hypothesis is true, would we be surprised by the data we observed?; which is the same as asking: would we have predicted the data we actually saw?

Seeing this difference also explains why sometimes frequentist and Bayesian approaches give different results when testing the same null-hypothesis based on the same observed data. The next section discusses such a case.