13.6 Testing hypotheses

The brms package also contains a useful function to address hypotheses about model parameters. The function brms::hypothesis can compute Bayes factors for point-valued hypotheses using the Savage-Dickey method. It also computes a binary test of whether a point-valued hypothesis is credible based on inclusion in a Bayesian credible interval. For interval-valued hypotheses \(\theta \in [a;b]\), the function brms::hypothesis computes the posterior odds (called evidence ratio in the context of this function):63 \[ \frac{P(\theta \in [a;b] \mid D)}{P(\theta \not \in [a;b] \mid D)} \]

Computing Bayes factors for point-valued hypotheses with brms::hypothesis requires proper priors for all parameters that are part of the hypothesis. It also requires taking samples from the priors of parameters.64 So, here is a function call of brms:brm which (i) specifies a reasonably unconstrained but proper parameter for the slope coefficient for year and (ii) also collects samples from the prior (by setting the option sample_prior = "yes"):

fit_temperature_weakinfo <- brm(
  # specify what to explain in terms of what
  #  using the formula syntax
  formula = avg_temp ~ year,
  # which data to use
  data = aida::data_WorldTemp,
  # weakly informative prior for slope
  prior = prior(student_t(1, 0, 1), coef = year),
  # option to sample from priors as well
  # (necessary for calculating BFs with Savage-Dickey)
  sample_prior = 'yes',
  # increase number of iterations (for precision of estimates)
  iter = 20000
)

Before addressing hypotheses about the slope parameter for year, let’s remind ourselves of the summary statistics for the posterior:

brms::posterior_samples(fit_temperature_weakinfo) %>% 
  pull(b_year) %>% 
  aida::summarize_sample_vector()
## # A tibble: 1 × 4
##   Parameter  `|95%`    mean  `95%|`
##   <chr>       <dbl>   <dbl>   <dbl>
## 1 ""        0.00565 0.00627 0.00689

The main “research hypothesis” of interest is whether the slope for year is credibly positive. This is an interval-valued hypothesis and we can test it like so:

hypothesis(fit_temperature_weakinfo, "year > 0")
## Hypothesis Tests for class b:
##   Hypothesis Estimate Est.Error CI.Lower CI.Upper Evid.Ratio Post.Prob Star
## 1 (year) > 0     0.01         0     0.01     0.01        Inf         1    *
## ---
## 'CI': 90%-CI for one-sided and 95%-CI for two-sided hypotheses.
## '*': For one-sided hypotheses, the posterior probability exceeds 95%;
## for two-sided hypotheses, the value tested against lies outside the 95%-CI.
## Posterior probabilities of point hypotheses assume equal prior probabilities.

The table shows the estimate for the slope of year, together with an estimated error, lower and upper bounds of a credible interval (95% by default). All of these numbers are rounded. It also shows the “Evidence ratio” which, for an interval-valued hypothesis is not the Bayes factor, but the posterior odds (see above). In the present case, an evidence ratio of Inf means that all posterior samples for the slope coefficient were positive. This is also expressed in the posterior probability (“Post.Prod” in the table) for the proposition that the interval-valued hypothesis is true (given data and model).

The following tests a point-valued hypothesis:

hypothesis(fit_temperature_weakinfo, "year = 0.005")
## Hypothesis Tests for class b:
##           Hypothesis Estimate Est.Error CI.Lower CI.Upper Evid.Ratio Post.Prob
## 1 (year)-(0.005) = 0        0         0        0        0       0.17      0.15
##   Star
## 1    *
## ---
## 'CI': 90%-CI for one-sided and 95%-CI for two-sided hypotheses.
## '*': For one-sided hypotheses, the posterior probability exceeds 95%;
## for two-sided hypotheses, the value tested against lies outside the 95%-CI.
## Posterior probabilities of point hypotheses assume equal prior probabilities.

For this point-valued hypothesis, the estimate (and associated error and credible interval) are calculated as a comparison against 0, as shown in the “Hypothesis” column. The evidence ratio given in the results table is the Bayes factor of the point-valued hypothesis against the embedding model (the full regression model with the prior we specified), as calculated by the Savage-Dickey method. As before, the posterior probability is also shown. The “Star” in this table indicates that the point-valued hypothesis is excluded from the computed credible interval, so that - if we adopted the (controversial) binary decision logic discussed in Chapter 11 - we would reject the tested hypothesis.

  1. Notice that for priors where \(P(\theta \in [a;b]) = 0.5\), the posterior odds equal the Bayes factor. For other priors, we’d need to correct the posterior odds by the priors to obtain Bayes factors, something that the brms package does not (presently seem to) do, unfortunately.↩︎

  2. It may seem unnecessary to take prior samples for parameters, because, after all, couldn’t we just look at the (closed-form) definition of the prior for that parameter? Well, that only works for top-level parameters, but not parameters in a hierarchical model which depend on the value of other parameters and which therefore have no (easily accessible) closed-form prior to look up.↩︎