parameter estimation:

\[\underbrace{P(\theta \, | \, D)}_{posterior} \propto \underbrace{P(\theta)}_{prior} \ \underbrace{P(D \, | \, \theta)}_{likelihood}\]

model comparison

\[\underbrace{\frac{P(M_1 \mid D)}{P(M_2 \mid D)}}_{\text{posterior odds}} = \underbrace{\frac{P(D \mid M_1)}{P(D \mid M_2)}}_{\text{Bayes factor}} \ \underbrace{\frac{P(M_1)}{P(M_2)}}_{\text{prior odds}}\]

prediction

1. model-based beliefs about what is likely to happen
   • practical decision making
2. model comparison
   • information criteria
   • Bayes factor
3. model criticism
   • is the model any good?
   • given belief in the model, should we be shocked by this new observation?
     [ \(p\)-values ]

in the general case, the \(p\)-value of observation \(x\) under null hypothesis \(H_0\), with sample space \(X\), sampling distribution \(P(\cdot \mid H_0) \in \Delta(X)\) and test statistic \(t \colon X \rightarrow \mathbb{R}\) is:

\[ p(x ; H_0, X, P(\cdot \mid H_0), t) = \int_{\left\{ \tilde{x} \in X \ \mid \ t(\tilde{x}) \ge t(x) \right\}} P(\tilde{x} \mid H_0) \ \text{d}\tilde{x}\]

intuitive slogan: probability of at least as extreme outcomes

for an exact test we get:

\[ p(x ; H_0, X, P(\cdot \mid H_0)) = \int_{\left\{ \tilde{x} \in X \ \mid \ P(\tilde{x} \mid H_0) \le P(x \mid H_0) \right\}} P(\tilde{x} \mid H_0) \ \text{d}\tilde{x}\]

intuitive slogan: probability of at least as unlikely outcomes

• data: we flip \(n=24\) times and observe \(k = 7\) successes
• null hypothesis: \(\theta = 0.5\)
• sampling distribution: binomial distribution

\[ B(k ; n = 24, \theta = 0.5) = \binom{n}{k} \theta^{k} \, (1-\theta)^{n-k} \]

binom.test(7,24)$p.value

## [1] 0.06391466

• observed: \(k = 7\) out of \(N = 24\) flips came up heads
• goal: estimate \(P(\theta \mid D)\) & determine posterior 95% HDI

regions of practical equivalence

• small regions \([\theta - \epsilon, \theta + \epsilon]\) around each \(\theta\)
  • values (practically) indistinguishable from \(\theta\)

credible values

• value \(\theta\) is rejectable if its ROPE lies entirely outside of posterior HDI
• value \(\theta\) is believable if its ROPE lies entirely whithin posterior HDI

NHST by ROPE for our example

\(\theta = 0.5\) is rejectable for all ROPEs with ca. \(\epsilon \le 0.02\)

• \(M_0\): \(\theta = 0.5\) & \(k \sim \text{Binomial}(0.5, N)\)
• \(M_1\): \(\theta \sim \text{Beta}(1,1)\) & \(k \sim \text{Binomial}(\theta, N)\)

"paradox": two established methods give contradictory results

k = 49581
N = 98451

binom.test(k, N)$p.value

## [1] 0.02364686

dbeta(0.5, k+1, N - k + 1)

## [1] 19.21139

• let the true bias be \(\theta = 0.5\)
• generate all possible outcomes \(k\) keeping \(N\) fixed
  • \(N \in \{ 10, 100, 1000, 10000, 100000 \}\)
  • true frequency of \(k\) is \(Binomial(k \mid N, \theta = 0.5)\)
• look at the frequency of test results, coded thus:

BF selects \(H_0\) correctly with prob. 0.986 for \(N = 10000\), and with 0.996 for \(N = 100000\).

• parameter estimation: what \(\theta\) to believe in?
• model comparison: which model is better than another?
• model criticism: is a given model plausible (enough)?

exponential forgetting model

y = c(.94, .77, .40, .26, .24, .16)
t = c(  1,   3,   6,   9,  12,  18)
obs = y*100

model{
  a ~ dunif(0,1.5)
  b ~ dunif(0,1.5)
  for (i in 1: 6){
    p[i] = min(max( a*exp(-t[i]*b), 0.0001), 0.9999)
    obs[i] ~ dbinom(p[i], 100)    # condition on data
    obsRep[i] ~ dbinom(p[i], 100) # replicate fake data
  }
}

• black dots: data
• blue dots: mean of replicated fake data
• blue bars: 95% HDIs of replicated fake data

• black dots: data
• blue dots: mean of replicated fake data
• blue bars: 95% HDIs of replicated fake data

• fix a data set \(x\) from a set of possible observations \(X\)
• fix a model \(M\) with \(P(\theta)\) and \(P(X = x \mid \theta)\)
  • NB: \(P(\theta)\) can be conditioned on data: prior/posterior predictive \(p\)-value
• fix a test statistic \(t \colon X, \theta \rightarrow \mathbb{R}\)
  • test statistic may depend on parameters
• Bayesian predictive \(p\)-value:

\[ p(x ; X, M = \tuple{P(\theta), P(x\mid\theta)}, t) = \int_{\left\{ \tilde{x} \in X \ \mid \ t(\tilde{x},\theta) \ge t(x, \theta) \right\}} \int P(\tilde{x} \mid \theta) P(\theta) \ \text{d}\theta \ \text{d}\tilde{x}\]

• compare to standard \(p\)-value definition:

\[ p(x ; H_0, X, P(\cdot \mid H_0), t) = \int_{\left\{ \tilde{x} \in X \ \mid \ t(\tilde{x}) \ge t(x) \right\}} P(\tilde{x} \mid H_0) \ \text{d}\tilde{x}\]

obs = c(1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0)
k = sum(obs) # 7
N = length(obs) #20

binom.test(k, N, 0.5)$p.value

## [1] 0.263176

Friday

• bootcampling GCM (Lee & Wagenmakers ch. 17)

Tuesday

• introduction to Stan