Homework 8: Hypothesis Testing

Instructions

• Create an Rmd-file with the matrikel number of each group member (equivalent to StudIP group) in the ‘author’ heading and answer the following questions.
• When all answers are ready, ‘Knit’ the document to produce a HTML file.
• Create a ZIP archive called “IDA_HW8-Group-XYZ.zip” (where ‘XYZ’ is your group) containing:
  • an R Markdown file “IDA_HW8-Group-XYZ.Rmd”
  • a knitted HTML document “IDA_HW8-Group-XYZ.html”
• Upload the ZIP archive on Stud.IP in your group folder before the deadline. You may upload as many times as you like before the deadline, only your final submission will count.
• Please do not suppress the code in the HTML-Output

• Include an R code chunk in your Rmarkdown file (the preamble) in which you set the following global options for the document, and set the options for this code chunk to echo = F (so as not to have it show up in your output):

• Load the package tidyverse.

Exercise 1: Adressing hypotheses about coin flips with hypothesis testing (45 points)

The goal of this exercise is to get experience in calculating \(p\)-values for different kinds of (directed and undirected) hypotheses. We use the (allegedly easiest) case of coin flips. We also would like to get more confident in how to report the results we obtain from our code.

To obtain the right inuitions for \(p\)-values for directed and undirected hypotheses, you can use the function plot_binomial provided below. It allows you to plot the (binomial) sampling distribution and to specify which (if any) parts of the plot you want to highlight. It also calculates the total probability for all the values (of \(k\)) in the vector supplied to its argument highlight. The result of this calculation is shown in the plot’s title. See the examples below to understand what the function does and how you can use it to develop better intuitions about \(p\)-values.

plot_binomial <- function(theta, N, highlight = NULL) {
  # put the data together
  plotData <- tibble(x = 0:N, y = dbinom(0:N, N, theta))
  # make a simple bar plot
  out_plot <- ggplot(plotData, aes(x = x , y = y )) + 
    geom_col(fill = "gray", width = 0.35) +
    labs(
      x = "test statistic k",
      y =  str_c("Binomial(k, ", N, ", ", theta, ")")
    )
  # if given, highlight some bars in red
  if (!is.null(highlight)) {
    plotData2 = tibble(x = highlight, y = dbinom(highlight, N, theta))
    out_plot <- out_plot + 
      geom_col(
        data = plotData2, 
        aes(x = x, y = y), 
        fill = "firebrick", 
        width = 0.35
      )  +
      ggtitle(
        str_c(
          "Prob. selected values: ", 
          sum(dbinom(highlight, N, theta)) %>% signif(5)
          )
        )
  }
  out_plot
}
plot_binomial(theta = 0.5, N = 24, highlight = c(7:16))

plot_binomial(
  theta = 0.5, 
  N = 24, 
  highlight = which(dbinom(0:24, 24,p=0.5) <= dbinom(7, 24,p=0.5))-1
)

In the following, you will be confronted with different scenarios, each of which has a different research hypothesis. For each of these, think about what it is that you want to learn from a test based on a \(p\)-value. For each scenario, you should therefore fix a suitable null hypothesis about the coin’s bias. Remember that a \(p\)-value quantifies evidence against the specified null-hypothesis (keeping an alternative hypothesis in the back of our minds to distinguish the case of genuinely testing a point-valued null hypothesis from the case of testing an interval-based null hypothesis via a single value used to generated the sampling distribution). You should therefore specify a null-hypothesis (and an alternative hypothesis) that is most conducive of sheddling light on your research question. (Once more, the goal of this exercise is for you to become more comfortable with the whole logic of using \(p\)-values to draw conclusions of interest for a research goal.) When asked to judge significance, please use a significance level of \(\alpha = 0.05\).

Exercise 2: Pearson’s \(\chi^2\)-test of goodness of fit (20 points)

The goal of this exercise is to make you feel comfortable with applying and interpreting the results of a Pearson \(\chi^2\)-test of goodness of fit.

Imagine you are on a funfair (German: Kirmes, Jahrmarkt). As usual, you head straight for the lottery booth (German: Losbude). The vendor adverises that of all tickets 5% are mega-winners, 15% are winners, 15% are free rides on the fairy-go-round (German: Karussell), 35% are consolation prizes (German: Trostpreise) and only the remaining 30% are blanks (German: Nieten). You are your nerdy self, as usual, and you buy 50 tickets and count the number of tickets in each category. What you got is this:

n_obs <- c(
  mega_winner = 1, # hurray!
  winner = 2,
  free_ride = 10,
  consolation = 18,
  blank = 19
)

expected <- c(
  mega_winner = 5,
  winner = 15,
  free_ride = 15,
  consolation = 35,
  draws = 30
) * sum(n_obs) / 100

Exercise 3: Some claims about frequentist testing (15 points)

The goal of this exercise is to make you think deeply about some of the key notions we discussed in class, and how they might relate to each other. On top of this, this exercise is a good preparation towards the final exam, which is likely to contain some truth-value judgement questions similar to this exercise. (Caveat: This is not to say that the kinds of statements to occur in the exam will be or feel exactly like these; they might be easier or, by the time, more familiar.)

For the following statements judge whether they are true or false. If you think that a case is somehow controversial, you can give one short sentence to justify your response.

1. A \(p\)-value of \(0.00615\) should be interpreted as implying that the probability that the null-hypothesis is true is below 1%.
2. If we obtain a 95% confidence interval of \([0.2; 0.4]\) for a binomial test, then we know that a two-sided binomial test for the null hypothesis \(\theta = 0.1\) will be statistically significant at the significance level \(\alpha = 0.05\).
3. If we obtain a significant test result for a two-sided binomial test for the null hypothesis \(\theta = 0.1\), we know that the corresponding 95% CI for the observed data will include the value 0.1.
4. The Central Limit Theorem implies that a binomial distribution for large enough \(N\) is closely approximated by a normal distribution, no matter what \(\theta\) we are assuming.
5. If we obtain a significant test result at significance level \(\alpha = 0.05\), then this means that when we repeat the exact same experiment we will find a significant result in at least 95% of the cases.