E.3 Chapter summary

In this chapter, we peeked into the abyss of psychology’s replication crisis and learned how biases towards significant outcomes fuel the engagement in questionable research practices, ultimately compromising the validity of the finding. We also learned that underpowered studies will not only hardly find true-positives, but also render detected effects useless. Lastly, the lack of transparency, be it in reported methods or keeping data and analysis code private, may account for moderators between the original and replication study.

Later, we looked at several promising solution attempts and initial prospects of improvement. It is important to note though, that there is no single panacea for all obstacles to good scientific practices. It is rather a combination of several measures, such as preregistration/Registered Reports, badges, TOP guidelines, and replication initiatives. The existence and increased implementation of such measures manifest that standards in psychological research are about to change. The replication crisis is followed by a credibility revolution, prompting psychology to reward what should be rewarded - scientific rigor and integrity.

Our little journey through the replication crisis ends here. Here are the most important points to keep in mind:

Limit exploitation of researcher degrees of freedom

  • Be aware of cognitive biases, such as confirmation bias, apophenia, and hindsight bias
  • Raise awareness and be aware of QRP’s and their implications for the validity of research outcomes
  • Preregister your study

Limit statistical fallacies

  • Learn more about Null Hypothesis Significance Testing
  • Consider adopting a Bayesian approach to hypothesis testing
  • Control for error rates
  • Make sure that your study has sufficient power (conduct a power analysis)
  • Keep in mind that correlation does not imply causation

Eliminate publication bias

  • Consider proposing an upcoming study as a Registered Report
  • Replicate!

Embrace transparency

  • Make your raw data, study materials, and analysis scripts available, along with
    • README files
    • Metadata/Codebooks
    • Descriptive comments and variable names in analysis scripts
  • You did not engage in selective reporting? Awesome! Let the reader know by explicitly writing a disclosure statement (e.g., 21-word solution)