title: “Statistical philosophy”
nocite: |
Gelman and Loken (2014), Dushoff, Kain, and Bolker (2019), Davidoff (1999), Pigliucci (2004), Berger (2003), McCullagh (2002), Gerber and Malhotra (2008), Goldacre (2011), Nieuwenhuis, Forstmann, and Wagenmakers (2011), Gelman and Stern (2006)

Introduction

This week is the introduction to the main statistical part of the course. We will be grappling with difficult, abstract questions about what we mean by scientific and statistical inference and about statistical philosophies.

Time spent grappling with these questions will make you a better scientist, and will provide you a strong foundation to navigate the various detailed questions that arise from particular questions and approaches.

Resources

Assignment

Pick one paper in your field. It can be a paper from your lab group or just a paper that you like. Carefully read any sections of the paper that report the results of statistical tests (including tests of assumptions; effects of nuisance parameters/covariates; and the primary scientific inferences). Write a document that gives the bibliographic reference of the paper and, for at least three of the tests done by the authors, (1) quote their presentation of the test results; (2) describe the issues, if any, with their presentation; (3) if there is any room for improvement in the presentation, write a revised statement that more accurately reflects what inferences we should make from the results. Table 1 of Dushoff, Kain, and Bolker (2019) may be useful (although you are not required to use “clarity” language).

See the assignment page to submit. Remember to email us the name of your repo file. Don’t submit in a proprietary format, like Word. Plain text is best, PDF is also OK.

Readings

We have listed a bunch of additional readings below: feel free to dip in!

You can also check out this fun web site on spurious correlations

References

Berger, James O. 2003. “Could Fisher, Jeffreys and Neyman Have Agreed on Testing?” Statistical Science 18 (1): 1–32. https://doi.org/10.1214/ss/1056397485.
Davidoff, Frank. 1999. “Standing Statistics Right Side Up.” Annals of Internal Medicine 130 (12): 1019–21. https://doi.org/10.7326/0003-4819-130-12-199906150-00022.
Dushoff, Jonathan, Morgan P. Kain, and Benjamin M. Bolker. 2019. “I Can See Clearly Now: Reinterpreting Statistical Significance.” Methods in Ecology and Evolution 10 (6): 756–59. https://doi.org/10.1111/2041-210X.13159.
Gelman, Andrew, and Eric Loken. 2014. “The Statistical Crisis in Science: Data-Dependent Analysis–a "Garden of Forking Paths"–Explains Why Many Statistically Significant Comparisons Don’t Hold Up.” American Scientist 102 (6): 460–60. http://link.galegroup.com/apps/doc/A389260653/AONE?u=ocul_mcmaster&sid=AONE&xid=4f4562c0.
Gelman, Andrew, and Hal Stern. 2006. “The Difference Between Significant and Not Significant Is Not Itself Statistically Significant.” The American Statistician 60 (4): 328–31. https://doi.org/10.1198/000313006X152649.
Gerber, Alan S., and Neil Malhotra. 2008. “Publication Bias in Empirical Sociological Research: Do Arbitrary Significance Levels Distort Published Results?” Sociological Methods & Research 37 (1): 3–30. https://doi.org/10.1177/0049124108318973.
Goldacre, Ben. 2011. “The Statistical Error That Just Keeps on Coming.” The Guardian, September. http://www.theguardian.com/commentisfree/2011/sep/09/bad-science-research-error.
Harrell, Frank. 2017. “Introduction.” Statistical Thinking. https://www.fharrell.com/post/introduction/.
McCullagh, Peter. 2002. “What Is a Statistical Model?” Annals of Statistics 30 (5): 1225–1310. https://doi.org/10.1214/aos/1035844977.
Nieuwenhuis, Sander, Birte U. Forstmann, and Eric-Jan Wagenmakers. 2011. “Erroneous Analyses of Interactions in Neuroscience: A Problem of Significance.” Nature Neuroscience 14 (9): 1105–7. https://doi.org/10.1038/nn.2886.
Nuzzo, Regina. 2014. “Scientific Method: Statistical Errors.” Nature 506 (7487): 150–52. https://doi.org/10.1038/506150a.
Pigliucci, Massimo. 2004. “Reject That Null Hypothesis!” https://web.archive.org/web/20040820160247fw_/http://life.bio.sunysb.edu/ee/pigliuccilab/handouts/reject_null_hypothesis.pdf.
Simmons, Joseph P., Leif D. Nelson, and Uri Simonsohn. 2011. “False-Positive Psychology Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.” Psychological Science 22 (11): 1359–66. https://doi.org/10.1177/0956797611417632.