Joining the Culture of Open Science¶
What is open science and why do it?¶
Check out Easing Into Open Science: A Tutorial for Graduate Students
Learning about open science¶
Blogs¶
Center for Open Science: https://cos.io/blog/
Data Colada: http://datacolada.org/
Michael Frank (Stanford): http://babieslearninglanguage.blogspot.com/search/label/Reproducibility
Russ Poldrack (Stanford): http://www.russpoldrack.org/
Sanjay Srivastava (University of Oregon): https://thehardestscience.com/
Simine Vazire (University of Melbourne): https://sometimesimwrong.typepad.com/
Books¶
Understanding Psychology as a Science (Zoltan Dienes): https://www.amazon.com/Understanding-Psychology-Science-Introduction-Statistical/dp/B00FGVUQWU
Rigor Mortis (Richard Harris): https://www.amazon.com/Rigor-Mortis-Science-Worthless-Billions/dp/0465097901
Transparent Reproducible Social Science (Garret Christensen, Jeremy Freese, Edward Miguel): https://www.amazon.com/Transparent-Reproducible-Social-Science-Research/dp/0520296958
Courses¶
Stanford Courses
CHPR 206: Meta-research: Appraising Research Findings, Bias, and Meta-analysis
Instructor: John Ioannidis
PSYCH 251: Statistical Methods for Behavioral and Social Sciences
Instructor: Mike Frank
Link to older version of class: https://web.stanford.edu/class/psych254/
HRP 264: Foundations of Statistical and Scientific Inference
Instructor: Steven Goodman
MED73N: Scientific Method and Bias
Instructor: John Ioannidis
External Courses
List of open science/reproducibility syllabi: https://osf.io/vkhbt/wiki/home/
Neuro summer workshop at University of Washington: https://neurohackademy.org
Daniel Lakens’ Coursera courses:
People/Twitter Handles¶
Andrew Gelman: @StatModeling
Brian Nosek: @BrianNosek
Charlie Ebersole: @CharlieEbersole
Chris Gorgolewski: @chrisgorgo
Daniël Lakens: @lakens
Eiko Fried: @EikoFried
EJ Wagenmakers: @EJWagenmakers
Joe Simmons: @jpsimmon
John Ioannidis: (no twitter)
Marcus Munafò: @MarcusMunafo
Michael C. Frank: @mcxfrank
Neil Lewis, Jr.: @NeilLewisJr
Rickard Carlsson: @RickCarlsson
Russ Poldrack: @russpoldrack
Sanjay Srivastava: @hardsci
Simine Vazire: @siminevazire
Steven Goodman: @goodmanmetrics
Tal Yarkoni: @talyarkoni
Uri Simonsohn: @uri_sohn
Podcasts¶
Everything Hertz: https://everythinghertz.com/
Marginally Significant: https://marginallysignificant.fireside.fm/
Open Science Radio: http://www.openscienceradio.org/
ReproducibiliTea: https://podcasts.apple.com/gb/podcast/reproducibilitea-podcast/id1406176044
The Black Goat: http://www.theblackgoatpodcast.com/
Two Psychologists Four Beers: https://fourbeers.fireside.fm/
*Quantitude: https://quantitudethepodcast.org/
*Very Bad Wizards: https://verybadwizards.fireside.fm/
*Not devoted solely to open science, but have relevant episodes
Foundational papers¶
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science, 22(11), 1359-1366. https://doi.org/10.1177/0956797611417632
Ioannidis, J. P. (2005). Why most published research findings are false. PLos Med, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124
Watson, M. (2015). When will ‘open science’ become simply ‘science’?. Genome biology, 16(1), 1-3. https://doi.org/10.1186/s13059-015-0669-2
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., … & Contestabile, M. (2015). Promoting an open research culture. Science, 348(6242), 1422-1425. https://doi.org/10.1126/science.aab2374
Wagenmakers, E. J., Wetzels, R., Borsboom, D., van der Maas, H. L., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7(6), 632-638. https://doi.org/10.1177/1745691612463078
Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., Du Sert, N. P., … & Ioannidis, J. P. (2017). A manifesto for reproducible science. Nature human behaviour, 1(1), 1-9. https://doi.org/10.1038/S41562-016-0021
Levin, N., & Leonelli, S. (2017). How does one “open” science? Questions of value in biological research. Science, Technology, & Human Values, 42(2), 280-305. https://doi.org/10.1177/0162243916672071
Gorgolewski, K. J., & Poldrack, R. A. (2016). A practical guide for improving transparency and reproducibility in neuroimaging research. PLoS biology, 14(7). https://doi.org/10.1371/journal.pbio.1002506
Byers-Heinlein, K., Bergmann, C., Davies, C., Frank, M. C., Hamlin, J. K., Kline, M., … & Mastroberardino, M. (2020). Building a collaborative psychological science: Lessons learned from ManyBabies 1. Canadian Psychology.
Talks¶
Association for Psychological Science: Improving the Reproducibility of Our Research Practices: https://www.youtube.com/playlist?list=PLMOU-iLiJIc0amNVabGXJ0liKwIwxqkO8
Metascience 2019 Symposium: https://www.metascience2019.org/presentations/
Doing open science¶
Tools/Platforms¶
Open Science Framework: https://osf.io/
GitHub: https://github.com/
AsPredicted: https://aspredicted.org/
PsyArXiv: https://psyarxiv.com/
Initiatives/Collaborations¶
Many Labs: https://osf.io/89vqh/
Psychological Science Accelerator: https://psysciacc.org/
repliCATS: https://replicats.research.unimelb.edu.au/
Open Neuro: https://openneuro.org/
Cognitive Atlas: http://www.cognitiveatlas.org/
NeuroVault: https://neurovault.org/
Many Babies: https://osf.io/rpw6d/
Organizations/Centers/Labs¶
Stanford METRICS: https://metrics.stanford.edu/
Stanford Center for Reproducible Neuroscience: https://reproducibility.stanford.edu/
BITSS: https://www.bitss.org/
Center for Open Science: https://cos.io/
Credibility Lab: https://credlab.wharton.upenn.edu/
Research on Research Institute: http://researchonresearch.org/
Societies/Conferences¶
Metascience Symposium: https://www.metascience2019.org/
Society for Improving Psychological Science (SIPS): https://improvingpsych.org/
Organization of Human Brain Mapping (OHBM) https://humanbrainmapping.org
Journals¶
TOP guidelines: journal rankings by open science practices: https://cos.io/top/
Specific journals
Advances in Methods and Practices in Psychological Science: https://journals.sagepub.com/home/ampa
Collabra: Psychology: https://www.collabra.org/
Meta-Psychology: https://open.lnu.se/index.php/metapsychology/about
Funding & Grants¶
NSF: Open Science for Research Data: https://nsf.gov/pubs/2020/nsf20068/nsf20068.jsp
Posted March 27, 2020
List of funding types offered through the NSF’s Science of Science and Innovation Policy: https://www.nsf.gov/funding/pgm_summ.jsp?pims_id=501084