Building a culture of open and reproducible science

Russ Poldrack

Stanford University

Culture: The norms, principles, and practices of an institution

Culture: The norms, principles, and practices of an institution

“Culture is created by the behaviors you tolerate” - Jacob Engel

Closed-science lab culture

• Competitive atmosphere
• High level of secrecy and paranoia
• Pressure to find specific results
• Lack of trust

Lab members are pitted against one another - They start out with an office in the cramped “loser room” - Publishing a paper in Science/Nature/Cell gets them a seat in a windowed office (and someone else gets kicked back to the “loser room”) - Lab members are forbidden to talk to members of another lab that studies the same topic, located in the same building - A postdoc is threatened with being fired (and losing their visa status) if they don’t find a result consistent with the PI’s hypothesis - The PI once scooped one of their own postdocs

Open-science lab culture

• Openness and transparency
• Collaborative atmosphere
• Trust

• Lab members trust that the PI has their best interests in mind
• PI works to ensure diversity, equity, and inclusiveness
• Lab members are expliclty encouraged to report any errors they discover in their own work
• All work is discussed openly in the lab, and lab members are encouraged to collaborate
• Lab members are expected to to post their code and data openly upon preprint submission
• Lab members are given space to do good work, and are not pressured to publish before the work is well validated

How can we move from closed to open science culture?

https://www.cos.io/blog/strategy-for-culture-change

Making open science normative

“Incentives drive behavior, and behavior spawns culture.” - Rob Asghar

Aligning the incentives

• Researchers should be rewarded for doing the right thing

• Problem:
  • Who makes the decisions about hiring, tenure, and funding?

Mutually reinforcing vectors for change

https://www.heliosopen.org/about

Increasing high-level support for open science

Translating open science into institutional policy

Higher Education Leadership Initiative on Open Scholarship

https://heliosopen.org

• At a high level, engagement from colleges and universities has three core components:
  • Presidential Commitment
  • Campus Engagement
  • Community of Practice

Institutional change will take time

• What can we do on our own in the meantime?
• Two case studies:
  • 1: Changing norms around errors
  • 2: Building infrastructure for collaborative software development

Case study #1: Changing norms around errors

• No human enterprise is free from errors
  • E.g. professional software developers make 1-50 errors per 1000 lines of code
• Rather than viewing errors as a sign of incompetence, we should view them as teachable moments

Normalizing the discussion of errors

https://russpoldrack.blogspot.com/2013/02/anatomy-of-coding-error.html

Normalizing the discussion of errors

https://psyarxiv.com/rsn5y/

Errors as a teachable moment: The Morbidity and Mortality Conference as a model

• Aims to identify the root causes of poor outcomes or near-misses
• Focuses on the system, with no blame or finger-pointing

https://hawaiiresidency.org/ob-gyn-residency/morbidity-and-mortality-m-m

Finding and fixing errors early

Finding and fixing errors early

We had posted a preprint describing some issues that we had identified with the stop-signal task in the ABCD Study, along with the code used for all of the analyses. The ABCD stop-signal team performed a detailed review our code and notified us of an error in the code that resulted in inaccurate estimation of one of the basic behavioral measures on the task (subsequently described in their response to our preprint).

Finding and fixing errors early

Root cause analysis

1. Flawed code review process

• The person who initially reviewed the code focused on the analysis code, rather than the preprocessing code where the error occurred

2. Time pressure

• We were pushing to complete the work quickly, and our speed-accuracy tradeoff was not as focused on accuracy as it should have been

Case study #2: Infrastructure for collaboration across labs

• There are many labs developing tools for neuroimaging analysis
  • Most of these groups write code to solve the same problem, duplicating effort
• We could reduce the wasted time and effort by working together

NMIND: Nevermind, this Method is Not Duplicated

Greg Kiar, Child Mind Institute

http://nmind.org

http://nmind.org

Conclusion

• We must all work to change the incentive structures of science
• We can all start now to establish the practices that will give rise to a culture of open and reproducible science

What are the norms that we want to encourage?

• Intellectual humility
• Community over competition
• Interpersonal respect and trust
• Openness and transparency

• ackonwledge the biases, limitations, and constraints on generalizability of our work
• Work together rather than competing
• Respect the contributions that each person can make, and embrace diversity of the community
• Share data, code, and ideas openly