Building a culture of open and reproducible science

Russ Poldrack

Stanford University

Culture: The norms, principles, and practices of an institution

Culture: The norms, principles, and practices of an institution

“Culture is created by the behaviors you tolerate” - Jacob Engel

Closed-science lab culture

  • Competitive atmosphere
  • High level of secrecy and paranoia
  • Pressure to find specific results
  • Lack of trust

Open-science lab culture

  • Openness and transparency
  • Collaborative atmosphere
  • Trust

How can we move from closed to open science culture?

Making open science normative

“Incentives drive behavior, and behavior spawns culture.” - Rob Asghar

Aligning the incentives

  • Researchers should be rewarded for doing the right thing
  • Problem:
    • Who makes the decisions about hiring, tenure, and funding?

Mutually reinforcing vectors for change

Increasing high-level support for open science

Translating open science into institutional policy

Higher Education Leadership Initiative on Open Scholarship

  • At a high level, engagement from colleges and universities has three core components:
    • Presidential Commitment
    • Campus Engagement
    • Community of Practice

Institutional change will take time

  • What can we do on our own in the meantime?
  • Two case studies:
    • 1: Changing norms around errors
    • 2: Building infrastructure for collaborative software development

Case study #1: Changing norms around errors

  • No human enterprise is free from errors
    • E.g. professional software developers make 1-50 errors per 1000 lines of code
  • Rather than viewing errors as a sign of incompetence, we should view them as teachable moments

Normalizing the discussion of errors

Normalizing the discussion of errors

Errors as a teachable moment: The Morbidity and Mortality Conference as a model

  • Aims to identify the root causes of poor outcomes or near-misses
  • Focuses on the system, with no blame or finger-pointing

Finding and fixing errors early

Finding and fixing errors early

We had posted a preprint describing some issues that we had identified with the stop-signal task in the ABCD Study, along with the code used for all of the analyses. The ABCD stop-signal team performed a detailed review our code and notified us of an error in the code that resulted in inaccurate estimation of one of the basic behavioral measures on the task (subsequently described in their response to our preprint).

Finding and fixing errors early

Root cause analysis

  1. Flawed code review process
  • The person who initially reviewed the code focused on the analysis code, rather than the preprocessing code where the error occurred
  1. Time pressure
  • We were pushing to complete the work quickly, and our speed-accuracy tradeoff was not as focused on accuracy as it should have been

Case study #2: Infrastructure for collaboration across labs

  • There are many labs developing tools for neuroimaging analysis
    • Most of these groups write code to solve the same problem, duplicating effort
  • We could reduce the wasted time and effort by working together

NMIND: Nevermind, this Method is Not Duplicated

Greg Kiar, Child Mind Institute


  • We must all work to change the incentive structures of science
  • We can all start now to establish the practices that will give rise to a culture of open and reproducible science

What are the norms that we want to encourage?

  • Intellectual humility
  • Community over competition
  • Interpersonal respect and trust
  • Openness and transparency