Contents

Pre-registration

What is pre-registration?

Pre-registration is the process of documenting one’s plans for a study and depositing those plans in a place where they are time-stamped and immutable (i.e. cannot be changed). There are various types of pre-registration:

  • Standard pre-registration: An investigator documents their plans in writing and submits them to a pre-registration service. This documents the researcher’s plans prior to undertaking the research, and provides investigators and reviewers with a way to distinguish a priori hypotheses from post-hoc exploratory analyses. The document may be kept private for some period of time, but is usually made public upon submission of the manuscript for publication.

  • Registered reports: An investigator writes a manuscript describing the motivation for a study and a detailed description of the methods, and submits it to a journal for peer review prior to undertaking the research. The manuscript is reviewed based on the importance of the research question and the quality of the methods. If accepted, the journal agrees to publish the paper regardless of the results, assuming that there are no problems with the implementation of the methods.

  • Registered replication report: A type of registered report in which the invesigators wish to attempt to replicate a particular published finding, usually involving multiple research sites.

Why is pre-registration important?

  • It forces the researcher to plan & think through both why and how they pursuing their research question question

  • It provides the researcher with a way to determine whether a hypothesis was truly held a priori, versus relying upon memory

  • It forces the reseacher to think through their analysis plan in more detail, potentially surfacing issues that could influence the design of the study

  • It helps prevent data-dependent decisions (including “p-hacking”) and establish credibility by avoiding the garden of forking paths: it’s a solution to p-hacking

  • Helps prevent selective reporting of measures

When can/should one pre-register their research?

A planned research activity can be pre-registered at any point, as long as the particular activity being registered has not started. However, there are several points at which registration is most common:

  • Prior to the collection of data for a project

  • After a reviewer has requested additional data collection or analyses on a manuscript

  • Prior to analysis of an existing or openly available dataset

Getting Started

The paper A Practical Guide for Transparency in Psychological Science by Klein and colleagues provides a good overview for the process of how to go about pre-registering a study:

Determine which repository to use

It’s best to use the repository that is most common in one’s particular field of study. Some options include:

Create your pre-registration document

In general, the more detailed the preregistration the better, but don’t let the perfect be the enemy of the good: Registering something is better than not registering anything.

For a psychological study, the Gold Standard would be to register:

  • sample size and inclusion/exclusion criteria

  • measures or stimuli to be used and their construction

  • specific predictions / hypotheses to be tested

  • description of preprocessing to be applied to the data

  • analyses to be performed, including description of statistical models and variables that will be included and the criteria for statistical decisions

In addition, one might rpre-egister all of the acquisition and analysis code to be used, and validate the analysis code on simulated data. This might be considered the “Platinum Standard”.

Submit the pre-registration

Submission often requires specifying an embargo period, during which time the pre-registration is private until the set expiration date. Reasons to embargo include:

  • Prevent others from copying your idea (if your study will take some time to carry out)

  • Prevent participants from seeing the purpose of the study (particularly relevant in field studies).

The maximum embargo period is generally at most 3 years after initial submission.

Frequently Asked Questions

What’s the difference between a study registry and a pre-registration?

A study registry is similar to a pre-registration registry, but with a slightly different goal. The goal of a study registry is to keep a record of all studies that are run, in order to address the file drawer problem, in which studies with null results (or results that contradict the researcher’s predictions) may go unreported. All clinical trials funded by the NIH are expected to register their study and report their results using the ClinicalTrials.gov registry. Study registries generally include the same information that is included in a pre-registration, and thus can be used for the same purpose as pre-registration in some cases.

What’s the difference between a pre-registration and a pre-analysis plan?

A pre-registration lists your study design, measures, hypotheses, exclusion criteria, planned analyses, etc. A “pre-analysis plan” is a term more often used in economics and includes these features of a pre-registration but also often provides mathematical specifications of all models and adjustments that will be run.

How detailed should your pre-registration be?

Depends on the research stage. Earlier stage projects could be simpler pre-registrations and later stage projects could include more details. Recently scholars have argued (here, here) that pre-regs should highlight the basic study design and primary outcomes, rather than all contingencies, for more early stage research projects.

If you’re just getting started, it may be helpful to start with simpler templates (example here). As you get more comfortable, you can move towards templates that may require more detail (see many different template examples here).

What kind of studies can I pre-register?

Every study can (and should) be pre-registered! This includes observational studies, meta-analyses, and secondary data analyses!

Do you pre-register projects that use secondary data?

Ideally, you will want to pre-register your project before you conduct any analyses, but this is not always possible. If you are working on a secondary data analysis, it’s possible that some or all parts of the data you are working on have already been published. Or, it is possible that different parts of the data are a part of studies in-progress and you are aware of certain patterns (e.g., they are discussed in lab meetings). What is important is that you are able to draw some boundaries at a particular point in time so that you are aware of what is a part of confirmatory versus exploratory research. In your pre-registration, you can be transparent about what you know when you prepare your analyses. It is important to consider that maybe a particular project may not be well-suited to a pre-registration, if you know you are already aware of too much. However, we encourage you to still draw up an analysis plan that can still be an internal guide for analyses and keep all project members aligned for primary versus exploratory analyses.

Do you pre-register pilots?

You can still pre-register a pilot: “If setting out on purely exploratory research or pilot studies, preregistration can still help you remember that intention at the end of that project, improve the transparency of your research” link

Do you pre-register exploratory analyses?

One primary goal of pre-registration is to distinguish confirmatory from exploratory analyses. Confirmatory analyses should be described in detail. Note that sometimes people distinguish between primary and secondary confirmatory analyses, although the term “secondary” also can be used to refer to “exploratory” analyses.

There are different approaches to registering exploratory analyses.

  1. Some people register non-confirmatory analyses as exploratory in their pre-registration. Sometimes people will specify the analyses and hypotheses, although this is not necessary.

  2. Some people don’t register exploratory analyses and then choose to report them as exploratory in their manuscript, if relevant.

  3. You can also carry out a split-sample strategy where you pre-register the fact that you will carry out a split-sample strategy and how the split will be done, then you draw your initial random subselection of your data to conduct exploratory analyses on, then you register your confirmatory hypotheses and analyses online that you will carry out on your held-out validation set, and, finally, you run those confirmatory analyses (see papers below).

Regardless of approach, in writing up exploratory analyses, it should be made clear which analyses are confirmatory and which are exploratory.

How to have flexibility where you need it?

  1. You could give contingency plans (“if-then” statements). For example, if you are assessing a scale based on a Western population in a new population in a different country, you could note “For all composite variables of 3 or more items, we will remove variables that have loading values below 0.3 on the first factor.”

  2. If you are conducting research in a new population or otherwise have lots of uncertainty, you can conduct exploratory analyses on a portion of the sample and then register confirmatory analyses on the remaining data. These papers below detail this “split-sample” strategy:

  3. As discussed above, it is common for studies to end up deviating in some way from their pre-registration. Your job as a researcher is to be as transparent as possible about these deviations. (see below)

How do you change / revise your registration?

Once you publish your pre-registration you cannot make changes to the document. However, you can create a new pre-registration and include a link to your older one, outlining the changes you’ve made.

  • “…you are free and encouraged to make a new pre-registration up to the point of data analysis. We recommend that you preserve the original preregistration, include a link to it in the new preregistration, and specify the changes that you are making and why.” - from Preregistration: A plan, not a prison

  • “The preregistration for an exploratory study might be brief, but it would nonetheless have value as an explicit record of which if any aspects of the study were specified a priori. That information is especially helpful in research that emphasizes null hypothesis significance testing. A thorough preregistration promotes transparency and openness and protects researchers from suspicions of p-hacking.” - from APS

How to report deviations of the pre-registered plan after data collection?

Note that deviations of the pre-registered data are common. You should report clearly why you deviated in the manuscript. You can also post a list of deviations on your pre-registration or project page. https://www.psychologicalscience.org/observer/research-preregistration-101

How to organize your OSF page?

On OSF, should I create separate components per pre-registration?

  • It’s up to you. You could create a separate project per study. Or you could create one project for an entire study. See our Data Sharing and Code Sharing resources for more on project organization.

A guide to open science for psychologists Code Sharing