What is peer review?

As someone who has not worked in research or with grants, I only had a vague understanding of peer review and the value it offers. It’s something that’s a big deal around ORAU, so I wanted to know more about what it is and how it works. To answer my questions, I talked with Justin Townsend, an associate manager in ORAU’s peer review group. Since he’s the expert, I thought I’d let his words shape this blog through our Q & A.

Q: How long have you been with ORAU?

A: I started at ORAU in September of 2015, so I’ve been here more than eight years now. I have been in the Peer Review group for all eight years.

Q: Big picture: what is peer review?

A: Peer review is an assessment of research or proposed research by experts in that specific field.

Q: Who are the players in peer review?

A: With what we do, there are typically three types of people involved: the customers, the principal investigators and the reviewers.

• Customer - The person or group we work with to facilitate the review. These people are looking for expert opinion and feedback. It could be that they want expert advice on the best candidates to award grant funding, or it could be the work is their own and they want to ensure there aren’t holes in their research.
• Principal investigator - The applicant, the person or group who is behind the work/research that is under review.
• Reviewers - This is the panel of subject matter experts we identify and select to provide input and scoring of the submitted material. Reviewers are not paid for their time. They’re contributing back to the community through their review services. Some reviewers are given honorarium, which is a small token of appreciation for their review effort.

Q: How is ORAU involved in peer review?

A: ORAU facilitates grant funding peer reviews from beginning to end. We use PeerNet, ORAU’s proprietary and easy-to-navigate web-based application. It is a full grants management system, meaning PeerNet facilitates proposal application intake, peer review AND grant awarding. It even houses our subject matter expert database. When we receive proposals, our research services group reads the content, locates and then secures experts to officially review the submitted proposals. Once all reviews are completed, we compile results and create reports to send to the customers for them to make an educated decision on their next steps.

Q: Why is peer review important?

A: Peer review enables our customers to make an educated decision on which proposals should be funded with the budget they have available. They get to stay out of the review process, which helps eliminate customer suggestion bias. Suggestion bias can happen when a researcher unconsciously influences the participants of a study. For example, the researcher may ask leading questions.

Also, it’s important to equip our customers with information. The customers enter the material in PeerNet for feedback. Then, they make decisions based on the reviewers’ feedback and are responsible for responding to the principal investigators (the researchers) who applied for funding. With comments and scores provided by some of the world’s best subject matter experts, the customer is able to explain why the principal investigator did or did not receive the grant they were seeking. As a government contractor, this helps us assure our customers that they can make sound and defensible funding decisions that based on a review process that ensures the utmost integrity.

Q: Do we make effort to include experts with differing professional opinions?

A: In a lot of our reviews, the reviewers never know what the other reviewers have written or see their evaluation. When the work calls for us to facilitate the reviewers getting together for a panelist review, we let them know that we are not asking them to reach a consensus. Each reviewer is free to share his or her own expert opinion. So, we typically see very differing perspectives on these reviews.

Q: Does ORAU focus specifically on scientific subject matter, or do we receive requests for peer review from all disciplines?

A: Our processes and PeerNet allows us flexibility to review various customer requests—award nominations, grant applications, project reviews and more. While most of the proposals we receive are medical in nature, we also see a lot of review requests connected to the National Science Foundation (NSF) and even private industry requests.

Q: What’s a memorable example of a proposal we’ve facilitated?

A: One of the first examples that comes to mind is the Florida Department of Health requested a review of the “Live Like Bella” pediatric cancer research initiative. That’s an on-going grant they offer. The goal of the received proposals is to expand Florida’s pediatric cancer research capacity, improve both research and treatment through greater pediatric enrollment in clinical trials and to reduce the impact of pediatric cancer on disparate groups. Vetting work like this ensures that strides will be made toward true impact. We love being a part of that.

Q: How long does the peer review process take for an average (if there is such a thing) submitted work or proposal?

A: The timeframe depends on the content. For most limited-submission customers (meaning the customer only allowed a set small number of proposals), we typically turn over our reports to the customer in a couple weeks. For larger-scale proposals (like the open calls we see with the Florida Department of Health and annual grant opportunities like “Live Like Bella”), we may work with those customers for months as we go through the application intake process, recruiting reviewers and providing final review reports.

Q: Does the peer review process happen remotely, or do we pull the team together?

A: It can be either. Most of our reviews are remote and the reviewers do not see or talk to each other, but for some of our work, we will have a panel of reviewers get together to discuss the proposals in person and/or virtually.