How do you assess the reliability and performance of a peer review contractor? Before trusting an organization with responsibility for proposal evaluations or program reviews, check their track record by asking questions that address the core elements of quality reviews. That’s a start.
This post lists the essential elements that can spell disaster for research grant managers and how to ensure your partner does not let you down on any of them—before you sign on the dotted line. Our advice? Don’t even start working with any company that can’t show a positive track record and proven solutions to the following fundamental factors.
The essential five requirements for peer review contractors:
1. Protect review integrity
Three major negative factors that can damage review integrity are conflict of interest (COI), failure to protect data, and reviewer bias. The first can be addressed by designing a well thought out software application for handling reviewer input and data. A sound review system will include initial and secondary checks for COI and include “fail safe” measures that prevent reviewer access to data once a COI has been identified. Using software that includes state-of-the-art data encryption/protection is essential. Proprietary systems (developed by the contractor, not off-the-shelf) add significant protection by making the programming code tougher for hackers to access.
When bias is discovered after the reviewer input is collected, it will cause considerable re-work to find another expert. Preventing bias from threatening review integrity requires communication combined with well-constructed policy. How are experts vetted and selected? What are the specific methods used to prevent bias? Bibliographic research will often identify trends that may indicate the potential for bias. Taking an extra step to communicate concerns quickly and cooperatively investigating potential bias can prevent trouble once the review begins.
2. Use criteria and evaluation process that match agency goals
A value-added partner will be able to describe options for conducting proposal evaluations and program reviews and explain the trade-offs between them. For instance, using NSF’s criteria works great for basic exploratory research and supports direct program manager input into funding decisions. See NSF’s Merit Review Facts for discussion on NSF program officers making recommendations to fund or decline a proposal as well as being encouraged to recommend high risk science and engineering projects for funding. Other organizations prefer to award funds strictly on the basis of reviewers’ scores.
3. Carefully design the review panel
An astute review manager adds value by configuring a powerful review panel. While the first instinct is to find and secure the services of the most eminently qualified individuals available, there are other considerations that can complicate the task. Adding less-celebrated experts who are doing cutting-edge work or others who do not reflect mainstream thinking on the particular research thrust can stimulate more insightful evaluation. And there’s the big picture to consider – bringing in less experienced scientists builds and spreads the knowledge base relevant to the sponsoring agency’s mission as well as preparing the next generation of reviewers.
This complexity can challenge contractors who are not fully committed to identifying and attracting a broad spectrum of experts to serve as reviewers. Asking a potential partner to explain how they find experts and who does that work (do they use technically qualified staff?) will reveal a great deal about how seriously they regard this important task. This isn’t standard human resources recruitment. See ORAU’s Subject Matter Expert Recruiting page to learn more about the ORAU approach.
4. Use strong data quality methods and design reports for their specific use
These two factors go together. In any review with multiple panelists and criteria, recording and compiling scores should be automated. Then checked. Then maybe checked again. And the automated system should be reviewed periodically to ensure it is recording and outputting what was expected. There are literally thousands of ways data can be misrepresented or recorded in a way that is misleading. A sound data quality assurance and control system is essential.
Have you ever seen a review manager working with multiple complicated spreadsheets, trying to figure out the final ranking of proposals? Have you ever done that? There’s a better way. An experienced review manager, working with a custom-designed software tool such as ORAU’s PeerNet, should be able to efficiently design reports that allow decision-makers to quickly understand the ratings and rankings from panelists and drill into details as they wish.
5. Maintain control of review operations
An experienced peer review expert once commented “A review is made up of thousands of tiny details that all have to occur in a specific sequence.” That requires a strong management approach. And to make it even tougher, there is no school or other training that will prepare an individual for the job. If you are purchasing peer review support, perhaps the most important factors are the management system used and the experience and qualifications of the review manager.
There are two critical elements that can be used to determine a potential partner’s ability to handle complex peer reviews. First, do they have a defined and cogent approach to the work, including documented processes, a robust quality assurance program, and the necessary software tools? Second, are their review managers specifically trained for that job, including performance testing and observation by senior management, perhaps even certification as project managers?
In the U.S., there are only a few truly qualified private organizations with a proven track record of managing independent peer reviews across multiple disciplines. Among those, even fewer have invested the resources to develop a comprehensive peer review support capability that includes a specific management approach, qualified staff, proven work processes, and a custom-develop set of software tools. Contact an ORAU peer review manager to discuss how we can work with you to make your reviews the best they can be. You can contact us at peerreview@orau.org or call (865) 241-7544.