ORAU Associate Manager Justin Townsend explains what peer review is and how ORAU is involved in the process of ensuring research dollars are funded with the utmost integrity.
What is peer review?
Each year, federal agencies award billions of dollars to fund research proposals and projects that are determined to show the most promise. With many qualified contenders, how can these agencies effectively decide which proposals are the best? The most time-tested and widely accepted method is peer review. ORAU is an independent third party with decades of experience managing all phases of scientific peer and merit reviews. While managing a fully integrated peer review and grants management system, known as PeerNet, our peer review team is always looking for innovative solutions to improve on current processes and capabilities.
Enhancements to PeerNet
PeerNet, ORAU’s proprietary system, is specifically designed to facilitate effective peer reviews through an easy-to-navigate online portal. Short for Proposal Evaluation and Electronic Review Network, PeerNet is a structured system where data is delivered quickly to inform decision making, where review scores and comments from subject matter experts are entered, and where common criteria and scoring methods allow for reliable comparisons. Recently, the PeerNet development team made several enhancements to the core product, including
- a new integrated database that provides access to more than 50,000 subject matter experts for assignment to evaluations,
- a preview mode that allows peer reviewers to see collection and evaluation questions much earlier in the peer review process, and
- a modification that allows for downloading a copy of the evaluation in Microsoft Word to make working offline easier.
The team continuously improves PeerNet’s functionality based on feedback received from customers, reviewers and system administrators.
Researching how AI and machine learning can enhance peer review
With the evolving technologies of artificial intelligence (AI) and machine learning, ORAU’s peer review team began researching how these technologies could potentially translate to a more effective peer review process. The team authored a white paper on which aspects of the peer review recruitment and science writing work could be positively enhanced by AI. The team identified steps in their processes that would be amenable to AI solutions, providing brief context and identifying potential tools. Importantly, a section was also devoted to how AI has and could be used to address accessibility for all. A final section discussed ethical considerations as well as risks when implementing AI into systems and processes such as a peer review.
Developing early career scientists/researchers to become reviewers through learning modules
Developing early career scientists and researchers to become reviewers is critical to broadening future reviewer pools. ORAU’s peer review team is working to develop a reviewer education curriculum targeting early career scientists and others less experienced with peer review. The curriculum will provide resources to develop and/or improve skills related to evaluating grants, proposal applications and funded projects. Educational topics will range from general subjects like how to be a reviewer to more complex topics focused on reviewer integrity and fairness, protecting research integrity and other areas specific to scientific peer review processes. The curriculum will be developed in alignment with other federal agency peer reviewer guidance best practices.