Peer Review Bias

Evidence Review: Peer review bias in the funding process. Main themes and interventions

Authors: Dr Stefanie Schneider, Dr Cat Morgan, Dr Marion Hersh, Dr Clayton Magill, Professor Kate Sang and Professor Robert MacIntosh

Click here to read our opinion piece in Research Professional News

Click here to download the Evidence Review

This study examines recent literature about the impact of bias in reviewing and assessing processes for research funding from complementary perspectives, such as applicant, reviewer and decision-maker. A further aim is to explore strategies that have been used to improve the processes, whether they have been successful or not. The study’s rationale is to improve understanding of existing biases, their impact and gaps in knowledge and practice.

The interventions in this report aim to combat bias and have been designed from our examination of the literature on bias in the peer review process for funding and wider reading of journal guidance documents and publishing processes. These can be applied across disciplines for various funders. EDICa suggests that any interventions introduced by funders are monitored to determine their success and that they should be openly reported on.

Research question: What reduces bias in the peer review process?

EDICa is a national project which aims to bring an intersectional perspective to the multiple disadvantages which can face marginalised researchers and innovators. EDICa’s vision is to create inclusive research cultures enabling diverse researchers to access and thrive in careers across the research and innovation (R&I) systems, removing barriers to full participation experienced by women, disabled, LGBTQI and racially minoritized researchers, and researchers with caring responsibilities.

Creating inclusive research culture relies on practical, actionable and evidence-based insights that support diversity of individuals in key aspects of their careers. In this sense, an under-explored set of issues pertains to the processes of applying for and evaluating funding applications for research funds and the related processes of seeking publication in peer-reviewed outlets.

Executive Summary

• 44% of the literature focuses on gender bias in the peer review process, half of which are from STEM.
• 18% of the literature concentrates on racial inequity.
• 14% of the literature examines institutional prestige.
• Bias impacts the funding process throughout various stages:
  • The funding call,
  • Preparing applications,
  • Preparing the review stage,
  • The review stage, and
  • Funding outcome.
• Groups of scholars affected by bias in the funding process are:
  • Women,
  • Early career researchers,
  • Disabled researchers,
  • Racially minoritized researchers, and
  • Researchers from less prestigious institutions.
• Internal barriers within institutions lead to fewer submissions for funding from these underrepresented groups.
• There is little evaluation of interventions that funding councils used to counter biases, the reasoning behind them or their impact.
• Funders often rely on scholars’ readiness to objectively evaluate their own bias and willingness to be objective.
• Advice or training to establish one’s own views and/or to overcome harmful or self-reinforcing outcomes is scarce.

Most funder guidance documents request that reviewers don’t consider journal-based metrics as the only means to measure impact.

Conclusion

This evidence review on peer review bias in the funding process set out to document evidence for both obstacles within the funding process as well as for interventions to overcome those obstacles. The literature that documents obstacles for women, early career and ethnic minority researchers is relatively rich, with gender being the most prominent category we found in our corpus of literature outputs. Women’s experiences as applicants as well the impact of their increasing inclusion as reviewers and assessors has been discussed across all stages of the funding process. Obstacles for early career and ethnic minority researchers as applicants as well as calls for their inclusion as reviewers were also found, but to a much lesser extent. We found very little evidence for experiences of disabled researchers and no accounts for any of the other protected characteristics of the Equality Act (2010).

We also found very little evidence for the effectiveness of any proposed interventions. The evidence review shows that all councils plan to play active roles in changing the funding process to make it more equitable, which is a positive finding. The initiatives are, however, still in their early stages, and all still require testing and refining. One of the first steps that has been taken is the collection and annual publication of EDI data on who receives funding from UKRI councils. This is in response to calls for more transparency, but our evidence review shows that this data only offers a partial picture of experiences in the funding process. Looking beyond established categories and investigating what data has so far been overlooked is a crucial first step to make the funding process truly inclusive.

As shown on the heatmap in Appendix A of our report, most research focuses on the final stages of the funding process. There are large gaps in evidence for processes that precede the submission of a funding bid. This includes multiple factors that enable applicants to prepare a competitive bid for large-scale funding, like the impact of professional networks that can advise or collaborate and internal funding bid review processes. There is also little evidence on the preparation of the review process by funders. While there are efforts to make panels more diverse, there is no evidence on effective EDI training methods for panel members and chairs. More research is needed to understand obstacles and enablers for marginalised scholars in every stage of the funding process. This evidence review is a work in progress and will add new evidence and results as the EDICa project continues and seeks to co-design interventions that remove barriers outlined in this review.

---