Reviewing Internal Peer Review of Grant Applications, Part 2: How to make it work better

A version of this article first appeared in Funding Insight in April 2019 and is reproduced with kind permission of Research Professional. For more articles like this, visit www.researchprofessional.com

Most universities have internal peer review processes for grant applications. In part one I discussed the different purposes of internal peer review and how they can cause confusion. I also wrote about how to ensure that we present internal peer review as helpful and supportive rather than a hurdle to be overcome. In this second and final part, I’m going to look at how we might do internal peer review of grant applications better.

Who do we ask to review?

The ideal reviewer is a senior academic with a track record of success with major research funding applications and some insight into the subject area. Even at research-intensive institutions, there is a limited supply and their time is valuable. Especially for reviewers in development-related topics because of the volume of Global Challenges Research Fund (GCRF) bids. [Alas, this example from 2019 has dated poorly.] Our instinct is to ask senior Profs, but I wonder if a closer review by someone less senior could be more useful. We should think beyond the usual suspects, as reviewing can be a developmental exercise. My experience has been that researchers who are rarely asked to review often throw themselves into the task with a lot more enthusiasm. They’re often delighted to be asked, and keen to do a good job.

Do we make internal peer review anonymous?

This is tricky. In my view – ideally – no. Being able to put feedback in the context of the reviewer’s background can be very valuable. I also think that people should be willing to stand behind their comments.

However, because internal peer review can have a filtering role, perhaps the protection of anonymity is required for reviewers to be willing to say that proposals shouldn’t go forward. Or perhaps even to be willing to criticise colleagues’ work at all. However, I would expect that the rationale for soft filtering out an application should be one that most applicants would accept and understand. For a hard filter – when only x number of applications can go forward from the institution – there would usually be a committee decision bound by collective responsibility. I’m not aware of any research or internal survey work done on internal peer reviewers and their attitudes to anonymization, and I’d be interested to see if anyone has looked at this.

How do we ask reviewers to review?

It’s not obvious how to review a grant application. Those without much experience may be reluctant to trust their instincts or judgement because “it’s not really my area”. A small number go the other way and go power crazy at the chance to sit in judgement – judging the proposal from their own personal, partisan perspective and completely write off entire academic disciplines and sub-disciplines.

One option is to ask reviewers to use the same form that the funder in question gives to referees or panel members. It’s a great idea in principle, but academics typically have a loathe-hate relationship with forms. But there are some specific questions we could ask reviewers in a structured way, or use as prompts. Fewer questions will get better answers.

What isn’t clear? What’s confusing or ambiguous?
What are the potential weaknesses?
What’s missing?
How could the application be improved?

If I were to ask a single question, it would be the pre-mortem.

If I could see into the future and tell you now that this application is not going to be funded, what will be the main reason?

This question helps home in on key weaknesses – it might be fit-to-call, it might be unclear methodology; it might be weak impact pathways; it might be the composition of the research team. It’s a good question for applicants to ask themselves.

How do we feed back?

It’s not enough for feedback to be correct, it must be presented in a way that maximises the chances that the PI will listen.

Ideally, I’d like a face-to-face meeting involving the internal reviewers, the Research Development Manager, the PI and possibly the co-investigators. The meeting would be a discussion of a full draft in which reviewers can offer their views and advice and the PI can respond, and ask questions about their impressions of the proposal. I like face-to-face meetings because of the feedback multiplier effect – one reviewer makes an observation, which triggers another in the second reviewer. A PI response to a particular point triggers a further observation or suggestion. If approached in the right spirit (and if well-chaired) this should be a constructive and supportive meeting aimed at maximising the applicant’s chances of success. It must not be a Dragon’s Den style ordeal.

In reality, with packed diaries and short notice calls, it’s going to be difficult to arrange such meetings. So we often have to default to email, which needs a lot of care, as nuance of tone and meaning can be lost. I would advise that feedback is sent through an intermediary – another task for your friendly neighbourhood research development manager – who can think about how to pass it on. Whether to forward it verbatim, add context or comments, or smooth off some abrasive edges. I’ve had a reviewer email me to say that she’s really busy and could I repackage her comments for forwarding? Happy to.

A good approach is to depersonalise the applicant – address the feedback to the draft application, not its authors. (“The current draft could be clearer on….” versus “You could be clearer on…”). But I think depersonalising the reviewers and their comments is a mistake – impersonal, formal language can come over as officious, high handed, and passive aggressive. It will make applicants less likely to engage, even if the advice is solid. Using (even rhetorical) questions rather than blunt statements invites engagement and reflection, rather than passing final judgement.

Which would you respond to best?

The panel’s view is that your summary section is poor and is an introduction to the topic, not a proper summary of your whole project. You should rewrite before submitting.

Or….

Could the summary be strengthened? We thought the draft version read more like an introduction to the topic, and we think reviewers are looking for a summary of the complete proposal in a nutshell. Is there time to revisit this section so it better summarises the project as a whole?

Institutions invest time and money in having arrangements that provide prospective PIs with detailed feedback from senior academic colleagues to improve their chances of success. But it’s all for nothing if the resulting advice is ineffective because of the way the feedback is communicated, or the way the whole process is presented or perceived by researchers.