A version of this article first appeared in Funding Insight in November 2021 and is reproduced with kind permission of Research Professional. For more articles like this, visit www.researchprofessional.com
When it comes to research funding, I’ve been poacher and gamekeeper. I’m a research development manager, but I’ve also served two terms as a public member of a research funding panel, and I still review the odd proposal. I’m going to draw upon that experience to try to explain how funding panels tend to work. They obviously vary depending on funder and scheme. So treat this article as based on a true story, but with certain scenes invented for dramatic purposes.
Once upon a time, in a hotel conference room…
I am one of about twenty-five other people sitting behind name plates and laptops round the outside of three sides of a horseshoe of tables. On the fourth side is a large screen. Most attendees are academic experts, drawn from a broad range of disciplines, to cover the full scheme remit. There’s no such thing as a perfect panel, of course. Applications may still benefit or suffer through a lack of expertise, or through there being a panel member who is an expert in precisely the topic addressed by the application.
There are a small number of non-academic reviewers. Also present are staff from the funder who will drive the PowerPoint, make sure the meeting runs smoothly, and remind panel members about the scheme rules, allowing the chair to focus on the business of the meeting. The chair has a very important role, and the panel chairs I’ve worked with are more like facilitators who guide discussions and focus minds rather than impose their will. Nevertheless, the chairs I’ve worked with have been very influential, but that’s because they’re nearly always right.
We’ll be here all day, having already spent several days preparing. Everyone serving on a funding panel takes their role and responsibility very seriously. It’s a lot of work to read and engage with every bid. In four years of doing this, I’ve never seen anyone under-prepared.
Each panel member has been assigned a few applications to introduce to the panel. The lead introducer is usually the panel member whose expertise is closest to the proposal. Her role is to summarise the application in five minutes or less. If it’s a full application, this will include comments at outline stage, the applicants’ response, and expert peer reviewers’ comments.
Expert peer review comments are invaluable, and in practice we won’t fund anything that none of the reviewers’ support. However, most sets of reviews are mixed: the most frustrating was when we had one good, one bad, one mediocre. I’d say that for around half of applications, reviewers’ comments contain at least one comment that’s based on an obvious misunderstanding. The introducer will mention it, and the panel will disregard it. Hostile or aggressive negative reviews are rare, and seldom influence panel members, who often disregard them entirely.
After her summary, the lead introducer gives the panel her own opinion in terms of strengths and weaknesses and score. Then two other introducers—including me or one of the other public representatives—adds anything that’s been missed or overlooked. We have a couple of minutes each and are told not to repeat previous points. I’ll comment from a public perspective. We’ll then add our scores, though all introducers can change their scores after the discussion.
The introducers set the tone for the discussion. We don’t have long to discuss each application—perhaps twenty minutes total for a full proposal. Discussions are civil and disagreements polite. We all have the same goal—to fund the best proposals—and there is scope for legitimate disagreement.
One phenomenon I have noted in the past was what I call a ‘criticism cascade’. Or, if you prefer, ‘the Statler and Waldorf’. Discussions start out positive, only for one critical comment to lead to another and another until the application’s fate is sealed. If that first criticism hadn’t been made, would the others be prompted? Or is it inevitable that someone will speak first, and then it all unravels? Are all potential criticism cascades always triggered, or do some get away with it?
After discussion, we submit our scores individually and electronically, and they appear (anonymised) on the PowerPoint. We have a standard scoring system, but more importantly, the panel has a shared understanding of what a six, a seven, or an eight should be. New members get the hang of this quickly. Scores from peer-review panel members used the same scale, but without the calibration, were less useful. When this is done, we move onto the next proposal.
At the end of the meeting, we’re shown an overall ranking, and where the funding cut-off is. The chair asks if we’re happy, especially if it is very close and/or the ranking does not seem to be reflective of the discussions. This is important because our role is not to assign grades to applications, but to rank them in priority order. We’re aware of the dangers of getting more critical as we go on, or being too positive or negative about the first application before we’re properly calibrated. Or being grumpy with the last application before lunch, or lax with the first one after it. Sometimes we’ll re-score.
Some funding decisions are very marginal between applications that are very different and thus very hard to compare. There are more high-quality proposals than there is funding. Many excellent projects go unfunded and there’s nothing panel members can do about it.
With outline applications, we’re only going to invite those that we think have a realistic chance of competing for funding – our chair’s usual tie-breaker question is “do we want to see this again?”
The moral of the story
What can we learn from this classic tale? Well, as with all great narratives, there’s more than one takeaway. In this case, there are three main pointers for what you should do as an applicant. You should:
Be confident that everyone is doing their best. I don’t think anyone would deny that there are elements of luck involved at all stages, but luck is only a factor for high quality, competitive applications. If your proposal is flawed, vague, unclear, or offers only marginal benefit, no amount of luck will help.
Make life as easy as possible for panel members, and particular for your introducers. It’s very difficult for anyone—no matter how intelligent and motivated—to hold so much information about so many varied and complex proposals in their head at once. This means you must think very carefully about who you’re writing for—look to the sidebar for an article on that—and craft your proposal accordingly.
You should also make the significance of your proposal clear, overt, and explicit – don’t rely on reviewers to work it out. What’s more, ensure that the summary, which everyone re-reads prior to your application being discussed, is as clear as can be and your proposal is structured so panel members can find key information quickly. Again, articles on those two crucial elements are in the sidebar.
Apply or volunteer to serve on a funding panel. Whether for a major funder, or for seedcorn funding within your own institution. Often you don’t have to be very senior—balance on panels is really important. It’s a vital service for your research community, you’ll meet a lot of new and interesting people, and you’ll see first hand how decisions are made.