A healthy portion of food for thought has been served up by the publication of a RAND Europe report into alternatives to peer review for research project funding. Peer review is something that I – as an alleged research funding professional -have rather taken for granted as being the natural and obvious way to allocate (increasingly) scarce resources. How do we decide who gets funded? Well, let’s ask experts to report, and then make a judgement based upon what those experts say. I’ve been aware of other ways, but I’ve not given them much thought – I’m a poacher, not a gamekeeper.
The Guardian Higher Education Network ran a poll over the second half of last week, and a whopping 70.8% of those who voted said that they had had a research proposal turned down thought the process should be changed. I’m aware of the limitations of peer review -it’s only as good as the peers, and the effort they’re prepared to make and the care they’re prepared to take with their review. Anyone who has had any involvement in research funding will be aware of examples where comments come back that are frankly baffling: drawing odd conclusions, obsessing over irrelevancies, wanting the research to be about something else, making unsupported statements, or assertions that are just demonstrably false.
[Personally, I hate it when ‘Reviewer Q’ remarks that the project “seems expensive”, without further comment or justification about what’s too expensive. That’s our carefully crafted budget you’re talking about there, Reviewer Q. It’s meticulously pedantic, and pedantically meticulous. We’ve Justified our Resources… so how about you justify your comment? I wonder how annoyed I’d get if I wrote the whole application…..]
One commentator on the Guardian poll page, dianthusmed, said that
Anyone voting to change the peer review process, I will not take you seriously unless you tell me what you’d replace it with.
And that’s surely the $64,000 question (at 80% fEC)…. we’re all more or less familiar with the potential shortcomings of peer review as a method of allocating funding, but if not peer review… then what?
In fact, the Rand Europe report is not an anti peer-review polemic, and deserves a more nuanced response than a “peer review: yes or no” on-line poll. The only sensible answer, surely, is: well, it depends what you want to achieve. The report itself aims to
inspire thinking amongst research funders by showing how the research funding review process can be changed, and to give funders the confidence to try novel methods by explaining where and how such approaches have been used previously.
But crucially…
This is not intended to replace peer review, which remains the best method for review of grant applications in many situations. Rather, we hope that by considering some of the alternatives to peer review, where appropriate, research funders will be able to support a wider portfolio of projects, leading to more innovative, high-impact work.
A number of the options in the report seem to be more related to changing the nature and scope for calls for proposals than changing the nature of peer review itself – many in ways that aren’t unfamiliar. But I’d like to pick out one idea for particular comment: sand pits.
I believe the origin of the term is from computing, where the term ‘sand box’ or ‘sand pit’ was used to describe an area for experimentation or testing, where no damage could be done to the overall system architecture. I guess the notion of harmless – even playful – experimentation is what advocates have in mind.
They sound like a very interesting idea – get a group of people with expertise to bring to bear on a particular problem, put them all in same place for a day, or a number of days, and see what emerges from discussions. It’s not really caught on yet in the social sciences, although social scientists have been involved, of course. The notion of cooperating rather than competing, and of new research collaborations forming, is an interesting and an appealing idea. As a way of bringing new perspectives to bear on a particular problem – especially an interdisciplinary problem – it looks like an attractive alternative.
There are problems, though. If there are more applications to participate than there are places, there will inevitably need to be choices made and applications accepted and rejected. I would imagine that questions of fit and balance would be relevant as well as questions of experience and expertise, but someone or some group of people will have to make choices. From the application forms I’ve seen, this is often on the basis of short CV and a short statement. So… don’t we end up relying on some element of peer review anyway?
Secondly, I wonder about equal opportunities. If a sand pit event is to take place over several days in a hotel, it will inevitably be difficult or even impossible for some to attend. Those who are parents and/or carers. Those who have timetabled lectures and tutorials. Those who have other professional or personal diary commitments that just can’t be moved. For a standard peer reviewed call, no-one is excluded completely because it clashes with an important family event. Can we be sure that all of the best researchers will even apply?
I should say that I’ve never attended a sandpit event, but I have attended graduate recruitment/selection events (offered, deferred, and finally declined, since you ask), and residential training courses. They’re all strange situations where both competitive and cooperative behaviours are rewarded, and I wonder how people react. If I were a funder, I’d be worried that the prizes might be going to the best social operators, rather than those with the best ideas. It’s a myth that academic brilliance is always found in inverse proportion to social skills, of course, but even so, my concern would be about whether one or more dominant figures could ending up forming projects around themselves. I also wonder about existing cliques or vested interests of whatever kind having a disproportionate influence.
I’m sure that effective facilitation and chairing can go a long way to minimising at least some of the potential problems, and while I think sandpits are an intriguing and promising alternative to peer review, they’re not without problems of their own. I’d be very interested to hear from anyone who’s attended a sandpit – am I doing them a disservice here?
Although I’m open to other ideas for distributing research funding – by all means, let’s be creative, and let’s look at alternatives – I don’t see a replacement for peer review. Which isn’t to say that there isn’t scope to improve the quality of peer review. Because, Reviewer Q, there certainly is.
And perhaps that’s the point that the 70.8% were trying to make.
