“It’s a bad review, we got a bad review …oh lord”

A picture of Clacton Pier
A large sandpit and a pier (re)view

A healthy portion of food for thought has been served up by the publication of a RAND Europe report into alternatives to peer review for research project funding.  Peer review is something that I – as an alleged research funding professional -have rather taken for granted as being the natural and obvious way to allocate (increasingly) scarce resources.  How do we decide who gets funded?  Well, let’s ask experts to report, and then make a judgement based upon what those experts say.  I’ve been aware of other ways, but I’ve not given them much thought – I’m a poacher, not a gamekeeper.

The Guardian Higher Education Network ran a poll over the second half of last week, and a whopping 70.8% of those who voted said that they had had a research proposal turned down  thought the process should be changed.  I’m aware of the limitations of peer review -it’s only as good as the peers, and the effort they’re prepared to make and the care they’re prepared to take with their review.  Anyone who has had any involvement in research funding will be aware of examples where comments come back that are frankly baffling: drawing odd conclusions, obsessing over irrelevancies, wanting the research to be about something else, making unsupported statements, or assertions that are just demonstrably false.

[Personally, I hate it when ‘Reviewer Q’ remarks that the project “seems expensive”, without further comment or justification about what’s too expensive.  That’s our carefully crafted budget you’re talking about there, Reviewer Q.  It’s meticulously pedantic, and pedantically meticulous.  We’ve Justified our Resources… so how about you justify your comment?  I wonder how annoyed I’d get if I wrote the whole application…..]

One commentator on the Guardian poll page, dianthusmed, said that

Anyone voting to change the peer review process, I will not take you seriously unless you tell me what you’d replace it with.

And that’s surely the $64,000 question (at 80% fEC)…. we’re all more or less familiar with the potential shortcomings of peer review as a method of allocating funding, but if not peer review… then what?

In fact, the Rand Europe report is not an anti peer-review polemic, and deserves a more nuanced response than a “peer review: yes or no” on-line poll.  The only sensible answer, surely, is: well, it depends what you want to achieve.  The report itself aims to

inspire thinking amongst research funders by showing how the research funding review process can be changed, and to give funders the confidence to try novel methods by explaining where and how such approaches have been used previously.

But crucially…

This is not intended to replace peer review, which remains the best method for review of grant applications in many situations. Rather, we hope that by considering some of the alternatives to peer review, where appropriate, research funders will be able to support a wider portfolio of projects, leading to more innovative, high-impact work.

A number of the options in the report seem to be more related to changing the nature and scope for calls for proposals than changing the nature of peer review itself – many in ways that aren’t unfamiliar.  But I’d like to pick out one idea for particular comment: sand pits.

I believe the origin of the term is from computing, where the term ‘sand box’ or ‘sand pit’ was used to describe an area for experimentation or testing, where no damage could be done to the overall system architecture.  I guess the notion of harmless – even playful – experimentation is what advocates have in mind.

They sound like a very interesting idea – get a group of people with expertise to bring to bear on a particular problem, put them all in same place for a day, or a number of days, and see what emerges from discussions.  It’s not really caught on yet in the social sciences, although social scientists have been involved, of course.  The notion of cooperating rather than competing, and of new research collaborations forming, is an interesting and an appealing idea.  As a way of bringing new perspectives to bear on a particular problem – especially an interdisciplinary problem – it looks like an attractive alternative.

There are problems, though.  If there are more applications to participate than there are places, there will inevitably need to be choices made and applications accepted and rejected.  I would imagine that questions of fit and balance would be relevant as well as questions of experience and expertise, but someone or some group of people will have to make choices.  From the application forms I’ve seen, this is often on the basis of short CV and a short statement.  So… don’t we end up relying on some element of peer review anyway?

Secondly, I wonder about equal opportunities.  If a sand pit event is to take place over several days in a hotel, it will inevitably be difficult or even impossible for some to attend. Those who are parents and/or carers. Those who have timetabled lectures and tutorials.  Those who have other professional or personal diary commitments that just can’t be moved.  For a standard peer reviewed call, no-one is excluded completely because it clashes with an important family event.  Can we be sure that all of the best researchers will even apply?

I should say that I’ve never attended a sandpit event, but I have attended graduate recruitment/selection events (offered, deferred, and finally declined, since you ask), and residential training courses.  They’re all strange situations where both competitive and cooperative behaviours are rewarded, and I wonder how people react.  If I were a funder, I’d be worried that the prizes might be going to the best social operators, rather than those with the best ideas.  It’s a myth that academic brilliance is always found in inverse proportion to social skills, of course, but even so, my concern would be about whether one or more dominant figures could ending up forming projects around themselves. I also wonder about existing cliques or vested interests of whatever kind having a disproportionate influence.

I’m sure that effective facilitation and chairing can go a long way to minimising at least some of the potential problems, and while I think sandpits are an intriguing and promising alternative to peer review, they’re not without problems of their own.  I’d be very interested to hear from anyone who’s attended a sandpit – am I doing them a disservice here?

Although I’m open to other ideas for distributing research funding – by all means, let’s be creative, and let’s look at alternatives – I don’t see a replacement for peer review.  Which isn’t to say that there isn’t scope to improve the quality of peer review.  Because, Reviewer Q, there certainly is.

And perhaps that’s the point that the 70.8% were trying to make.

ESRC Centres and Large Grants competition

A pot of gold at the end of a rainbow
"The leprechauns wanted in principle to leave a pot of gold at the end of your rainbow, but unfortunately funds are limited...."

The ESRC Centres and Large Grants competition was launched earlier this week.

We already knew a few things – that the full call would be out sometime this month, that it would have some steer towards some version of the three strategic priorities, and that there would be funding for about 5 centres or projects at £2m-£5m each.  We knew that the new scheme would be a combination of the formerly-separate Centres and Large Grant schemes.  Although there’s an argument for that a ‘Centre’ and a ‘Large Grant’ are different beasts, this seems to me like another example of a sensible merger of schemes, as with the new Future Research Leaders call combining the First Grants and the Postdoc Fellowship schemes.

We also knew that competition would be fierce.   It must be eighteen months, perhaps longer, since the last comparable call.  It wouldn’t be surprising, then, if there are two calls worth of ideas and projects being prepared for this call.  Unsurprisingly, there’s an outline proposal stage, followed by an invited full proposal stage, followed by short listing for interviews.  It would be interesting to know how many applications the ESRC foresee making it through each stage.  I’m sure this will depend in part on the quality of the applications they receive, but they must have a rough ratio in mind.  Whichever way you look at it, even for those with exceptional ideas, the odds aren’t great.  But then, they seldom are.

So, what do we know now that we didn’t know before?

We know that there are three areas – each an aspect of one of the three priorities – which the ESRC would “particularly welcome” applications on:

Risk: The importance of risk and its relationship with behaviours: for individuals and organisations, understanding the role of attitudes, decisions and consequences; for organisations and society the implications of public and practitioner constructions of risk and divergent framings; the challenges for effective governance, national and international – and the significance of social gradients and inequalities in essential areas of risk…

Behaviour change: Causes and agents of behavioural change: understanding how social norms, signals and triggers such as new technologies or novel regulation impact on decisions and actions of people, social groups and organisations, how and why behaviour changes at key periods and in what social, national and international contexts – thus informing the development and evaluation of interventions…

Community, participation and democracy in an era of austerity: Understanding how individuals and communities most effectively make their voices heard, and how social and physical mobility changes when in countries like the UK, the state retrenches…

Some might regard the third priority area as a brave move after the AHRC controversy.  But I guess as long as no-one mentions the government’s “BS” by name, probably no-one will notice.  And it is a legitimate and important area for research.

So….. three themes, an open element, and five to be funded.  One per theme and two open seems a likely outcome, though I’m sure that’s not pre-decided.  I guess the question for those with a project in mind is how far they’re willing and able to bend it to meet the themes, or whether they just ignore the steer and aim directly at the open element of the call.  And the question for the decision-makers is how they respond to bids that are covered in crowbar marks that are hidden under a thin veneer of priority-speak.  I think my advice to potential applicants would probably be to either to write an application that speaks directly and indisputably to one of the three areas of steer, or to go for the open element.  Or to swerve this call entirely, and go for the Research Grants scheme, which has an upper limit of £2m, the same as the lower limit for this call.

What else is striking about the call?   That academic merit alone won’t be enough.  Not to be trusted with up to £5 million quid of taxpayer’s cash in a time of austerity.

“…but it is likely that successful applications will be led by experienced researchers who are internationally recognised and have a well established publication track record within their field of study, and where we can be assured of the ability to manage a large scale research project.” [my underline]

And from the list of assessment criteria:

“A robust management structure with a nominated director(s) (for Centre applications) and clear arrangements for co-ordination and management of the strategic direction of the Centre/Grant”

At outline stage, one page of the available four for the Case for Support needs to be a Management Plan.  A full quarter of the available free text space, even at this early stage of the process.  The ‘Pathways to Impact’ document is not part of the outline stage, but the Management Plan is.  That surely tells its own story – have a strong account about project management to tell, or don’t expect to make it to the next stage.

And of course, it makes sense.  If I were in the unenviable position of thinning the field in the search for the famous five to be funded, one sifting approach I’d want to use is to knock out any that – regardless of the brilliance of their ideas – I don’t feel absolutely confident in trusting with the money.  These are massive, massive investments, and they’ve got to deliver.  They’ve got to give the ERSC success stories to shout about, given the relative generosity of the flat cash CSR settlement.  They just have to.

I hope there’s space for creativity and delegation in management planning, though, rather than expecting a superhuman PI to do everything.  And I hope other kinds of management experience (Head of School and similar roles, pre-academic career experience) as well as running large research projects will be acceptable assurances of ability.  In the medium and long term, though, with the fractured funding landscape, I can’t help but wonder how people are meant to get experience of leading projects.

One other thing struck me.  I was half-expecting that there might be some kind of ‘demand management’ measure here, perhaps limiting each institution to submitting one bid as lead partner.  But I’m pleased to see that there’s nothing like that – institutions aren’t in a good position to chose between competing proposals, as they lack experts without a conflict of interest.  Which is one of the reasons why I’m against Quota systems of demand management.

Demand management.  The fractured funding landscape.  Two things I promise I’ll blog about soon.

ESRC UK Tour 2011

A bus heading for Swindon

The ESRC have published the slides from presentations given at a number of regional events over the last few months.

I was fortunate enough to be able to attend the East Midlands event hosted at the University of Leicester.  Although much of the information in the presentations was already available in either the ‘Delivery Plan’ or the ‘Demand Management’ proposals (and I’ll be blogging about the latter at some point soon) it was well worth attending for the little extra snippets of information and the subtle nuances, and the opportunity to ask questions.

I know the massive changes to the funding landscape and the proposed ‘demand management’ measures mean that we’re living in unusual times, but I’d like to see a similar ‘grand tour’ become a regular fixture.

Mind your PQQs…

A picture of Gollum from the recent Lord of the Rings films
"We wants it! We wants the precious tender documents!"

One of the things I do on a semi-regular basis in the search for research funding for colleagues is look at public sector tenders for possible opportunities.  The vast majority aren’t relevant for my purposes – I’m sure that tender for cadaver transport (presumably to remove the skeletons from the cupboards?) was a good opportunity for someone, though.

Something I’ve never understood – and if anyone can explain it I’d love to know – is why the UK public sector goes to such lengths to make it difficult to get hold of the full set of documents for their calls for tenders.  Usually there’s the briefest of summaries provided in a publicly accessible form, and it’s this that I usually send on.  To get the full picture, you need to log in to the relevant agency’s shiny web tendering system.

But you can’t log in until you’ve created an account.

And you can only create an account by giving full details of your organisation.  Who you are, where you are, what you do, what type of organisation you are, how many employees the organisation has.  Once that all been entered, then there are usually some activity codes to select to reflect the organisations main line of business.   No doubt this is all crushingly important – I wouldn’t want anyone to mistakenly think that the University of Nottingham was (a) an SME; or (b) a supplier of paperclip-related products.  It’s as if all these information requests are a precursor to a deep and meaningful long-standing relationship, when all I’m doing – for now – is having a quick flirt with a potentially attractive and available funding opportunity (“my academic mate might fancy you”) that may not prove to be our type.

I’m never sure whether to register as ‘University of Nottingham’ or ‘Nottingham University Business School’.  But the problem with signing up as Nottingham University Business School is that it doesn’t have a separate legal existence, while the problem with signing up as UoN is that no-one else can do so later.  I’ve had problems before when using a shiny electronic tendering system with trying to hunt down the institutional username and password set up as a result of a tentative interest in some other call which everyone’s long since forgotten about.

And… am I the institutional contact?  I can’t sign or submit stuff on behalf of the university, so… no.  However, I want the information, and I become very unpopular if I enter the details of the people who can sign stuff off so that they get bombarded with increasingly cryptic shiny electronic tendering system messages.

So far, so tedious.  I’ve put in the minimum possible level of information about UoN or NUBS, and I’ve taken a view on who ought to be the contact.  I’m now mildly annoyed.

Can I see the tender documents, now, please?

Well, no.  The system is going to email you first to confirm your account.  You’ll need to wait about ten minutes and then click on the link.  I’m now quite frustrated.

Okay, so now….

No.  You’ll need to log in again, using the username and password that we’ll send you, possibly in separate emails.  In about another ten minutes.  I’m now very annoyed, and I’m telling myself how more annoyed I’m going to be if this goes nowhere.

Right, so…

Ah, no.  You’ll have to change the password first.  Which means you need to think of a new one that’s memorable yet not important.  You might have to share it with others involved in any tender, so it shouldn’t be your internet banking password, your facebook page, your twitter account, your blog, or your university email password.

Then – and only then – can you see the precious documents.   Of course, if it turns out that the academic who asked you for your information turns out not to be interested, the price of the information you extracted is being bombarded with cryptic emails from the shiny system demanding that you log in… from now until the closing date.  And perhaps months later.  There’s nothing I find more helpful than being informed that some shiny electronic tendering system that I’d long forgotten about is being upgraded to an even shinier system and won’t be available from 7:30 to 7:32 on Sunday week.  I promise I’ll try not to let it spoil my weekend.

I understand why procurement people want to know who is interested in their tender.  I’d want to know that too.  I’d want to know if my call was reaching SMEs, or universities, or consultancies, or overseas, and so on.  I’d want to know who was interested, but didn’t tender, and perhaps even why.  But seriously…. does the process have to be so onerous?  Can’t the information just be made freely available – or at least available with a much shorter registration process?  I don’t want to register and set up a supplier profile on your shiny new system.

I.  Just.  Want.  The.  Call.  Information.  Please?

ESRC Future Research Leaders call announced

A Brazilian football badge
Brazilian international footballers... guaranteed four stars....

The ESRC has recently launched their long-awaited Future Research Leaders scheme, and it’s a mixture of good news and not so good news.

The good news first – that there’s a scheme at all, and that there’s funding at all.  As senior ESRC staff are quick to point out, the research councils did well to get a ‘flat cash’ settlement in the comprehensive spending review.  It could be much, much worse.  Another piece of good news, I think, has been the merger of the old ‘First Grants Scheme’ and the ‘Post-doctoral Fellowship’ scheme.  The problem with the PDF was that those who had a permanent academic contract could not apply.  I don’t know about other disciplines, but in Business and Management, I think it’s fair to say that most of the best and brightest career young researchers would be snapped up.  Now, it’s possible that some of the best and brightest might have turned down a permanent academic (research and teaching) contract for a year or so of concentrated research time, but that would be a brave move.  So I wonder if the ESRC ended up funding the best of the best who didn’t get permanent jobs – but perhaps that’s unfair.

So… limitations of the old PDF scheme and reduced budgets make a consolidated scheme seem sensible.  But the change in emphasis is clear even from the language.  The clue’s in the name – with the old ‘First Grants Scheme’, it was about outstanding career young researchers with outstanding ideas who hadn’t yet had a chance to be PI on their own project.  Make no mistake – it was always very competitive, and before the ESRC introduced an outline stage, the success rates were lower than for the late lamented Small Grants Scheme.  But ‘Future Research Leaders’ strikes a rather different note.  When I first heard the name I thought this marked a shift from the a broad scheme, to a much more narrow, much more elitist one.  And that’s been confirmed by the call specification.

“We expect to see only a limited number of outline applications from a single research organisation; only bids from outstanding individuals, with the potential in Research Excellence Framework terms to become the 4* researchers of the future, should be submitted through this call”

And there are other limitations too.  If I remember rightly, the old FGS eligibility rules were for seven years post-PhD.  With FRL, we’re down to four years.  Add in the fact that there was no call last year because of the Comprehensive Spending Review, and it’s obvious that a whole cohort of early career researchers will miss out on this opportunity.  The only people who should be applying are those sitting right at the centre of a Venn Diagram of demonstrable 4* potential, post-doc experience eligibility, and having an absolutely first class outstanding project.  Anyone else looking at this call, frankly, is wasting their time.

While I’m not sure about the eligibility rule changes, did anyone really think that those getting funding through this scheme or its predecessors weren’t the 4*ers of the future?  Perhaps this is just an example of the ESRC being more up front about its funding criteria – or, better – what it actually takes to get funding through this call.  But I do think that the current social science research funding landscape has very serious problems.  Yes, let’s encourage the 4*s of the future, but we also need 3*s, and even 2*s and 1*s, both in their own right, and to properly exploit, comment upon, and explore the implications and applications of 4* research.  But the dysfunction of the funding landscape is a topic for another blog.

But….. no-one can accuse the ESRC of not being absolutely up front about this.  And it’s not hard to see why.  With no call for two years, other funding sources drying up, institutional hunger for attracting research funding, rising teaching loads across the sector, and promotion incentives for grant getting, there was a real danger of the ESRC drowning in a tidal wave of applications.  In many ways, this is the first test of the ESRC’s “demand management” request for institutions to self-regulate.  Let’s see if we’re capable.