Funding Policy | Cash for Questions: Research funding, policy and development

ESRC success rates 2014/2015 – a quick and dirty commentary

The ESRC has issued its annual report and accounts for the financial year 2014/15, and they don’t make good reading. As predicted by Brian Lingley and Phil Ward back in January on the basis of the figures from the July open call, the success rate is well down – to 13% – from the 25% I commented on last year , 27% on 2012-13 and 14% of 2011-2012.

Believe it or not there is a staw-grasping positive way of looking at these figures… of which more later.

This research professional article has a nice overview which I can’t add much to, so read it first. Three caveats about these figures, though…

They’re for the standard open call research grant scheme, not for all calls/schemes
They relate to the financial year, not the academic year
It’s very difficult to compare year-on-year due to changes to the scheme rules, including minimum and maximum thresholds which have changed substantially.

In previous years I’ve focused on how different academic disciplines have got on, but there’s probably very little to add. You can read them for yourself (p. 38), but the report only bothers to calculate success rates for the disciplines with the highest numbers of applications – presumably beyond that there’s little statistical significance. I could be claiming that it’s been a bumper year for Education research, which for years bumped along at the bottom of the league table with Business and Management Studies in terms of success rates, but which this year received 3 awards from 22 applications, tracking the average success rate. Political Science and Socio-Legal Studies did well, as they always tend to do. But it’s generalising from small numbers.

As last year, there is also a table of success rates by institution. In an earlier section on demand management, the report states that the ESRC “are discussing ways of enhancing performance with those HEIs where application volume is high and quality is relatively weak”. But as with last year, it’s hard to see from the raw success rate figures which these institutions might be – though of course detailed institutional profiles showing the final scores for applications might tell a very different story. Last year I picked out Leeds (10/0), Edinburgh (8/1), and Southampton (14/2) as doing poorly, and Kings College (7/3), King Leicester III (9/4), Oxford (14/6) as doing well – though again, one more or less success changes the picture.

This year, Leeds (8/1) and Edinburgh (6/1) have stats that look much better. Southampton doesn’t look to have improved (12/0) at all, and is one of the worst performers. Of those who did well last year, none did so well this year – Kings were down to 11/1, Leicester 2/0, and Oxford 11/2. Along with Southampton, this year’s poor performers were Durham (10/0), UCL (15/1) and Sheffield (11/0) – though all three had respectable enough scores last time. This year’s standouts were Cambridge at 10/4. Perhaps someone with more time than me can combine success rates from the last two years, and I’m sure someone at the ESRC already has….

So… on the basis of success rates alone, probably only Southampton jumps out as doing consistently poorly. But again, much depends on the quality profile of the applications being submitted – it’s entirely possible that they were very unlucky, and that small numbers mask much more slapdash grant submission behaviour from other institutions. And of course, these figures only relate to the lead institution as far as I know.

It’s worth noting that demand management has worked… after a fashion.

We remain committed to managing application volume, with
the aim of focusing sector-wide efforts on the submission
of a fewer number of higher quality proposals with a
genuine chance of funding. General progress is positive.
Application volume is down by 48 per cent on pre-demand
management levels – close to our target of 50 per cent.
Quality is improving with the proportion of applications now
in the ‘fundable range’ up by 13 per cent on pre-demand
management levels, to 42 per cent. (p. 21).

I remember the target of reducing the numbers of applications received by 50% as being regarded as very ambitious at the time, and even if some of it was achieved by changing scheme rules to increase the minimum value of a grant application and banning resubmissions, it’s still some achievement. Back in October 2011 I argued that the ESRC had started to talk optimistically about meeting that target after researcher sanctions (in some form) had started to look inevitable. And in November 2012 things looked nicely on track.

But reducing brute numbers of applications is all very well. But if only 42% of applications are within the “fundable range”, then that’s a problem because it means that a lot of applications being submitted still aren’t good enough.This is where there’s cause for optimism – if less than half of the applications are fundable, your own chances should be more than double the average success rate – assuming that your application is of “fundable” quality. So there’s your good news. Problem is, no-one applies who doesn’t think their application is fundable.

Internal peer review/demand management processes are often framed in terms of improving the quality of what gets submitted, but perhaps not enough of a filtering process. So we refine and we polish and we make 101 incremental improvements… but ultimately you can’t polish a sow’s ear. Or something.

Proper internal filtering is really, really hard to do – sometimes it’s just easier to let stuff from people who won’t be told through and see if what happens is exactly what you think will happen, which it always is. There’s also a fine line (though one I think that can be held and defended) between preventing perceived uncompetitive applications from doing so and impinging on academic freedom. I don’t think telling someone they can’t submit a crap application is infringing their academic freedom, but any such decisions need to be taken with a great deal of care. There’s always the possibility of suspicion of ulterior motives – be it personal, be it subject or methods-based prejudice, or senior people just overstepping the mark and inappropriately imposing their convictions (ideological, methodological etc) on others. Like the external examiner who insists on “more of me” on the reading list….

The elephant in the room, of course, is the flat cash settlement and the fact that that’s now really biting, and that there’s nowhere near enough funding to go around for all of the quality social science research that’s badly needed. But we can’t do much about that – and we can do something about the quality of the applications we’re submitting and allowing to be submitted.

I wrote something for research professional a few years back on how not to do demand management/filtering processes, and I think it still stands up reasonably well and is even quite funny in places (though I say so myself). So I’m going to link to it, as I seem to be linking to a disproportionate amount of my back catalogue in this post.

A combination of a new minimum of £350k for the ESRC standard research grants scheme and the latest drop in success rates makes me think it’s worth writing a companion piece to this blog post about potential ESRC applicants need to consider before applying, and what I think is expected of a “fundable” application.

Hopefully something for the autumn…. a few other things to write about first.

ESRC – sweeping changes to the standard grants scheme

The ESRC have just announced a huge change to their standard grants scheme, and I think it’s fair to say that it’s going to prove somewhat controversial.

At the moment, it’s possible to apply to the ESRC Standard Grant Scheme at any time for grants of between £200k and £2million. From the end of June this year, the minimum threshold will raise from £200k to £350k, and the maximum threshold will drop from £2m to £1m.

Probably those numbers don’t mean very much to you if you’re not familiar with research grant costing, but as a rough rule of thumb, a full time researcher for a year (including employment costs and overheads) comes to somewhere around £70k-80k. So a rough rule of thumb I used to use was that if your project needed two years of researcher time, it was big enough. So… for £350k you’d probably need three researcher years, a decent amount of PI and Co-I time, and a fair chunk of non-pay costs. That’s a big project. I don’t have my filed in front of me as I’m writing this, so maybe I’ll add a better illustration later on.

This isn’t the first time the lower limit has been raised. Up until February 2011, there used to be a “Small Grants Scheme” for projects up to £200k before that was shut, with £200k becoming the new minimum. The argument at the time was that larger grants delivered more, and had fewer overheads in terms of the costs of reviewing, processing and administering. And although the idea was that they’d help early career researchers, the figures didn’t really show that.

The reasons given for this change are a little ~~disingenuous~~ puzzling. Firstly, this:

The changes are a response to the pattern of demand that is being placed on the standard grants scheme by the social science community. The average value of a standard grant application has steadily increased and is now close to £500,000, so we have adjusted the centre of gravity of the scheme to reflect applicant behaviour.

Now that’s an interesting tidbit of information – I wouldn’t have guessed that the “average value” would be that high, but you don’t have to be an expert in statistics (and believe me, in spite of giving 110% in maths class at school I’m not one) to wonder what “average” means, and further, why it even matters. This might be an attempt at justification, but I don’t see why this provides a rationale for change.

Then we have this….

The changes are also a response to feedback from our Grant Assessment Panels who have found it increasingly difficult to assess and compare the value of applications ranging from £200,000 to £2 million, where there is variable level of detail on project design, costs and deliverables. This issue has become more acute as the number of grant applications over £1 million has steadily increased over the last two years. Narrowing the funding range of the scheme will help to maintain the robustness of the assessment process, ensuring all applications get a fair hearing.

I have every sympathy for the Grant Assessment Panel members here – how do you choose between funding one £2m project and funding 10 x £200k projects, or any combination you can think of? It’s not so much comparing apples to oranges as comparing grapes to water melons. And they’re right to point out the “variable” level of detail provided – but that’s only because their own rules give a maximum of 6 A4 page for the Case for Support for projects under £1m and 12 for those over. If you think that sounds superficially reasonable, then notice that it’s potentially double the space to argue for ten times the money. I’ve supported applications of £1m+ and 12 sides of A4 is nowhere near enough, compared to the relative luxury of 6 sides for £200k. This is a problem.

In my view it makes sense to “introduce an annual open competition for grants between £1 million and £2.5 million”, which is what the ESRC propose to do. So I think there’s a good argument for lowering the upper threshold from £2m to £1m and setting it up as a separate competition. I know the ESRC want to reduce the number of calls/schemes, but this makes sense. As things stand I’ve regularly steered people away from the Centres/Large Grants competition towards Standard Grants instead, where I think success rates will be higher and they’ll get a fairer hearing. So I’d be all in favour of having some kind of single Centres/Large/Huge/Grants of Unusual Size competition.

But nothing here seems to me to be an argument for raising the lower limit.

But finally, I think we come to what I suspect is the real reason, and judging by Twitter comments so far, I’m not alone in thinking this.

We anticipate that these changes will reduce the volume of applications we receive through the Standard Grants scheme. That will increase overall success rates for those who do apply as well as reducing the peer review requirements we need to place on the social science community.

There’s a real problem with ESRC success rates, which dropped to 10% in the July open call, with over half the “excellent” proposals unfunded. This is down from around 25% success rates, much improved in the last few years. I don’t know whether this is a blip – perhaps a few very expensive projects were funded and a lot of cheaper ones missed out – but it’s not good news. So it’s hard not to see this change as driven entirely by a desire to get success rates up, and perhaps an indication that this wasn’t a blip.

In a recent interview with Adam Smith of Research Professional, Chief Exec Jane Eliot recently appeared to rule out the option of individual sanctions which had been threatened if institutional restraint failed to bring down the number of poor quality applications and it appears that the problem is not so much poor quality applications as lots of high quality applications, not enough money, plummeting success rates, and something needing to be done.

All this raises some difficult questions.

Where are social science researchers now supposed to go for funding for projects whose “natural size” is between £10k (British Academy Small Grants) and £350k, the proposed new minimum threshold? There’s only really the Leverhulme Trust, whose schemes will suit some project types and but not others, and they’re not exclusively a social science funder.
Where will the next generation of PIs to be entrusted with £350k of taxpayer’s money have an opportunity to cut their teeth, both in terms of proving themselves academically and managerially?
What about career young researchers? At least here we can expect a further announcement – there has been talk of merging the ‘future leaders scheme’ into Standard Grants, so perhaps there will be a lower minimum for them. But we’ll see.
Given that the minimum threshold has been almost doubled, what consultation has been carried out? I’m just a humble Business School Research Manager (I mean I’m humble, my Business School is outstanding, obviously) so perhaps it’s not surprising that this the first I’ve heard. But was there any meaningful consultation over this? Is there any evidence underpinning claims for the efficiency of fewer, longer and larger grants?
How do institutions respond? I guess one way will be to work harder to create bigger gestalt projects with multiple themes and streams and work packages. But surely expectations of grant getting for promotion and other purposes need to be dialled right back, if they haven’t been already. Do we encourage or resist a rush to get applications in before the change, at a time when success rates will inevitably be dire?

Of course, the underlying problem is that there’s not enough money in the ESRC’s budget to support excellent social science after years and years of “flat cash” settlements. And it’s hard to see what can be done about that in the current political climate.

ESRC success rates 2013/2014

The ESRC Annual Report for 2013-14 has been out for quite a while now, and a quick summary and analysis from me is long overdue.

Although I was tempted to skip straight through all of the good news stories about ESRC successes and investments and dive straight in looking for success rates, I’m glad I took the time to at least skim read some of the earlier stuff. When you’re involved in the minutiae of supporting research, it’s sometimes easy to miss the big picture of all the great stuff that’s being produced by social science researchers and supported by the ESRC. Chapeau, everyone.

In terms of interesting policy stuff, it’s great to read that the “Urgency Grants” mechanism for rapid responses to “rare or unforeseen events” which I’ve blogged about before is being used, and has funded work “on the Philippines typhoon, UK floods, and the Syrian crisis”. While I’ve not been involved in supporting an Urgency Grant application, it’s great to know that the mechanism is there, that it works, and that at least some projects have been funded.

The “demand management” agenda

This is what the report has to say on “demand management” – the concerted effort to reduce the number of applications submitted, so as to increase the success rates and (more importantly) reduce the wasted effort of writing and reviewing applications with little realistic chance of success.

Progress remains positive with an overall reduction in application numbers of 41 per cent, close to our target of 50 per cent. Success rates have also increased to 31 per cent, comparable with our RCUK partners. The overall quality of applications is up, whilst peer review requirements are down.

There are, however, signs that this positive momentum may

be under threat as in certain schemes application volume is

beginning to rise once again. For example, in the Research

Grants scheme the proposal count has recently exceeded

pre-demand management levels. It is critical that all HEIs

continue to build upon early successes, maintaining the

downward pressure on the submission of applications across

all schemes.

It was always likely that “demand management” might be the victim of its own success – as success rates creep up again, getting a grant appears more likely and so researchers and research managers encourage and submit more applications. Other factors might also be involved – the stage of the REF cycle, for example. Or perhaps now talk of researcher or institutional sanctions has faded away, there’s less incentive for restraint.

Another possibility is that some universities haven’t yet got the message or don’t think it applies to them. It’s also not hard to imagine that the kinds of internal review mechanisms that some of us have had for years and that we’re all now supposed to have are focusing on improving the quality of applications, rather than filtering out uncompetitive ideas. But is anyone disgracing themselves?

Looking down the list of successes by institution (p. 41) it’s hard to pick out any obvious bad behaviour. Most of those who’ve submitted more than 10 applications have an above-average success rate. You’d only really pick out Leeds (10 applications, none funded), Edinburgh (8/1) and Southampton (14/2), and a clutch of institutions on 5/0, (including top-funded Essex, surprisingly) but in all those cases one or two more successes would change the picture. Similarly for the top performers – Kings College (7/3), King Leicester III (9/4), Oxford (14/6) – hard to make much of a case for the excellence or inadequacy of internal peer review systems from these figures alone. What might be more interesting is a list of applications by institution which failed to reach the required minimum standard, but that’s not been made public to the best of my knowledge. And of course, all these figures only refer to the response mode Standard Grant applications in the financial year (not academic year) 2013-14.

Concentration of Funding

Another interesting stat (well, true for some values of “interesting”) concerns the level of concentration of funding. The report records the expenditure levels for the top eleven (why 11, no idea…) institutions by research expenditure and by training expenditure. Interesting question for you… what percentage of the total expenditure do the top 11 institutions get? I could tell you, but if I tell you without making you guess first, it’ll just confirm what you already think about concentration of funding. So I’m only going to tell you that (unsurprisingly) training expenditure is more concentrated than research funding. The figures you can look up for yourself. Go on, have a guess, go and check (p. 44) and see how close you are.

Research Funding by Discipline

On page 40, and usually the most interesting/contentious. Overall success rate was 25% – a little down from last year, but a huge improvement on 14% two years ago.

Big winners? History (4 from 6); Linguistics (5 from 9), social anthropology (4 from 9), Political and International Studies (9 from 22), and Psychology (26 from 88, – just under 30% of all grants funded were in psychology). Big losers? Education (1 from 27), Human Geography (1 from 19), Management and Business Studies (2 from 22).

Has this changed much from previous years? Well, you can read what I said last year and the year before on this, but overall it’s hard to say because we’re talking about relatively small numbers for most subjects, and because some discipline classifications have changed over the last few years. But, once again, for the third year in a row, Business and Management and Education do very, very poorly.

Human Geography has also had a below average success rate for the last few years, but going from 1 in 19 from 3 from 14 probably isn’t that dramatic a collapse – though it’s certainly a bad year. I always make a point of trying to be nice about Human Geography, because I suspect they know where I live. Where all of us live. Oh, and Psychology gets a huge slice of the overall funding, albeit not a disproportionate one given the number of applications.

Which kinds of brings us back to the same questions I asked in my most-read-ever piece – what on earth is going on with Education and Business and management research, and why do they do so badly with the ESRC? I still don’t have an entirely satisfactory answer.

I’ve put together a table showing changes to disciplinary success rates over the last few years which I’m happy to share, but you’ll have to email me for a copy. I’ve not uploaded it here because I need to check it again with fresh eyes before it’s used – fiddly, all those tables and numbers.

Pre-mortems: Tell me why your current grant application or research project will fail

I came across a really interesting idea the other ~~day~~ week via the Freakonomics podcast – the idea of a project “pre-mortem” or “prospective hindsight” They interviewed Gary Klein who described it as follows:

KLEIN: I need you to be in a relaxed state of mind. So lean back in your chair. Get yourself calm and just a little bit dreamy. I don’t want any daydreaming but I just want you to be ready to be thinking about things. And I’m looking in a crystal ball. And uh, oh, gosh…the image in the crystal ball is a really ugly image. And this is a six-month effort. We are now three months into the effort and it’s clear that this project has failed. There’s no doubt about it. There’s no way that it’s going to succeed. Oh, and I’m looking at another scene a few months later, the project is over and we don’t even want to talk about it. And when we pass each other in the hall, we don’t even make eye contact. It’s that painful. OK. So this project has failed, no doubt about it [….] I want each of you to write down all the reasons why this project has failed. We know it failed. No doubts. Write down why it failed.

The thinking here is that such an approach to projects reduces overconfidence, and elsewhere the podcast discusses the problems of overconfidence, “go fever”, the Challenger shuttle disaster, and how cultural/organisational issues can make it difficult to bring up potential problems and obstacles. The pre-mortem exercise might free people from that, and encourages people (as a team) to find reasons for failure and then respond to them. I don’t do full justice to the arguments here, but you can listen to it for yourself (or read the transcript) at the link above. It reminds me of some of the material covered in a MOOC I took which showed how very small changes in the way that questions are posed and framed can make surprisingly large differences to the decisions that people make, so perhaps this very subtle shift in mindset might be useful.

How might we use the idea of a pre-mortem in research development? My first thought was about grant applications. Would it help to get the applicants to undertake the pre-mortem exercise? I’m not sure that overconfidence is often a huge problem among research teams (a kind of grumpy, passive-aggressive form of entitled pessimism is probably more common), so perhaps the kind of groupthink overconfidence/excessive positivity is less of an issue than in larger project teams where nobody wants to be the one to be negative. But perhaps there’s value in asking the question anyway, and re-focusing applicants on the fact that they’re writing an application for reviewers and for a funding body, not for themselves. A reminder that the views, priorities, and (mis)interpretations of others are crucial to their chances of success or failure.

Would it help to say to internal reviewers “assume this project wasn’t funded – tell me why”? Possibly. It might flush out issues that reviewers may be too polite or insufficiently assertive to raise otherwise, and again, focuses minds on the nature of the process as a competition. It could also help reviewers identify where the biggest danger for the application lies.

Another way it could usefully be used is in helping applicants risk assess their own project. Saying to them “you got funded, but didn’t achieve the objectives you set for yourself. Why not?” might be a good way of identifying project risks to minimise in the management plan, or risks to alleviate through better advanced planning. It might prompt researchers to think more cautiously about the project timescale, especially around issues that are largely out of their control.

So… has anyone used anything like this before in research development? Might it be a useful way of thinking? Why will your current application fail?

Adam Golberg announces new post about Ministers inserting themselves into research grant announcements

“You might very well think that as your hypothesis, but I couldn’t possibly comment”

Here’s something I’ve been wondering recently. Is it just me, or have major research council funding announcements started to be made by government ministers, rather than by the, er, research councils?

Here’s a couple of examples that caught my eye from the last week or so. First, David Willetts MP “announces £29 million of funding for ESRC Centres and Large Grants“. Thanks Dave! To be fair, he is Minster of State for Universities and Science. Rather more puzzling is George Osborne announcing “22 new Centres for Doctoral Training“, though apparently he found the money as Chancellor of the Exchequer. Seems a bit tenuous to me.

So I had a quick look back through the ESRC and EPSRC press release archives to see if the prominence of government ministers in research council funding announcements was a new thing or not. Because I hadn’t noticed it before. With the ESRC, it is new. Here’s the equivalent announcement from last year in which no government minister is mentioned. With the EPSRC, it’s being going on for longer. This year’s archive and the 2013 archive show government ministers (mainly Willetts, sometimes Cable or Osborne) front and centre in major announcements. In 2012 they get a name check, but normally in the second or third paragraph, not in the headline, and don’t get a picture of themselves attached to the story.

Does any of this matter? Perhaps not, but here’s why I think it’s worth mentioning. The Haldane Principle is generally defined as “decisions about what to spend research funds on should be made by researchers rather than politicians”. And one of my worries is that in closely associating political figures with funding decisions, the wrong impression is given. Read the recent ESRC announcement again, and it’s only when you get down to the ‘Notes for Editors’ section that there’s any indication that there was a competition, and you have to infer quite heavily from those notes that decisions were taken independently of government.

Why is this happening? It might be for quite benign reasons – perhaps research council PR people think (probably not unreasonably) that name-checking a government minister gives them a greater chance of media coverage. But I worry that it might be for less benign reasons related to political spin – seeking credit and basking in the reflected glory of all these new investments, which to the non-expert eye look to be something novel, rather than research council business as usual. To be fair, there are good arguments for thinking that the current government does deserve some credit for protecting research budgets – a flat cash settlement (i.e. cut only be the rate of inflation each year) is less good than many want, but better than many feared. But it would be deeply misleading if the general public were to think that these announcements represented anything above and beyond the normal day-to-day work of the research councils.

Jo VanEvery tells me via Twitter that ministerial announcements are normal practice in Canada, but something doesn’t quite sit right with me about this, and it’s not a party political worry. I feel there’s a real risk of appearing to politicise research. If government claims credit, it’s reasonable for the opposition to criticise… now that might be the level of investment, but might it extend to the investments chosen? Or do politicians know better than to go there for cheap political points?

Or should we stop worrying and just embrace it? It’s not clear that many people outside of the research ‘industry’ notice anyway (though the graphene announcement was very high profile), and so perhaps the chances of the electorate being misled (about this, at least) are fairly small.

But we could go further. MEPs to announce Horizon 2020 funding? Perhaps Nick Clegg should announce the results of the British Academy/Leverhulme Small Grants Scheme, although given the Victorian origins of investments and wealth supporting work of the Leverhulme Trust, perhaps the honour should go to the ghosts of Gladstone or Disraeli.