Application advice | Cash for Questions: Research funding, policy and development

Responding to Referees

This post is co-authored by Adam Golberg of Cash for Questions (UK), and Jonathan O’Donnell and Tseen Khoo of The Research Whisperer (Australia).

It arises out of a comment that Jonathan made about understanding and responding to referees on one of Adam’s posts about what to do if your grant application is unsuccessful. This seemed like a good topic for an article of its own, so here it is, cross-posted to our respective blogs.

A quick opening note on terminology: We use ‘referee’ or ‘assessor’ to refer to academics who read and review research grant applications, then feed their comments into the final decision-making process. Terminology varies a bit between funders, and between the UK and Australia. We’re not talking about journal referees, although some of the advice that follows may also apply there.

————————————-

There are funding schemes that offer applicants the opportunity to respond to referees’ comments. These responses are then considered alongside the assessors’ scores/comments by the funding panel. Some funders (including the Economic and Social Research Council [ESRC] in the UK) have a filtering process before this point, so if you are being asked to respond to referees’ comments, you should consider it a positive sign as not all applications get this far. Others, such as the Australian Research Council (ARC), offer you the chance to write a rejoinder regardless of the level of referees’ reports.

If the funding body offers you the option of a response, you should consider your response as one of the most important parts of the application process. A good response can draw the sting from criticisms, emphasise the positive comments, and enhance your chances of getting funding. A bad one can doom your application.

And if you submit no response at all? That can signal negative things about your project and research team that might live on beyond this grant round.

The first thing you might need to do when you get the referees’ comments about your grant application is kick the (imaginary) cat.* This is an important process. Embrace it.

When that’s out of your system, here are four strategies for putting together a persuasive response and pulling that slaved-over application across the funding finish line.

1. Attitude and tone

Be nice. Start with a brief statement thanking the anonymous referees for their careful and insightful comments, even if actually you suspect some of them are idiots who haven’t read your masterpiece properly. Think carefully about the tone of the rest of the response as well. You’re aiming for calm, measured, and appropriately assertive. There’s nothing wrong with saying that a referee is just plain wrong on a particular point, but do it calmly and politely. If you’re unhappy about a criticism or reviewer, there’s a good chance that it will take several drafts before you eliminate all the spikiness from the text. If it makes you feel better (and it might), you can write what you really think in the tone that you think it in but, whatever you do, don’t send that version! This is the version that may spontaneously combust from the deadly mixture of vitriol and pleading contained within.

Preparing a response is not about comprehensively refuting every criticism, or establishing intellectual superiority over the referees. You need to sift the comments to identify the ones that really matter. What are the criticisms (or backhanded compliments) that will harm your cause? Highlight those and answer them methodically (see below). Petty argy-bargy isn’t worth spending your time on.

2. Understanding and interpreting referees’ comments

One UK funder provides referee report templates that invite the referees to state their level of familiarity with the topic and even a little about their research background, so that the final decision-making panel can put their comments into context. This is a great idea, and we would encourage other funding agencies to embrace it.

Beyond this volunteered information (if provided), never assume you know who the referee is, or that you can infer anything else about them because you could be going way off-base with your rant against econometricians who don’t ‘get’ sociological work. If there’s one thing worse than an ad hominem response, it’s an ad hominem response aimed at the wrong target!

One exercise that you might find useful is to produce a matrix listing all of the criticisms, and indicating the referee(s) who made those objections. As these reports are produced independently, the more referees make a particular point, the more problematic it might be. This tabled information can be sorted by section (e.g. methodology, impact/dissemination plan, alternative approaches). You can then repeat the exercise with the positive comments that were made. While assimilating and processing information is a task that academics tend to be good at, it’s worth being systematic about this because it’s easy to overlook praise or attach too much weight to objections that are the most irritating.

Also, look out for, and highlight, any requests that you do a different project. Sometimes, these can be as obvious as “you should be doing Y instead”, where Y is a rather different project and probably closer to the reviewer’s own interests. These can be quite difficult criticisms to deal with, as what they are proposing may be sensible enough, but not what you want to do. In such cases, stick to your guns, be clear what you want to do, and why it’s of at least as much value as the alternative proposal.

Using the matrix that you have prepared, consider further how damaging each criticism might be in the minds of the decision makers. Using a combination of weight of opinion (positive remarks on a particular point minus criticisms) and multiplying by potential damage, you should now have a sense of which are the most serious criticisms.

Preparing a response is not a task to be attempted in isolation. You should involve other members of your team, and make full use of your research support office and senior colleagues (who are not directly involved in the application). Take advantage of assistance in interpreting the referees’ comments, and reviewing multiple drafts of your response.

Don’t read the assessor reports by themselves; you should also go back to your whole application, several times if necessary. It has probably been some time since you submitted the application, and new eyes and a bit of distance will help you to see the application as the referees may have seen it. You could pinpoint the reasons for particular criticisms, or misunderstandings that you assumed they made. While their criticisms may not be valid for the application you thought you wrote, they may very well be so for the one that you actually submitted.

3. The response

You should plan to use the available space in line with the exercise above, setting aside space for each criticism in proportion to its risk of stopping you getting funded.

Quibbles about your budgeted expenditure for hotel accommodation are insignificant compared to objections that question your entire approach, devalue your track-record, invalidate your methodology, or claim that you’re adding little that’s new to the sum of human knowledge. So, your response should:

Make it easy for the decision-makers: Be clear and concise.
Be specific when rebutting from the application. For example: “As we stated on page 24, paragraph 3…”. However, don’t lose sight of the need to create a document that can be understood in isolation as far as possible.
If possible and appropriate, introduce something that you’ve done in the time since submission to rebut a negative comment (be careful, though, as some schemes may not allow the introduction of new material).
Acknowledge any misunderstandings that arise from the application’s explanatory shortcomings or limitations of space, and be open to new clarifications.
Be grateful for the positive comments, but focus on rebutting the negative comments.

4. Be the reviewer

For the best way to really get an idea of what the response dynamic is all about in these funding rounds, consider becoming a grant referee. Once you’ve assessed a few applications and cut your teeth on a whole funding round (they can often be year-long processes), you quickly learn about the demands of the job and how regular referees ‘value’ applications.

Look out for chances to be on grant assessment panels, and say yes to invitations to review for various professional bodies or government agencies. Almost all funding schemes could do with a larger and more diverse pool of academics to act as their ‘gate-keepers’.

Finally: Remember to keep your eyes on the prize. The purpose of this response exercise is to give your project the best possible chance of getting funding. It is an inherent part of many funding rounds these days, and not only an afterthought to your application.

* The writers and their respective organisations do not, in any way, endorse the mistreatment of animals. We love cats. We don’t kick them, and neither should you. It’s just an expression. For those who’ve never met it, it means ‘to vent your frustration and powerlessness’.

I’ve disabled comments on this entry so that we can keep conversations on this article to one place – please head over to the Research Whisperer if you’d like to comment. (AG).

Coping with rejection: What to do if your grant application is unsuccessful. Part 2: Next Steps

Look, I know I said that not getting funded doesn't mean they disliked your proposal, but I need a picture and it's either this or a picture of Simon Cowell with his thumb down. Think on.

In the first part of this series, I argued that it’s important not to misunderstand or misinterpret the reasons for a grant application being unsuccessful. In the comments, Jo VanEvery shared a phrase that she’s heard from a senior figure at one of the Canadian Research Councils – that research funding “is not a test, it’s a contest”. Not getting funded doesn’t necessarily mean that your research isn’t considered to be of high quality. This second entry is about what steps to consider next.

1. Some words of wisdom

‘Tis a lesson you should heed: Try, try, try again.
If at first you don’t succeed, Try, try, try again
William Edward Hickson (1803-1870)

The definition of insanity is doing the same thing over and over but expecting different results
Ben Franklin, Albert Einstein, or Narcotics Anonymous

I like these quotes because they’re both correct in their own way. There’s value to Hickson’s exhortation. Success rates are low for most schemes and most funders, so even if you’ve done everything right, the chances are against you. To be successful, you need a degree of resilience to look for another funder or a new project, rather than embarking on a decade-long sulk, muttering plaintively about how “the ESRC doesn’t like” your research whenever the topic of external funding is raised.

However Franklin et al (or al?) also have a point about not learning from the experience, and repeating the same mistakes without learning anything as you drift from application to application. While doing this, you can convince yourself that research funding is a lottery (which it isn’t) and all you have to do is to submit enough applications and eventually your number will come up (which it won’t). This is the kind of approach (on the part of institutions as well as individuals) that’s pushed us close to ‘demand management’ measures with the ESRC. More on learning from the experience in a moment or two.

2. Can you do the research anyway?

This might seem like an odd question to ask, but it’s always the first one I ask academic colleagues who’ve been unsuccessful with a grant application (yes, this does happen, even at Nottingham University Business School). The main component of most research projects is staff time. And if you’re fortunate enough to be employed by a research-intensive institution which gives you a generous research time allocation, then this shouldn’t be a problem. Granted, you can’t have that full time research associate you wanted, but could you cut down the project and take on some or all of that work yourself or between the investigators? Could you involve more people – perhaps junior colleagues – to help cover the work? Would others be willing to be involved if they can either co-author or be sole author on some of the outputs? Could it be a PhD project?

Directly incurred research expenses are more of a problem – transcription costs, data costs, travel and expenses – especially if you and your co-investigators don’t have personal research accounts to dip into. But if it turns out that all you need is your expenses paying, then a number of other funding options become viable – some external, but perhaps also some internal.

Of course, doing it anyway isn’t always possible, but it’s worth asking yourself and your team that question. It’s also one that’s well worth asking before you decide to apply for funding.

3. What can you learn for next time?

It’s not nice not getting your project funded. Part of you probably wants to lock that application away and not think about it again. Move onwards and upwards, and perhaps trying again with another research idea. While resilience is important, it’s just as important to learn whatever lessons there are to learn to give yourself the best possible chance next time.

One lesson you might be able to take from the experience is about planning the application. If you found yourself running out of time, or not getting sufficient input from senior colleagues, not taking full advantage of the support available within your institution, well, that’s a lesson to learn. Give yourself more time, start earlier before the deadline, and don’t make yourself rush it. If you did all this last time, remember that you did, and the difference that it made. If you didn’t, then the fact is that your application was almost certainly not as strong as it could have been. And if your application document is not the strongest possible iteration of your research idea, your chances of getting funded are pretty minimal.

I’d recommend reading through your application and the call guidance notes once again in the light of referees’ comments. Now that you have sufficient distance from the application, you should ‘referee’ it yourself as well. What would you do better next time? Not necessarily individual application-specific aspects, but more general points. Did your application address the priorities of the call specifically enough, or were the crowbar marks far too visible? Did you get the balance right between exposition and background and writing about the current project? Did you pay enough attention to each section? Did you actually answer the questions asked? Do you understand any criticisms that the referees had?

4. Can you reapply? Should you reapply?

If it’s the ESRC you’re thinking about, then the answer’s no unless you’re invited. I think we’re still waiting on guidance from the ESRC about what constitutes a resubmission, but if you find yourself thinking about how much you might need to tinker with your unsuccessful project to make it a fresh submission, then the chances are that you’ll be barking up the wrong tree. Worst case scenario is that it’s thrown straight out without review, and best case is probably that you end up with something a little too contrived to stand any serious chance of funding.

Some other research funders do allow resubmissions, but generally you will need to declare it. While you might get lucky with a straight resubmission, my sense is that if it was unsuccessful once it will be unsuccessful again. But if you were to thoroughly revise it, polish it, take advice from anyone willing to give it, and have one more go, well, who knows?

But there’s really no shame in walking away. Onwards and upwards to the next idea. Let this one go for now, and working on something new and fresh and exciting instead. Just remember everything that you learnt along the way. One former colleague once told me that he usually got at least one paper out of an application even it was unsuccessful. I don’t know how true that might be more generally, but you’ve obviously done a literature review and come up with some ideas for future research. Might there be a paper in all that somewhere?

Another option which I hinted at earlier when I mentioned looking for the directly incurred costs only is resubmitting to another funder. My advice on this is simple…. don’t resubmit to another funder. Or at least, don’t treat it like a resubmission. Every research funder, every scheme, has different interests and priorities. You wrote an application for one funder, which presumably was tailored to that funder (it was, wasn’t it?). So a few alterations probably won’t be enough.

For one thing, the application form is almost certainly different, and that eight page monstrosity won’t fit into two pages. But cut it down crudely, and if it reads like it’s been cut down crudely, you have no chance. I’ve never worked for a research funding body (unless you count internal schemes where I’ve had a role in managing the process), but I would imagine that if I did, the best way to annoy me (other than using the word ‘impactful‘) would be sending me some other funder’s cast-offs. It’s not quite like romancing a potential new partner and using your old flame’s name by mistake, but you get the picture. Your new funder wants to feel special and loved. They want you to have picked out them – and them alone – for their unique and enlightened approach to funding. Only they can fill the hole in your ~~heart~~ wallet, and satisfy your deep yearning for fulfilment.

And where should you look if your first choice funder does not return your affections? Well, I’m not going to tell you (not without a consultancy fee, anyway). But I’m sure your research funding office will be able to help find you some new prospective partners.

Coping with rejection: What to do if your grant application is unsuccessful. Part 1: Understand what it means…. and what it doesn’t mean

You can't have any research funding. In this life, or the next....

Some application and assessment processes are for limited goods, and some are for unlimited goods, and it’s important to understand the difference. PhD vivas and driving tests are assessments for unlimited goods – there’s no limit on how many PhDs or driving licenses can be issued. In principle, everyone could have one if they met the requirements. You’re not going to fail your driving test because there are better drivers than you. Other processes are for limited goods – there is (usually) only one job vacancy that you’re all competing for, only so many papers that a top journal accept, and only so much grant money available.

You’d think this was a fairly obvious point to make. But talking to researchers who have been unsuccessful with a particular application, there’s sometimes more than a hint of hurt in their voices as they discuss it, and talk in terms of their research being rejected, or not being judged good enough. They end up taking it rather personally. And given the amount of time and effort that must researchers put into their applications, that’s not surprising.

It reminds me of an unsuccessful job applicant whose opening gambit at a feedback meeting was to ask me why I didn’t think that she was good enough to do the job. Well, my answer was that I was very confident that she could do the job, it’s just that there was someone more qualified and only one post to fill. In this case, the unsuccessful applicant was simply unlucky – an exceptional applicant was offered the job, and nothing she could have said or done (short of assassination) would have made much difference. While I couldn’t give the applicant the job she wanted or make the disappointment go away, I could at least pass on the panel’s unanimous verdict on her appointability. My impression was that this restored some lost confidence, and did something to salve the hurt and disappointment. You did the best that you could. With better luck you’ll get the next one.

Of course, with grant applications, the chances are that you won’t get to speak to the chair of the panel who will explain the decision. You’ll either get a letter with the decision and something about how oversubscribed the scheme was and how hard the decisions were, which might or might not be true. Your application might have missed out by a fraction, or been one of the first into the discard pile.

Some funders, like the ESRC, will pass on anonymised referees’ comments, but oddly, this isn’t always constructive and can even damage confidence in the quality of the peer review process. In my experience, every batch of referees’ comments will contain at least one weird, wrong-headed, careless, or downright bizarre comment, and sometimes several. Perhaps a claim about the current state of knowledge that’s just plain wrong, a misunderstanding that can only come from not reading the application properly, and/or criticising it on the spurious grounds of not being the project that they would have done. These apples are fine as far as they go, but they should really taste of oranges. I like oranges.

Don’t get me wrong – most referees’ reports that I see are careful, conscientious, and insightful, but it’s those misconceived criticisms that unsuccessful applicants will remember. Even ahead of the valid ones. And sometimes they will conclude that its those wrong criticisms that are the reason for not getting funded. Everything else was positive, so that one negative review must be the reason, yes? Well, maybe not. It’s also possible that that bizarre comment was discounted by the panel too, and the reason that your project wasn’t funded was simply that the money ran out before they reached your project. But we don’t know. I really, really, really want to believe that that’s the case when referees write that a project is “too expensive” without explaining how or why. I hope the panel read our carefully constructed budget and our detailed justification for resources and treat that comment with the fECing contempt that it deserves.

Fortunately, the ESRC have announced changes to procedures which allow not only a right of reply to referees, but also to communicate the final grade awarded. This should give a much stronger indication of whether it was a near miss or miles off. Of course, the news that an application was miles off the required standard may come gifted wrapped with sanctions. So it’s not all good news.

But this is where we should be heading with feedback. Funders shouldn’t be shy about saying that the application was a no-hoper, and they should be giving as much detail as possible. Not so long ago, I was copied into a lovely rejection letter, if there’s any such thing. It passed on comments, included some platitudes, but also told the applicant what the overall ranking was (very close, but no cigar) and how many applications there were (many more than the team expected). Now at least one of the comments was surprising, but we know the application was taken seriously and given a thorough review. And that’s something….

So… in conclusion…. just because your project wasn’t funded doesn’t (necessarily) mean that it wasn’t fundable. And don’t take it personally. It’s not personal. Just the business of research funding.

Outstanding researcher or Oustanding grant writer?

The Times Higher has a report on Sir Paul Nurse‘s ‘Anniversary Day’ address to the Royal Society. Although the Royal Society is a learned society in the natural rather than the social sciences, he makes an interesting distinction that seems to have – more or less unchallenged – become a piece of received wisdom across many if not all fields of research.

Here’s part of what Sir Paul had to say (my underline added)

Given this emphasis on the primacy of the individuals carrying out the research, decisions should be guided by the effectiveness of the researchers making the research proposal. The most useful criterion for effectiveness is immediate past progress. Those that have recently carried out high quality research are most likely to continue to do so. In coming to research funding decisions the objective is not to simply support those that write good quality grant proposals but those that will actually carry out good quality research. So more attention should be given to actual performance rather than planned activity. Obviously such an emphasis needs to be tempered for those who have only a limited recent past record, such as early career researchers or those with a break in their careers. In these cases making more use of face-to-face interviews can be very helpful in determining the quality of the researcher making the application.

I guess my first reaction to this is to wonder whether interviews are the best way of deciding research funding for early career researchers. Apart from the cost, inconvenience and potential equal opportunities issues of holding interviews, I wonder if they’re even a particularly good way of making decisions. When it comes to job interviews, I’ve seen many cases where interview performance seems to take undue priority over CV and experience. And if the argument is that sometimes the best researchers aren’t the best communicators (which is fair), it’s not clear to me how an interview will help.

My second reaction is to wonder about the right balance between funding excellent research and funding excellent researchers. And I think this is really the point that Sir Paul is making. But that’s a subject for another entry, another time. Coming soon!

My third reaction – and what this entry is about – is the increasingly common assumption that there is one tribe of researchers who can write outstanding applications, and another which actually does outstanding research. One really good expression of this can be found in a cartoon at the ever-excellent Research Counselling. Okay, so it’s only a cartoon, but it wouldn’t have made it there unless it was tapping into some deeper cultural assumptions. This article from the Times Higher back at the start of November speaks of ‘Dr Plods’ – for whom getting funding is an aim in itself – and ‘Dr Sparks’ – the ones who deserve it – and there seems to be little challenge from readers in the comments section below.

But does this assumption have any basis in fact? Are those who get funded mere journeymen and women researchers, mere average intellects, whose sole mark of distinction is their ability to toady effectively to remote and out-of-touch funding bodies? To spot the research priority flavour-of-the-month from the latest Delivery Plan, and cynically twist their research plans to match it? It’s a comforting thought for the increasingly large number of people who don’t get funding for their project. We’d all like to be the brilliant-but-eccentric-misunderstood-radical-unappreciated genius, who doesn’t play by the rules, cuts a few corners but gets the job done, and to hell with the pencil pushers ~~at the DA’s office in city hall~~ in RCUK’s offices in downtown Swindon. A weird kind of cross between Albert Einstein and Jimmy McNulty from ‘The Wire’.

While I don’t think anyone is seriously claiming that the Sparks-and-Plods picture should be taken literally, I’m not even sure how much truth there is in it as a parable or generalisation. For one thing, I don’t see how anyone could realistically Plod their way very far from priority to priority as they change and still have a convincing track record for all of them. I’m sure that a lot of deserving proposals don’t get funded, but I doubt very much that many undeserving proposals do get the green light. The brute fact is that there are more good ideas than there is money to spend on funding them, and the chances of that changing in the near future are pretty much zero. I think that’s one part of what’s powering this belief – if good stuff isn’t being funded, that must be because mediocre stuff is being funded. Right? Er, well…. probably not. I think the reality is that it’s the Sparks who get funded, but it’s those Sparks who are better able to communicate their ideas and make a convincing case for fit with funders’ or scheme priorities. Plods, and their ‘incremental’ research (a term that damns with faint praise in some ESRC referee’s reports that I’ve seen) shouldn’t even be applying to the ESRC – or at least not to the standard Research Grants scheme.

A share of this Sparks/Plods view is probably caused by the impact agenda. If impact is hard for the social sciences, it’s at least ten times as hard for basic research in many of the natural sciences. I can understand why people don’t like the impact agenda, and I can understand why people are hostile. However, I’ve always understood the impact agenda as far as research funding applications are concerned is that if a project has the potential for impact, it ought to, and there ought to be a good, solid, thought through, realistic, and defensible plan for bringing it about. If there genuinely is no impact, argue the case in the impact statement. Consider this, from the RCUK impact FAQ.

How do Pathways to Impact affect funding decisions within the peer review process?

The primary criterion within the peer review process for all Research Councils is excellent research. This has always been the case and remains unchanged. As such, problematic research with an excellent Pathways to Impact will not be funded. There are a number of other criteria that are assessed within research proposals, and Pathways to Impact is now one of those (along with e.g. management of the research and academic beneficiaries).

Of course, how this plays out in practice is another matter, but every indication I’ve had from the ESRC is that this is taken very seriously. Research excellence comes first. Impact (and other factors) second. These may end up being used in tie-breakers, but if it’s not excellent, it won’t get funded. Things may be different at the other Research Councils that I know less about, especially the EPSRC which is repositioning itself as a sponsor of research, and is busy dividing and subdividing and prioritising research areas for expansion or contraction in funding terms.

It’s worth recalling that it’s academics who make decisions on funding. It’s not Suits in Swindon. It’s academics. Your peers. I’d be willing to take seriously arguments that the form of peer review that we have can lead to conservatism and caution in funding decisions. But I find it much harder to accept the argument that senior academics – researchers and achievers in their own right – are funding projects of mediocre quality but good impact stories ahead of genuinely innovative, ground-breaking research which could drive the relevant discipline forward.

But I guess my message to anyone reading this who considers herself to be more of a ‘Doctor Spark’ who is losing out to ‘Doctor Plod’ is to point out that it’s easier for Sparky to do what Ploddy does well than vice versa. Ploddy will never match your genius, but you can get the help of academic colleagues and your friendly neighbourhood research officer – some of whom are uber-Plods, which in at least some cases is a large part of the reason why they’re doing their job rather than yours.

Want funding? Maximise your chances of getting it. Want to win? Learn the rules of the game and play it better. Might your impact plan be holding you back? Take advantage of any support that your institution offers you – and if it does, be aware of the advantage that this gives you. Might your problem be the art of grant writing? Communicating your ideas to a non-specialised audience? To reviewers and panel members from a cognate discipline? To a referee not from your precise area? Take advice. Get others to read it. Take their impressions and even their misunderstandings seriously.

Or you could write an application with little consideration for impact, with little concern for clarity of expression or the likely audience, and then if you’re unsuccessful, you can console yourself with the thought that it’s the system, not you, that’s at fault.

Estimating Investigator and Researcher Time on a Project

Prompted in part by an interesting discussion of the importance of the budget in establishing the overall credibility and shape of a research proposal at the ever-excellent Research Whisperer, I thought I’d put fingers-to-keyboard on the vexed issue of estimating staff time on research grant applications. Partly this is to share some of what I do and what I recommend, but mostly it’s to ask others how they approach it. Comments, thoughts, suggestions, and experiences welcome as ever in the comments below.

Estimating staff time is by some distance the hardest part of the budget, and often when I discuss this with academic colleagues, we end up in a kind of “I dunno, what do you reckon?” impasse. (I should say that it’s me who speaks like that, not them). I’ve never been involved in a research project (other than my ‘desk research’ MPhil), so I’ve really no idea, and I don’t have any particular insight into how long certain tasks will take. Career young academics, and the increasing number of academics who have never had the opportunity to lead on a research project, have little experience to draw upon, and even those who have experience seem to find it difficult. I can understand that, because I’d find it difficult too. If someone where to ask me how estimate how much time I spent over the course of a year on (say) duties related to my role as School Research Ethics Officer, I’d struggle to give them in answer in terms of a percentage FTE or in terms of a total number of working days.

It’s further complicated by the convenient fiction that academic staff work the standard five day weeks, and 37.5 hour working weeks. Even those who don’t regularly work evenings and weekends will almost certainly have flexible working patterns that make it even harder to estimate how long something will take. This means that the standard units of staff time that are most often used – total days and percentage of full time – aren’t straightforwardly convertible from one currency to the other. To make matters worse, some European funding sources seem to prefer ‘person months’. Rather than the standard working week, I think this probably reflects the reality of academic work in many institutions:

The response to my question about how much time the project will take is often answered with another question. What will the funder pay for? What looks right? What feels right? Longer, thinner project, or shorter and more intensive? The answer is always and inevitably, ‘it depends’. It depends upon what is right for the project. A longer less intensive project might make sense if you have to wait for survey responses to come in, or for other things to happen. On the other hand, if it’s something that you can and want to work on intensively, go for it. But the project has to come first. What would it take to properly take on this research challenge?

Often the answer is a bit hand-wavy and guesstimate-ish. Oh, I don’t know….say… about, two days per week? Two and a half? Would they fund three? This is a generally a good starting point. What’s not a good starting point is the kind of fifteen-minutes-every-Tuesday approach, where you end up with a ‘salami project’ – a few people involved, but sliced so thinly it’s hard to see how anything will get done. This just isn’t credible, and it’s very unlikely to get funded by the likes of the ESRC (and other funders too, I’m sure) if they don’t believe that you’ll possibly be able to deliver for the amount of money (time) you’re asking for. Either they’ll conclude that you don’t understand what your own research requires (which is fatal for any chance of funding), or they’ll think that you’re trying to pull a fast one in terms of the value for money criterion. And they won’t like that much either.

The other way to make sure that you’re not going to get funded is to go the other way and become greedy. I’ve noticed that – oddly – academics seldom over-estimate their own time, but over-estimate the amount of researcher time required. I’ve seen potential bids before now that include a heavy smattering of research associates, but no clear idea what it actually is they’ll be doing all day, other than doing stuff that the lead investigator doesn’t want to do. In a UK context, overheads are calculated on the basis of investigator and researcher time, so including researchers is doubly expensive. One way round this that I often recommend is to ensure that what’s required is actually a research associate (who attracts overheads) rather than an academic-related project manager or administrator (who doesn’t). But if it’s hard to estimate how long it will take you to do any given task, it’s doubly hard to estimate how long it will take someone else – usually someone less experienced and less skilled – and perhaps from a cognate sub-discipline rather than from your own.

My usual advice is for would-be principal investigators to draw up a table showing the various phases of the project as rows, and project staff as columns. In addition to the main phases of the research, extra rows should be added for the various stages of dissemination and impact activity; for project coordination with colleagues; for line management of any researchers or administrative/managerial staff; for professional development where appropriate. Don’t forget travelling time associated with meetings, fieldwork, conferences etc. The main thing that I usually see underestimated is project management time. As principal investigator, you will have to meet reasonably regularly with your finance people, and you’ll have to manage and direct the project research associates. Far too many academics seem to see RAs as clones who will instinctively know what to do and don’t need much in the way of direction, advice, or feedback.

Once this is done, I suggest adding up the columns and then working backwards from those total numbers of project days to a percentage FTE, or days per week, or whatever alternative metric you prefer. In the UK, the research councils assuming that 220 days = 1 working year, so you can use this to calculate the percentage of full time. The number that you come out with should feel intuitively about right. If it doesn’t, then something has probably gone wrong. Go back to the table, and adjust things a little. By going to and fro, from the precision of the table to the intuition of the percentage of time, you should reach what my philosophical hero and subject of my thesis, John Rawls, called ‘reflective equilibrium’. Though he wasn’t talking about investigator time. Once you’re happy with the result, you should probably use the figure from the table, and you should certainly consider putting the table in full in the ‘justification for resources’ section.

Something I’m starting at the moment as part of the end-of-project review process is to go back to the original estimates of staff time, and to get a sense from the research team about how accurate they were, and what they would estimate differently next time, if anything. The two things that have come out strongly from this so far I’ve outlined above – managing staff and project administration – but I’ll be looking for others.

So…. over to you. How do you estimate researcher and investigator time? Have you been involved in a funded project? If so, what did you miss in your forecasts (if anything)? What would you do differently next time?