{"id":440,"date":"2012-03-20T01:00:11","date_gmt":"2012-03-20T01:00:11","guid":{"rendered":"https:\/\/socialscienceresearchfunding.co.uk\/?p=440"},"modified":"2012-03-20T08:11:05","modified_gmt":"2012-03-20T08:11:05","slug":"responding-to-referees","status":"publish","type":"post","link":"https:\/\/socialscienceresearchfunding.co.uk\/?p=440","title":{"rendered":"Responding to Referees"},"content":{"rendered":"
\"\"
Preliminary evidence appears to show that this approach to responding to referees is - on balance - probably sub-optimal. (Photo by Tseen Khoo)<\/figcaption><\/figure>\n

This post is co-authored by Adam Golberg of Cash for Questions<\/a> (UK), and Jonathan O\u2019Donnell and Tseen Khoo of The Research Whisperer<\/a> (Australia).<\/em><\/p>\n

It arises out of a comment that Jonathan made about understanding and responding to referees on one of Adam\u2019s posts about what to do if your grant application is unsuccessful<\/a>. This seemed like a good topic for an article of its own, so here it is, cross-posted to our respective blogs.<\/em><\/p>\n

A quick opening note on terminology: We use \u2018referee\u2019 or \u2018assessor\u2019 to refer to academics who read and review research grant applications, then feed their comments into the final decision-making process. Terminology varies a bit between funders, and between the UK and Australia. We\u2019re not talking about journal referees, although some of the advice that follows may also apply there.<\/em><\/p>\n

————————————-<\/p>\n

There are funding schemes that offer applicants the opportunity to respond to referees\u2019 comments. These responses are then considered alongside the assessors\u2019 scores\/comments by the funding panel. Some funders (including the Economic and Social Research Council [ESRC<\/a>] in the UK) have a filtering process before this point, so if you are being asked to respond to referees\u2019 comments, you should consider it a positive sign as not all applications get this far. Others, such as the Australian Research Council (ARC<\/a>), offer you the chance to write a rejoinder regardless of the level of referees\u2019 reports.<\/p>\n

If the funding body offers you the option of a response, you should consider your response as one of the most important parts of the application process. \u00a0A good response can draw the sting from criticisms, emphasise the positive comments, and enhance your chances of getting funding. \u00a0A bad one can doom your application.<\/p>\n

And if you submit no response at all? That can signal negative things about your project and research team that might live on beyond this grant round.<\/p>\n

The first thing you might need to do when you get the referees\u2019 comments about your grant application is kick the (imaginary) cat.* This is an important process. Embrace it.<\/p>\n

When that\u2019s out of your system, here are four strategies for putting together a persuasive response and pulling that slaved-over application across the funding finish line.<\/p>\n

1. Attitude and tone<\/h3>\n

Be nice. \u00a0Start with a brief statement thanking the anonymous referees for their careful and insightful comments, even if actually you suspect some of them are idiots who haven\u2019t read your masterpiece properly. Think carefully about the tone of the rest of the response as well. \u00a0You\u2019re aiming for calm, measured, and appropriately assertive. \u00a0There\u2019s nothing wrong with saying that a referee is just plain wrong on a particular point, but do it calmly and politely. \u00a0If you\u2019re unhappy about a criticism or reviewer, there\u2019s a good chance that it will take several drafts before you eliminate all the spikiness from the text. \u00a0If it makes you feel better (and it might), you can write what you really think in the tone that you think it in but, whatever you do, don\u2019t send that version! This is the version that may spontaneously combust from the deadly mixture of vitriol and pleading contained within.<\/p>\n

Preparing a response is not about comprehensively refuting every criticism, or establishing intellectual superiority over the referees. You need to sift the comments to identify the ones that really matter. What are the criticisms (or backhanded compliments) that will harm your cause? Highlight those and answer them methodically (see below). Petty argy-bargy isn\u2019t worth spending your time on.<\/p>\n

2. Understanding and interpreting referees’ comments<\/h3>\n

One UK funder provides referee report templates that invite the referees to state their level of familiarity with the topic and even a little about their research background, so that the final decision-making panel can put their comments into context. This is a great idea, and we would encourage other funding agencies to embrace it.<\/p>\n

Beyond this volunteered information (if provided), never assume you know who the referee is, or that you can infer anything else about them because you could be going way off-base with your rant against econometricians who don\u2019t \u2018get\u2019 sociological work. If there\u2019s one thing worse than an ad hominem response, it\u2019s an ad hominem response aimed at the wrong target!<\/p>\n

One exercise that you might find useful is to produce a matrix listing all of the criticisms, and indicating the referee(s) who made those objections. As these reports are produced independently, the more referees make a particular point, the more problematic it might be. \u00a0This tabled information can be sorted by section (e.g. methodology, impact\/dissemination plan, alternative approaches). You can then repeat the exercise with the positive comments that were made. While assimilating and processing information is a task that academics tend to be good at, it\u2019s worth being systematic about this because it\u2019s easy to overlook praise or attach too much weight to objections that are the most irritating.<\/p>\n

Also, look out for, and highlight, any requests that you do a different project. Sometimes, these can be as obvious as \u201cyou should be doing Y instead\u201d, where Y is a rather different project and probably closer to the reviewer\u2019s own interests. These can be quite difficult criticisms to deal with, as what they are proposing may be sensible enough, but not what you want to do. \u00a0In such cases, stick to your guns, be clear what you want to do, and why it\u2019s of at least as much value as the alternative proposal.<\/p>\n

Using the matrix that you have prepared, consider further how damaging each criticism might be in the minds of the decision makers. \u00a0Using a combination of weight of opinion (positive remarks on a particular point minus criticisms) and multiplying by potential damage, you should now have a sense of which are the most serious criticisms.<\/p>\n

Preparing a response is not a task to be attempted in isolation. You should involve other members of your team, and make full use of your research support office and senior colleagues (who are not directly involved in the application). Take advantage of assistance in interpreting the referees\u2019 comments, and reviewing multiple drafts of your response.<\/p>\n

Don\u2019t read the assessor reports by themselves; you should also go back to your whole application, several times if necessary. It has probably been some time since you submitted the application, and new eyes and a bit of distance will help you to see the application as the referees may have seen it. You could pinpoint the reasons for particular criticisms, or misunderstandings that you assumed they made. While their criticisms may not be valid for the application you thought you wrote, they may very well be so for the one that you actually submitted.<\/p>\n

3. The response<\/h3>\n

You should plan to use the available space in line with the exercise above, setting aside space for each criticism in proportion to its risk of stopping you getting funded.<\/p>\n

Quibbles about your budgeted expenditure for hotel accommodation are insignificant compared to objections that question your entire approach, devalue your track-record, invalidate your methodology, or claim that you\u2019re adding little that\u2019s new to the sum of human knowledge. So, your response should:<\/p>\n