{"id":375,"date":"2012-02-06T00:18:52","date_gmt":"2012-02-06T00:18:52","guid":{"rendered":"https:\/\/socialscienceresearchfunding.co.uk\/?p=375"},"modified":"2012-02-06T00:18:52","modified_gmt":"2012-02-06T00:18:52","slug":"a-partial-qualified-cautious-defence-of-the-research-excellence-framework-ref","status":"publish","type":"post","link":"https:\/\/socialscienceresearchfunding.co.uk\/?p=375","title":{"rendered":"A partial, qualified, cautious defence of the Research Excellence Framework (REF)"},"content":{"rendered":"
\"\"
No hilarious visual puns on REF \/ Referees from me....<\/figcaption><\/figure>\n

There’s been a constant stream of negative articles about the Research Excellence Framework (for non-UK readers, this is the “system for assessing the quality of research in UK higher education institutions”) over the last few months, and two more have appeared recently (from David Shaw, writing in the Times Higher<\/a>, and from Peter Wells on the LSE Impact Blog<\/a>)\u00a0 which have prompted me to respond with something of a defence of the Research Excellence Framework.<\/p>\n

One crucial fact that I left out of the description of the REF in the previous paragraph is that “funding bodies intend to use the assessment outcomes to inform the selective allocation of their research funding to HEIs, with effect from 2015-16”.\u00a0 And I think this is a fact that’s also overlooked by some critics.\u00a0 While a lot of talk is about prestige and ‘league tables’, what’s really driving the process is the need to have some mechanism for divvying out the cash for funding research – QR funding.\u00a0 We could most likely do without a “system for assessing the quality of research” across every discipline and every UK university in a single exercise using common criteria, but we can’t do without a method of dividing up the cake as long as there’s still cake to share out.<\/p>\n

In spite of the current spirit of perpetual revolution in the sector, money\u00a0 is still paid (via HEFCE) to universities for research, without much in the way of strings attached.\u00a0 This basic, core funding is one half of the dual funding system for research in the UK – the other half being funding for individual research projects and other activities through the Research Councils.\u00a0 What universities do with their QR funding varies, but I think typically a lot of it is in staff salaries, so that the number of staff in any given discipline is partly a function of teaching income and research income.<\/p>\n

I do have sympathy for some of the arguments against the REF, but I find myself returning to the same question – if not this way, then how?\u00a0<\/em><\/p>\n

It’s unfair to expect anyone who objects to any aspect of the REF to furnish the reader with a fully worked up alternative, but constructive criticism must at least point the way.\u00a0 One person who doesn’t fight shy of coming up with an alternative is Patrick Dunleavy<\/a>, who has argued for a ‘digital census’ involving the use of citation data as a cheap, simple, and transparent replacement for the REF.\u00a0 That’s not a debate I feel qualified to participate in, but my sense is that Dunleavy’s position on this is a minority one in UK academia.<\/p>\n

In general, I think that criticisms of the REF tend to fall into the following broad categories.\u00a0 I don’t claim to address decisively every last criticism made (hence the title), but for what it’s worth, here are the categories that I’ve identified, and what I think the arguments are.<\/p>\n

1.\u00a0 Criticism over details<\/p>\n

The REF team have a difficult balancing act.\u00a0 On the one hand,\u00a0 they need rules which are sensitive to the very real differences between different academic disciplines.\u00a0 On the other, fairness and efficiency calls for as much similarity in approach, rules, and working methods as possible between panels.\u00a0 The more differences between panels, the greater the chances of confusion and of mistakes being made in the process of planning and submitting REF returns which could seriously affect both notional league table placing and cold hard cash.\u00a0 The more complicated the process, the greater the transaction costs.\u00a0\u00a0 Which brings me onto the second balancing act.\u00a0 On the one hand, it needs to be a rigorous and thorough process, with so much public money at stake.\u00a0 On the other hand, it needs to be lean and efficient, minimising the demands on the time of institutions, researchers, and panel members.\u00a0\u00a0 This isn’t to say that the compromise reached on any given point between particularism and uniformity, and between rigour and efficiency, is necessarily the right one, of course.\u00a0 But it’s not easy.<\/p>\n

2.\u00a0 Impact<\/p>\n

The use of impact at all.\u00a0 The relative weighting of impact.\u00a0 The particular approach to impact.\u00a0 The degree of uncertainty about impact.\u00a0 It’s a step into the unknown for everyone, but I would have thought that the idea that there be some notion of impact, some expectation that where academic research makes a difference in the real world, we should ensure it does so.\u00a0 I have much more sympathy for some academic disciplines than others as regards objections to the impact agenda.\u00a0 Impact is really a subject for a blog post in itself, but for now, it’s worth noting that it would be inconsistent to argue against the inclusion of impact in the REF and also<\/em> to argue that it’s too narrow in terms of what it values and what it assesses.<\/p>\n

3.\u00a0 Encouraging game playing<\/p>\n

While it’s true that the REF will encourage game playing in similar (though different) ways to its predecessors, I can’t help but think this is inevitable and would also be true of every possible alternative method of assessment.\u00a0 And what some would regard as gaming, others would regard as just doing what is asked of them.<\/p>\n

One particular ‘game’ that is played – or, if you prefer, strategic decision that is made – is about what the threshold to submit is.\u00a0 It’s clear that there’s no incentive to include those whose outputs are likely to fall below the minimum threshold for attracting funding.\u00a0 But it’s common for some institutions for some disciplines to have a minimum above this, with one eye not only on the QR funding, but also on league table position.\u00a0 There are two arguments that can be made against this.\u00a0 One is that QR funding shouldn’t be so heavily concentrated on the top rated submissions and\/or that more funding should be available.\u00a0 But that’s not an argument against the REF as such.\u00a0 The other is that institutions should be obliged to submit everyone.\u00a0 But the costs of doing so would be huge, and it’s not clear to me what the advantages would be – would we really get better or more accurate results with which to share out the funding.\u00a0 Because ultimately the REF is not about individuals, but institutions.<\/p>\n

4. Perverse incentives<\/p>\n

David Shaw, in the Times Higher, sees a very dangerous incentive in the REF.<\/p>\n

REF incentivises the dishonest attribution of authorship. If your boss asked you to add someone’s name to a paper because otherwise they wouldn’t be entered into the REF, it could be hard to refuse.<\/p><\/blockquote>\n

I don’t find this terribly convincing.\u00a0 While I’m sure that there will be game playing around who should be credited with co-authored publications, I’d see that as acceptable in a way that the fraudulent activity that Shaw fears (but stresses that he’s not experienced first-hand) just isn’t.\u00a0 There is opportunity for\u00a0 – and temptations to – fraud, bad behaviour and misconduct in pretty much everything we do, from marking students’ work to reporting our student numbers to graduate destinations.\u00a0 I’m not clear how that makes any of these activities ‘unethical’ in the way his article seems to argue.\u00a0 Fraud is low in our sector, and if anyone does commit fraud, it’s a huge scandal and heads roll.\u00a0 It ruins careers and leaves a long shadow over institutions.\u00a0 Even leaving aside the residual decency and professionalism that’s the norm in our sector, it would be a brave Machiavellian Research Director who would risk attempting this kind of fraud.\u00a0 To make it work, you need the cooperation and the silence of two academic researchers for every single publication.\u00a0 Risk versus reward – just not worth it.<\/p>\n

Peter Wells, on the LSE blog, makes the point that the REF acts as an active disincentive for researchers to co-author papers with colleagues at their own institution, as only one can return the output to the REF.\u00a0 That’s an oversimplification, but it’s certainly true that there’s active discouragement of the submission of the same output multiple times in the same return.\u00a0 There’s no such problem if the co-author is at another institution, of course.\u00a0 However, I’m not convinced that this theoretical disincentive makes a huge difference in practice.\u00a0 Don’t academics co-author papers with the most appropriate colleague, whether internal or external?\u00a0 How often – really – does a researcher chose to write something with a colleague at another institution rather than a colleague down the corridor?\u00a0 For REF reasons alone?\u00a0 And might the REF incentive to include junior colleagues as co-authors that Shaw identifies work in the other direction, for genuinely co-authored pieces?<\/p>\n

In general, proving the theoretical possibility of a perverse incentive is not sufficient to prove its impact in reality.<\/p>\n

5.\u00a0 Impact on morale<\/p>\n

There’s no doubt that the REF causes stress and insecurity and can add significantly to the workload of those involved in leading on it.\u00a0 There’s no doubt that it’s a worrying time, waiting for news of the outcome of the R&R paper that will get you over whatever line your institution has set for inclusion.\u00a0 I’m sure it’s not pleasant being called in for a meeting with the Research Director to answer for your progress towards your REF targets, even with the most supportive regime.<\/p>\n

However…. and please don’t hate me for this…. so what?\u00a0 I’m not sure that the bare fact that something causes stress and insecurity is a decisive argument.\u00a0 Sure, there’s a prima facie<\/em> for trying to make people’s lives better rather than worse, but that’s about it.\u00a0 And again, what alternative system which would be equally effective at dishing out the cash while being less stressful?\u00a0 The fact is that every job – including university jobs – is sometimes stressful and has downsides rather than upsides.\u00a0 Among academic staff, the number one stress factor I’m seeing at the moment is marking, not the REF.<\/p>\n

6.\u00a0 Effect on HE culture<\/p>\n

I’ve got more time for this argument than for the stress argument, but I think a lot of the blame is misdirected.\u00a0 Take Peter Wells’ rather utopian account of what might replace the REF:<\/p>\n

For example, everybody should be included, as should all activities.\u00a0 It is partly by virtue of the \u2018teaching\u2019 staff undertaking a higher teaching load that the research active staff can achieve their publications results; without academic admissions tutors working long hours to process student applications there would be nobody to receive research-led teaching, and insufficient funds to support the University.<\/p><\/blockquote>\n

What’s being described here is not in any sense a ‘Research Excellence Framework’.\u00a0 It’s a much broader ‘Academic Excellence Framework’, and that doesn’t strike me as something that’s particularly easy to assess.\u00a0 How on earth could we go about assessing absolutely everything that absolutely everyone does?\u00a0 Why would we give out research<\/em> cash according to how good an admissions tutor someone is?<\/p>\n

I suspect that what underlies this – and some of David Shaw’s concerns as well – is a much deeper unease about the relative prestige and status attached to different academic roles: the research superstar; the old fashioned teaching and research lecturer; those with heavy teaching and admin loads who are de facto<\/em> teaching only; and those who are de jure<\/em> teaching only.\u00a0 There is certainly a strong sense that teaching is undervalued – in appointments, promotions, in status, and in other ways.\u00a0 Those with higher teaching and admin workloads do enable others to research in precisely the way that Shaw argues, and respect and recognition for those tasks is certainly due.\u00a0 And I think the advent of increased tuition fees is going to change things, and for the better in the sense of the profile and status of excellent teaching.<\/p>\n

But I’m not sure why any of these status problems are the fault of the REF.\u00a0 The REF is about assessing research excellence and giving out the cash accordingly.\u00a0 If the REF is allowed to drive everything, and non-inclusion is such a badge of dishonour that the contributions of academics in other areas are overlooked, well, that’s a serious problem.\u00a0 But it’s an institutional one, and not one that follows inevitably from the REF.\u00a0 We could completely change the way the REF works tomorrow, and it will make very little difference to the underlying status problem.<\/p>\n

It’s not been my intention here to refute each and every argument against the REF, and I don’t think I’ve even addressed directly all of Shaw and Wells’ objections.\u00a0 What I have tried to do is to stress the real purpose of the REF, the difficulty of the task facing the REF team, and make a few limited observations about the kinds of objections that have been put forward.\u00a0 And all without a picture of Pierluigi Collina<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

There’s been a constant stream of negative articles about the Research Excellence Framework (for non-UK readers, this is the “system for assessing the quality of research in UK higher education institutions”) over the last few months, and two more have appeared recently (from David Shaw, writing in the Times Higher, and from Peter Wells on … Continue reading “A partial, qualified, cautious defence of the Research Excellence Framework (REF)”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[6,18,20,19],"tags":[],"class_list":["post-375","post","type-post","status-publish","format-standard","hentry","category-funding","category-research-costs","category-research-impact","category-university-culture"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t