An Impact Statement: Part 1: Impact and the REF

If your research leads directly or indirectly to this, we'll be having words.....

Partly inspired by a twitter conversation and partly to try to bring some semblance of order my own thoughts, I’m going to have a go about writing about impact.  Roughly, I’d argue that:

  • The impact agenda is – broadly – a good thing
  • Although there are areas of uncertainty and plenty of scope for collective learning, I think the whole area is much less opaque than many commentators seem to think
  • While the Research Councils and the REF have a common definition of ‘impact’, they’re looking at it from different ends of the telescope.

This post will come in three parts.  In part one, I’ll try to sketch a bit of background and say something position of impact in the REF.  In part two, I’ll turn to the Research Councils and think about how ‘impact’ differs from previous different – but related – agendas.  In part three, I’ll pose some questions that are puzzling me about impact and test my thinking with examples.

Why Impact?

What’s going on?  Where’s it come from?  What’s driving it?  I’d argue that to understand the impact agenda properly, it’s important to first understand the motivations.  Broadly speaking, I think there are two.

Firstly, I think it arises from a worry about a gap between academic research and those who might find it useful in some way.  How may valuable insights of various kinds from various disciplines have never got further than an academic journal or conference?  While some academics have always considered providing policy advice or writing for practitioner journals as a key part of their role as academics, I’m sure that’s not universally true.  I can imagine some of these researchers now complaining like music obsessives that they were into impact before anyone else and it sold out and went all mainstream.  As I’ve argued previously, one advantage of the impact agenda is that it gives engaged academics some long overdue recognition, as well as a much greater incentive for others to become involved in impact related activities.

Secondly, I think it’s about finding concrete, credible, and communicable evidence of the importance and value of academic research.  If we want to keep research funding at current levels, there’s a need to show return on investment and that the taxpayer is getting value for money.  Some will cringe at the reduction of the importance and value of research to such crude and instrumentalist terms, but we live in a crude and instrumentalist age.  There is an overwhelming case for the social and economic benefits of research, and that case must be made.  Whether we like it or not, no government of any likely hue is just going to keep signing the cheques.  The champions of research in policy circles do not intend to go naked into the conference chamber when they fight our corner.  To what extent the impact agenda comes directly from government, or whether it’s a pre-emptive move, I’m not quite sure.  But the effect is pretty much the same.

What’s Impact in the REF?

The REF definition of impact is as follows:

140. For the purposes of the REF, impact is defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia (as set out in paragraph 143).
141. Impact includes, but is not limited to, an effect on, change or benefit to:
• the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding
• of an audience, beneficiary, community, constituency, organisation or individuals
• in any geographic location whether locally, regionally, nationally or internationally.
142. Impact includes the reduction or prevention of harm, risk, cost or other negative effects.
Assessment Framework and Guidance on Submissions
, page 26.

Paragraph 143 goes on to rule out academic impact on the grounds that it’s assessed in the outputs and environment section.  Fair enough.  More controversially, it goes on to state that “impacts on students, teaching, and other activities within the submitting HEI are excluded”.  But it’s possible to understand the reasoning.  If it were included, there’s a danger that far too impact case studies would be about how research affects teaching – and while that’s important, I don’t think we’d want it to dominate.  There’s also an argument that the link between research and teaching ought to be so obvious that there’s no need to measure it for particular reward.  In practical terms, I think it would be hard to measure.  I might know how my new theory has changed how I teach my module on (say) organisational behaviour to undergraduates, but it would be hard to track that change across all UK business schools.  I’d also worry about the possible perverse incentives on the shape of the curriculum that allowing impact on teaching might create.

The Main Panel C (the panel for most social sciences) criteria state that:

The main panel acknowledges that impact within its remit may take many forms and occur in a wide range of spheres. These may include (but are not restricted to): creativity, culture
and society; the economy, commerce or organisations; the environment; health and welfare; practitioners and professional services; public policy, law and services.
The categories used to define spheres of impact, for the purpose of this document, inevitably overlap and should not be taken as restrictive. Case studies may describe impacts which have affected more than one sphere. (para 77, pg. 68)

There’s actually a lot of detail and some good illustrations of what forms impact might take, and I’d recommend having a read.  I wonder how many academics not directly involved in REF preparations have read this?  One difficulty is finding it – it’s not the easiest document to track down.  For my non-social science reader(s), the other panel working methods can be found here.  Helpfully, nothing on that page will tell you which panel is which, but (roughly) Panel A is health and life sciences; B is natural sciences, computers, maths and engineering; C is social science; and D humanities.  Each panel criteria document has a table with examples of impact.

What else do we know about the place of impact in the REF?  Well, we know that impact has to have occurred in the REF period (1 January 2008 to 31 July 2013) and that impact has to be underpinned by excellent research (at least 2*) produced at the submitting university at some point between 1 January 1993 and 31 December 2013.  It doesn’t matter if the researchers producing the research are still at the institution – while publications move with the author, impact stays with the institution.  However, I can’t help wondering if an excessive reliance on research undertaken by departed staff won’t look too much like trading on past glories.  But probably it’s about getting the balance right.  The number of case studies required is approximately 1 per 8 FTE submitted, but see page 28 of the guidance document for a table.

Impact will have a weighting of 20%, with environment 15% and outputs (publications) 65%, and it looks likely that the weighting of impact will increase next time.  However, I wouldn’t be at all surprised if the actual contribution ends up being less than that.  If there’s a general trend that overall scores for impact are lower than that of (say) publications, then the contribution will end up being less than 20%.  My understanding is that for some units of assessment, environment was consistently rated more highly, thus de facto increasing the weighting.  Unfortunately this is just a recollection of something I read years ago, and which I can’t now find.  But if this is right, and if impact does come in with lower marks overall, we neglect environment at our peril.

Jobs in university administration

This man had hair before he started shortlisting.....

The Guardian Higher Education network recently hosted a careers clinic on ‘How to break into university administration‘, and I posted a few thoughts that I thought might be useful.  According to my referral stats for my blog, a number of visitors end up here with similar questions about both recruitment processes and what it’s like to work for a university.  I think it’s mainly my post on Academics vs University Administrators part 94 that gets those hits.  I’ve also been asked by friends and relatives for my very limited wisdom on this topic.

I also think it’s good to share this information, because one of my worries whenever I’m involved in recruiting staff is that we end up employing people who are best at writing applications and being interviewed.  In my particular line of work, that’s fine – if you can’t write a strong job application against set criteria, you probably shouldn’t be helping academics with grant applications.  But that’s the exception.

So what follows is me spilling the beans on my very limited experience of recruiting administrative staff in two institutions, both as panel chair and as an external panel member.  I’m not an HR expert.  I’m not a careers advisor.  But for what it’s worth, what follows is an edited and expanded version of what I posted on the Guardian page.

——————————————————————————

When an administrative job is advertised, a document called a ‘person specification’ is drawn up. Formats vary, but usually this is a list of skills, attributes, experiences, and attitudes that are either classed as “essential” or “desirable”. Often it’ll say which part of the recruitment process these will be examined (application, aptitude test, or interview).

In all of the recruitment I’ve been involved in, this is an absolutely vital document. Decisions about who to short list for interview and who not to and ultimately who to appoint will be made on the basis of this person specification and justified on that basis.  And we must be able to justify our decisions if challenged.  As panel chair I was required to (briefly) explain reasons for rejection for everyone we didn’t interview, and then everyone we didn’t appoint.  I’m sure the importance of the person specification isn’t unique to universities.

To get an interview, an applicant needs to show that they meet all of the essential criteria and as many of the desirable ones as possible. My advice to applicants is that if they don’t have some of the desirable criteria, they should make the case for having something equivalent, or a plan to get that skill. For example, if a person spec lists “web design” as desirable and you can’t do it, express willingness to go on a course. For bonus points, find a course that you’d like to go on.  If you’re offered an interview, you can use the person spec to predict the interview questions – they’ll be questions aimed at getting evidence about your fit with the person spec.  You could do worse than to imagine that you’re on the interview panel and think of the questions you’d ask to get evidence about candidates’ fit with those criteria.  Chances are you won’t be a million miles off.

Unfortunately, if you don’t meet the essential criteria, it’s a waste of time applying.  You won’t get an interview.

As an applicant, your job in your application form is to make it as obvious as possible to the panel members that you meet the criteria. Back it up with evidence and at least some detail. If a criterion concerns supporting committees with minute taking and agenda prep, don’t just assert you’ve done it – say a bit about the committee, and what you did exactly, and how you did it.  Culturally, we’re not good at blowing our own trumpets, and a good and effective way round this is to just stick to the facts.  Don’t tell, show.

Panel members really appreciate it when applicants make it easy – they can just look down the person spec, look through the application, and tick, tick, tick, you’re on the potential interviewees pile.  Don’t make panel members guess or try to interpret what you say to measure it against the criteria.  There’s nothing more frustrating than an applicant who might be exactly what we need, but who hasn’t made a strong enough or clear enough case, especially about transferable skills.

Panel members can tell the difference between an application that’s being tweaked slightly and sent to every job vacancy, and one that’s been tailored for that particular vacancy. Do that, put in the effort, and you will stand out, because so many people don’t. Take the application seriously, and you’ll be taken seriously in turn. And spell check and proof read is your friend.  A good admin vacancy in a university in the current climate attract hundreds of applications.  That’s not an exaggeration.

Two other tips. One is always ask for feedback if you’re unsuccessful at interview. In every process I’ve been involved in, there’s useful feedback there for you if you want it. Even if it’s “someone else was better suited, and there’s nothing you could have done differently/better”, you still want to know that. If you were good, chances are that the university in question would like you to apply again in the future. The second is to always take up any offer of an informal conversation in advance of applying.  If you can ask sensible questions that show you’ve read all the documents thoroughly, there’s a chance that you’ll be remembered when you apply. You won’t get special treatment, but it can’t hurt.

Jobs will be advertised in a variety of places, depending on the grade and the degree of specialism needed.  Universities will have a list of current vacancies on their websites, and often use local papers for non-specialist roles.  Jobs.ac.uk is also widely used, and has customisable searches/vacancy emails, as well as some more good advice on job seeking.

Finally….. every job interview process that I’ve been involved with has attracted outstanding candidates. Some with little work experience, some with NHS or local authority admin experience, many from the private sector too. Universities are generally good employers and good places to work. It’s competitive at the best of times, and will be doubly so now.

————————————————————————–

The fact that most of you reading this not only (a) already have university jobs; and (b) know perfectly well how the recruitment process works isn’t lost on me.  But this one’s for my random google visitors.  Normal service will be resuming shortly.