Initial Reactions to HEFCE’s ‘Initial decisions on REF 2021’

This lunchtime HEFCE have announced some more “Initial Decisions” on REF 2021, which I’ve summarised below.

Slightly frustratingly, the details are scattered across a few documents, and it’s easy to miss some of them. There’s an exec summary,  a circular letter (which is more of a rectangle, really), the main text of the report that can be downloaded from the bottom of the exec summary page (along with an annex listing UoAs and further particulars for panel chair roles)… and annex A on a further consultation staff return and output portability, downloadable from the bottom of the circular letter page.

I’ve had a go at a quick summary, by bullet point theme rather than in the order they appear, or in a grand narrative sweep. This is one of my knee-jerk pieces, and I’ve added a few thoughts of my own. But it’s early days, and possibly I’ve missed something or misunderstood, so please let me know.

Outputs

  • Reserve output allowed where publication may not appear in time
  • Worth only 60% of total mark this time (see scoring system)

I think the reduction in the contribution of outputs to the overall mark (at the expense of impact) is probably what surprised me most, and I suspect this will be controversial. I think the original plan was for environment to be downgraded to make way, but there’s a lot more demanded from the environment statement this time (see below) so it’s been protected. Great to have the option of submitting an insurance publication in case one of the in-press ones doesn’t appear by close of play.

Panels/Units of Assessment

  • Each sub-panel to have at least one appointed member for interdisciplinary research “with a specific role to ensure its equitable assessment”. New identifier/flag for interdisciplinary outputs to capture
  • Single UoA for engineering, multiple submissions allowed
  • Archaeology split from Geography and Environmental studies – now separate
  • Film and Screen Studies to be explicitly included in UoA 33 with Dance, Drama, Performing Arts
  • Decisions on forensic science and criminology (concerns about visibility) due in Autumn
  • Mapping staff to UoAs will be done by institutions, not HESA cost centres, but may ask for more info in the event of any “major variances” from HESA data.

What do people think about a single UoA for engineering? That’s not an area I support much. Is this just tidying up, or does this has greater implications? Is it ironic that forensic science and criminology have been left a cop show cliff-hanger ending?

Environment

  • Expansion of Unit of Assessment environment section to include sections on:
    • Structures to support interdisciplinary research
    • Supporting collaboration with “organisations beyond higher education”
    • Impact template will now be in the environment element
    • Approach to open research/open access
    • Supporting equality and diversity
  • More quant data in UoA environment template (we don’t know what yet)
  • Standard Institution level information
  • Non-assessed invite only pilot for institution level environment statement
  • Expansion of environment section is given as a justification for maintaining it at 15% of score rather than reducing as expected.

The inclusion of a statement about support for interdisciplinary work is interesting, as this moves beyond merely addressing justifiable criticism about the fate of interdisciplinary research (see the welcome addition to each UoA of an appointed ‘Member for Interdisciplinarity’ above). This makes it compulsory, and an end in itself. This will go down better in some UoAs than others.

Impact

  • Institutional level impact case studies will be piloted, but not assessed
  • Moves towards unifying definitions of “impact” and “academic impact” between REF and Research Councils – both part of dual funding system for research
  • Impact on teaching/curriculum will count as impact – more guidance to be published
  • Underpinning work “at least equivalent to 2*” and published between 1st Jan 2000 and 31st Dec 2020. Impact must take place between 1st Aug 2013 and 31st July 2020
  • New impact case study template, more questions asked, more directed, more standardised, more “prefatory” material to make assessment easier.
  • Require “routine provision of audit evidence” for case study templates, but not given to panel
  • Uncertain yet on formula for calculating number of case study requirements, but overall “should not significantly exceed… 2014”. Will be done on some measure of “volume of activity”, possibly outputs
  • Continuation of case studies from 2014 is allowed, but must meet date rules for both impact and publication, need to declare it is a continuation.
  • Increased to 25% of total score

And like a modern day impact superhero, he comes Mark Reed aka Fast Track Impact with a blog post of his own on the impact implications of the latest announcement. I have to say that I’m pleased that we’re only having a pilot for institutional case studies, because I’m not sure that’s a go-er.

Assessment and Scoring system

  • Sub-panels may decide to use metrics/citation data, but will set out criteria statements stating whether/how they’ll use it. HEFCE will provide the citation data
  • As 2014, overall excellence profile, 3 sub-profiles (outputs, impact, environment)
  • Five point scale from unclassified to 4*
  • Outputs 60, Impact 25, Environment 15. Increase of impact to 25, but as extra environment info sought, has come at the expense of outputs.

There was some talk of a possible necessity for a 5* category to be able to differentiate at the very top. but I don’t think this gained much traction.

But on the really big questions… further consultation (deadline 29th Sept):

There’s been some kicking into the short grass, but things are looking a bit clearer…

(1) Staff submission:

All staff “with a significant responsibility to undertake research” will be submitted, but “no single indicator identifies those within the scope of the exercise”.  Institutions have the option of submitting 100% of staff who meet the core eligibility requirement OR come up with a code of practice that they’ll use to decide who is eligible. Audit-able evidence will be required and Institutions can choose different options for different UoAs.

Proposed core eligibility requirements – staff must meet all of the following:

  • “have an academic employment function of ‘research only’ or ‘teaching and research’
  • are independent researchers [i.e. not research assistants unless ‘demonstrably’ independent]
  • hold minimum employment of 0.2 full time equivalent
  • have a substantive connection to the submitting institution.”

I like this as an approach – it throws the question back to universities, and leaves it up to them whether they think it’s worth the time and trouble running an exercise in one or more UoAs. And I think the proposed core requirements look sensible, and faithful to the core aim which is to maximise the number of researchers returned and prevent the hyper selectivity game being played.

(2) Transition arrangements for non-portability of publications.

HEFCE are consulting on either:

(a) “The simplified model, whereby outputs would be eligible for return by the originating institution (i.e. the institution where the research output was demonstrably generated and at which the member of staff was employed) as well as by the newly employing institution”.
or
(b) “The hybrid approach, with a deadline (to be determined), after which a limited number of outputs would transfer with staff, with eligibility otherwise linked to the originating institution. (This would mean operating two rules for portability in this exercise: the outputs of staff employed before the specified date falling under the 2014 rules of full portability; outputs from staff employed after this date would fall under the new rules.)”

I wrote a previous post on portability and non-portability when the Stern Review was first published, which I still think is broadly correct.

I wonder how simple the simplified model will be… if we end having to return n=2 publications, and choosing those publications from a list of everything published by everyone while they worked here. But it’s probably less work than having a cut off date.

More to follow….