The Stern Review – Publications, Portability, and Panic

Research Managers everywhere, earlier today.

The Stern Review on the future of the REF is out today, and there are any number of good summaries of the key recommendations that you can read. You could also follow the #sternreview hashtag on Twitter, or read it for yourself. It’s not particularly long, and it’s an easy read considering. The first point worth noting is that these are recommendations, not final policy, and they’re certainly nothing like a worked up final set of guidance notes for the next REF. I won’t repeat the summary, and I won’t add much on the impact issue, which Prof Mark Reed aka @fasttrackimpact has covered already.

The issue that has set twitter ablaze is that of portability – that is, which institution gets to return an academic’s publications when she moves from one institution to another. Under the old rules, there was full portability. So if Professor Portia Bililty moved from one institution to another in the final months of a REF cycle, all of her publications would come with her, and would all be returnable by her new employer. Her old employer lost all claim. Impact was different – that remained with the institution where it was created.

This caused problems. As the report puts it

72. There is a problem in the current REF system associated with the demonstrable increase in the number of individuals being recruited from other institutions shortly before the census date. This has costs for the UK HEI system in terms of recruitment and retention. An institution might invest very significantly in the recruitment, start up and future career of a faculty member, only to see the transfer market prior to REF drastically reduce the returns to that investment. This is a distortion to investment incentives in the direction of short-termism and can encourage rent-seeking by individuals and put pressure on budgets.

There was also some fairly grubby game-playing whereby big names from outside the UK were brought in on fractional contracts for their publications alone. To be fair, I’ve heard about places where this was done for other reasons, where these big names regularly attended their new fractional employer, helped develop research culture, mentored career young researchers and published articles with existing faculty. But let’s not pretend that happened everywhere.

So there’s a problem to be solved.

Stern’s response is to say that outputs – like impact – will no longer be portable.

73. We therefore recommend that outputs should be submitted only by the institution where the output was demonstrably generated. If individuals transfer between institutions (including from overseas) during the REF period, their works should be allocated to the HEI where they were based when the work was accepted for publication. A smaller maximum number of outputs might be permitted for the outputs of staff who have left an institution through retirement or to another HEI. Bearing in mind Recommendation 2, which recommends that any individual should be able to submit up to six outputs, a maximum of three outputs from those who have left the institution in the REF period would seem appropriate.
74. HEIs hiring staff during the REF cycle would be able to include them in their staff return. But they would be able to include only outputs by the individual that have been accepted for publication after joining the institution. Disincentivising short-term and narrowly-motivated movement across the sector, whilst still incentivising long-term investment in people will benefit UK research and should also encourage greater collaboration across the system.

I have to say that my first reaction to this will be extremely positive. The poaching and gameplaying were very dispiriting, and this just seems…. fairer.

However, looking at the Twitter reaction, the response was rather different. Concern was expressed that this would make it very difficult for researchers to move institutions, and it would make it especially difficult for early career researchers. I’ve been back and forth on this, and I’m no longer convinced that this is such a problem.

Let’s play Fantasy REF Manager 2020. It’s the start of the 2016/2017 season academic year. All of the existing publications from my squad of academics are mine to return, whatever happens to them and whatever career choices they make. Let’s say that one of my promising youth players  early career researchers gets an offer for elsewhere. I can try to match or beat whatever offer she has, but whatever happens, my team gets credit for the publications she’s produced. Let’s say that she moves on, and I want to recruit a replacement, and I identify the person I want. He’s got some great publications which he can’t bring with him… but I don’t need them, because I’ve got those belonging to his predecessor. Of course, I’d be very interested in track record, but I’m appointing entirely on potential. His job is to pick up where she left off.

Might recruiting on potential actually work in favour of early career researchers? Under the old system, if I were a short termist manager, I’d probably favour the solid early-mid career plodder who can bring me a number of guaranteed, safe publications, rather than someone who is much longer on promise but shorter on actual published publications. Might it also bring an end to the system where very early career researchers were advantaged just by having *any* bankable publications that had actually appeared?

I wonder if some early career researchers are so used to a system where they’re (unfairly) judged by the sole criterion of potential REF contribution that they’re imagining a scenario where they – and perhaps they alone – are being prevented from using the only thing that makes them employable. Institutions with foresight and with long term planning have always recruited on the basis of potential and other indicators and factors beyond the REF, and this change may force more of them to do that.

However, I can see a few problems that I might have as Fantasy REF Manager. The example above presumed one-in, one-out. But what if I want to increase the size of my squad through building new areas of specialism, or put together an entirely new School or Research Group? This might present more of a problem, because it’ll take much longer for me to see any REF benefits in exchange for my investment. However, rival managers would argue that the old rules meant I could do an academic-Chelsea or academic-Manchester City, and just buy all those REF benefits straight away. And that doesn’t feel right.

Another problem might be if I was worried about returning publications from people who have left. What image to it give to the REF panel if more than a certain small percentage of your returned publications are from researchers who’ve left? Would it make us look like we were trading on past glories, while in fact we’d deteriorated rapidly? Perhaps some guidance to the panels that they’re to take no account of this in assessing submissions would help here, and a clear signal that a good publication by a researcher-past has the same value as researcher-current.

Does the new system give me as the Fantasy REF Manager too much power over my players, early career or not? I’m not sure. It’s true that I have their publications in the bag, so they can’t threaten me with taking them away. But I’m still going to want to keep them on my team if I think they’re going to continue to produce work of that standard that I want in the future. If I don’t think that – for whatever reason – then I’ve no reason to want to keep them. They can still hold me to ransom, but what they’re holding over me is their future potential, not recent past glories. And to me, that seems more like an appropriate correction in the balance of power. Though… might any discrimination be more likely to be against career elderly researchers who I think are winding down? Not sure.

Of course, there are compromise positions between full portability and no portability. Perhaps a one or two year window of portability, and perhaps longer for early career researchers… though that might give some too great an advantage. That would be an improvement on the status quo, and might assuage some worries that a lot of ECRs (judging by my timeline on Twitter, anyway) have at the moment.

Even with a window, there are potential problems around game-playing. Do researchers looking for a move hold off from submitting their papers? Might they filibuster corrections and final changes? Might editors be pressurised to delay formal acceptances? Are we clear what constitutes a formal date of acceptance (open access experience suggests not)? And probably most seriously, might papers “under review” rather than papers published be the new currency?

Probably the last point is what worries me most, but I think these are relatively small issues, and I’d be worried if hiring decisions were based on such small margins. But perhaps they are.

This article is entirely knee-jerk. I’m making it up as I go along, changing my mind, being influenced. But I think that ECRs have less to worry about than many fear, and I think my tentative view is that limiting portability – either entirely, or with a narrow window – is significantly better than the current situation of unlimited portability. But I may have missed something, and I’m open to convincing.

Please feel free to tell me what I’ve missed in the comments, or tweet me.

UPDATE: 29th July AM

I’ve been following the discussion on Twitter with some interest, and I’ve been reflecting on whether or not there’s a particular issue for early career researchers. As I said earlier, I’ve been going backwards and forwards on this. Martin Eve has written an excellent post in which he argues that some of the concern may be because

“the current hiring paradigm is so geared towards REF and research it can be hard to imagine what a new hiring environment looks like”

He also makes an important point about ownership of IP, which a lot of academics don’t seem to understand.

Athene Donald has written a really interesting post in which she describes “egregious examples” of game-playing which she’s seen first hand, and anyone who doesn’t think this is a serious issue needs to read this. She also draws much-needed attention to a major benefit of the proposals – that returning everyone and having returning nx2 publications does away with all of the personal circumstances exceptions work required last time to earn the right to submit fewer than four outputs – this is difficult and time consuming for institutions, and potentially distressing for individuals. She also echoes Martin Eve’s point about some career young researchers not being able to think into a new paradigm yet by recalling her long experience of REFs and RAEs.

However, while I do – on the whole – think that some early career researchers are overreacting, perhaps not understanding that the game changes for everyone, and that appointments are now on potential, not on recent publishing history. And that this might benefit them as I argued above.

Having said that, I am now persuaded that there are good arguments for an exception to the portability rules for ECRs. My sense is that there’s a fair amount of mining and developing the PhD for publications that could be done, but after that, there has to come a stage of moving on to the next thing, adding new strings to the bow, and that that might in principle be a less productive time in terms of publishing. And although I think at least some ECR worries are misplaced, if what I’m reading on Twitter is representative, I think there’s a case for taking them seriously and doing something to assuage those fears with an exemption or limited exemption. There’s a lot that’s positive about the Stern Review, but I think the confidence of the ECR community is important in itself.

Some really interesting issues have been raised that relate to detail and to exceptions and which would have to be ironed out later, but are worth consideration. Can an institution claim the publications of a teaching fellow? (I’d argue no). What happens to publications accepted when the author has two fractional (and presumably temporary) contracts? (I’d argue they can’t be claimed, certainly not if the contract is sessional). What if the author is unemployed?

One argument I’ve read a few times is that there’s a strong incentive for institutions to hire from within, rather than from without. But I’m not clear why that is – in my example above, I already have any publications from internal candidates, whether or not I make an internal appointment. I can’t have the publications of anyone from outside – so it’s a case of the internal candidates future publications (plus broader contribution, but let’s take that as read) versus the external candidate’s. I think that sounds like a reasonably level playing field, but perhaps I’m missing something. I suppose I wouldn’t have to return publications of someone who’s left if I make an internal appointment, but if there’s no penalty (formal or informal) for this, why should I – as Fantasy REF Manager -care? If there were portability, I’d be choosing between the internal’s past and potential, and the external’s past and potential. That might change my calculations, depending on those publications – though actually if the internal’s publications were co-authored with existing faculty I might not mind if they go. So…. yes, there is a whole swamp of unintended consequences here, but I’m not sure whether allowing ECR portability helps any.

14 thoughts on “The Stern Review – Publications, Portability, and Panic”

  1. I think there’s a big difference between linking an output to an institution for REF purposes and the institution “owning” the underlying research.

    Most of the Twitter debate is people arguing that they have produced research “in their own time” or “on maternity leave”. But this overlooks other forms of support universities provide. Most academics I know use university-provided computers and software, databases, and library facilities even for research outputs they “write at home”. It doesn’t seem unreasonable for the REF to compensate universities for such investments. It doesn’t mean the “own” anything. It does mean that academics can’t play the transfer market in the same way.

    Ultimately, I think this is what motivates much of the twitterati.

    The ECR issue, for instance, is a complete red herring fanned by self-interest. ECRs (myself included) are always worried about their employment prospects. That’s a given. But if the recruitment market is based on potential to produce research rather than ‘buying in’ past success – this must benefit ECRs. They are comparatively cheap and should have a stack of research waiting to be published.

    1. Good points. I’d certainly agree about there being confusion between who gets the right to “return” a publication for the REF, and “ownership”… though I think a lot of people might get a shock if they checked their contracts as regards ownership of IP.

      I’ve also been a bit disappointed by the way that institutional help and support and investment for researchers (not least in paying their salaries) has been so quickly dismissed by some. But it’s obvious that a lot of ECRs think (almost certainly rightly) that they’re being pretty shoddily treated by their current employers. And certainly I’d expect anyone on (fractional) teaching only contracts to be able to take their publications with them. It’s clearly unfair for the University of Machiavelli to claim my publications in exchange for a sessional teaching contract for a few first year tutorials and nothing else.

  2. Portabilty = social mobility.
    Under the REF 2014 rules, academics at crappy universities but with great publications managed to move to much better universities.
    This movement will be MUCH harder if the hiring paradigm moves from REF outputs to “potential”.

    REF portability is the only bargaining chip held by mid-career academics who wish to move to a better institution or be promoted quickly.

    Let’s take the case of a mid-career academic who publishes a monograph after 5-6 years of work. Ground breaking project – double weighted, four stars.
    Portability allows for this output to become the bargaining chip to get promoted to a chair either at the “home” HEI or elsewhere.
    Now take portability away: no HEI will be interested in hiring this same academic on potential: who knows whether another “super” output will come? Even if it does come….what if it takes another 5-6? The time horizon is too long.
    Concurrently, the home institution will have no reason to promote this same mid-career academic, as they have already acquired the “super” output in question for REF purposes.

    1. I guess it depends what you think the purpose of the REF is.

      If, above all else, you take it to be a mechanism to secure “quick promotion” and increased pay for individual academics in the absence of more suitable HR practices then fine. If you want to call this social mobility feel free. I don’t think it is. I think it is a “wealth effect” whereby academics with social capital and networks get their name on things just because of their name.

      But, if you take it (as I assumed everyone did) to be a mechanism to assess the research output of units of assessment/departments/universities and to distribute funding to compensate departments that have invested successfully in their research staff, then things change.

      Let’s consider your example from this perspective. An unfashionable uni has taken a risk on an ECR. They have supported them as best they could and given them a permanent post in the hope they will have a long term relationship. The uni lacks funding for a full research infrastructure and have high teaching loads but they try to support their staff. One ECR produces a ground breaking monograph. A success for them all.

      UNTIL… The ECR uses this monograph to secure quick promotion at a higher ranked institution. When it comes to the next REF, the lower ranked institution has little to return and its status as a low ranked institution is confirmed. Heads roll. Conditions get worse. They reduce their research capacity and hire more fractional and teaching only academics. The higher ranked institution look like they are brilliant. But post REF, they worry they have increased their staff costs too much and seek to claw it back by removing incentives and freezing promotions for several years. And the whole cycles continues.

      What is the point of this. How does it help to assess the performance of either university? Portability completely undermines REF. There is no reason for it if we want to assess how well UoAs, departments and universities are performing. It means all REF does is reward departments for hiring people not developing them. But no REF = portability is worthless. This is why so many worried academics are focusing on this issue above everything else in Stern.

      I would rather we talk separately about how to create more effective HR practices in universities than use REF as a surrogate.

      1. Do you know what ‘social mobility’ means? and do you think the point of REF is to ensure academics are socially mobile?

  3. I think we need to be clear what the REF is for.

    If the aim is to provide a mechanism for staff to secure advanced promotion or move institutions then fine. Portability is a good thing.

    But I always assumed that the point was to assess universities’ ability to produce research – with funding awarded to compensate specific universities for funding research active staff. As such, the aim is to measure as precisely as possible how well a university has done at producing research.

    So let’s work with your example through this lens rather than the individual career lens. A lower ranked institution has taken a risk on a ECR who top ranked institutions wouldn’t touch. It supported them to produce a ground breaking monograph. They then see their research star leave taking their REF outputs with them to a higher ranked institution that wouldn’t invest in the ECR to begin with. Come REF results, that higher ranked institution can now claim credit and funding for the ground breaking monograph that they did nothing to support. The lower ranked institution sees no benefit (and probably thinks, why bother investing in research let’s just put everyone on to teaching contracts!?). In what way does that help anyone other than the individual concerned?

    Wouldn’t it be better to have a proper system of promotion for individuals to deal with the problems you highlight?

  4. It seems a shame that research associates or honorary research fellows and visiting professors cannot be include in the REF. They may have a highly prolific output which has been facilitated by the university to which they are affiliated, and yet that university can get no credit for this kind of activity.

    If the assessment is to be holistic, it should include all research output that has been made possible by a named university.

  5. If the REF is to be holistic and reflect the research activities of a University, then it should be possible to include the research output of honorary fellows and visiting professors. After all, they may not receive a salary but their contribution to that research environment can be very high and productive. The university that has facilitated their work cannot get the credit under the existing system.

  6. What about us postdocs on 1 year contracts having to change institutions (and countries) each year? By the time a journal article gets published I could have changed institutions twice, which makes a bit of a mockery of institutional “ownership”.

Comments are closed.