I came across a really interesting idea the other day week via the Freakonomics podcast – the idea of a project “pre-mortem” or “prospective hindsight” They interviewed Gary Klein who described it as follows:
KLEIN: I need you to be in a relaxed state of mind. So lean back in your chair. Get yourself calm and just a little bit dreamy. I don’t want any daydreaming but I just want you to be ready to be thinking about things. And I’m looking in a crystal ball. And uh, oh, gosh…the image in the crystal ball is a really ugly image. And this is a six-month effort. We are now three months into the effort and it’s clear that this project has failed. There’s no doubt about it. There’s no way that it’s going to succeed. Oh, and I’m looking at another scene a few months later, the project is over and we don’t even want to talk about it. And when we pass each other in the hall, we don’t even make eye contact. It’s that painful. OK. So this project has failed, no doubt about it [….] I want each of you to write down all the reasons why this project has failed. We know it failed. No doubts. Write down why it failed.
The thinking here is that such an approach to projects reduces overconfidence, and elsewhere the podcast discusses the problems of overconfidence, “go fever”, the Challenger shuttle disaster, and how cultural/organisational issues can make it difficult to bring up potential problems and obstacles. The pre-mortem exercise might free people from that, and encourages people (as a team) to find reasons for failure and then respond to them. I don’t do full justice to the arguments here, but you can listen to it for yourself (or read the transcript) at the link above. It reminds me of some of the material covered in a MOOC I took which showed how very small changes in the way that questions are posed and framed can make surprisingly large differences to the decisions that people make, so perhaps this very subtle shift in mindset might be useful.
How might we use the idea of a pre-mortem in research development? My first thought was about grant applications. Would it help to get the applicants to undertake the pre-mortem exercise? I’m not sure that overconfidence is often a huge problem among research teams (a kind of grumpy, passive-aggressive form of entitled pessimism is probably more common), so perhaps the kind of groupthink overconfidence/excessive positivity is less of an issue than in larger project teams where nobody wants to be the one to be negative. But perhaps there’s value in asking the question anyway, and re-focusing applicants on the fact that they’re writing an application for reviewers and for a funding body, not for themselves. A reminder that the views, priorities, and (mis)interpretations of others are crucial to their chances of success or failure.
Would it help to say to internal reviewers “assume this project wasn’t funded – tell me why”? Possibly. It might flush out issues that reviewers may be too polite or insufficiently assertive to raise otherwise, and again, focuses minds on the nature of the process as a competition. It could also help reviewers identify where the biggest danger for the application lies.
Another way it could usefully be used is in helping applicants risk assess their own project. Saying to them “you got funded, but didn’t achieve the objectives you set for yourself. Why not?” might be a good way of identifying project risks to minimise in the management plan, or risks to alleviate through better advanced planning. It might prompt researchers to think more cautiously about the project timescale, especially around issues that are largely out of their control.
So… has anyone used anything like this before in research development? Might it be a useful way of thinking? Why will your current application fail?
I’ve not seen this before, but I’m keen to try it out. Sounds really interesting.
I think my version would be something like, “OK, your application definitely hasn’t been funded. We know that it failed. No doubts. Write down why it failed.”
At the ‘Funders Forum’ event at the recent ARMA conference the panel of Research Council representatives were asked about the the difference between a ‘good’ and a ‘very good’ bid. Along with the academic conveying their enthusiasm, they said it often came down to eradication of doubt. If there were two bids of equal scientific merit, panels will go for the one where they are most convinced of it’s feasibility. The suggestion was made that internal peer review should include the question “If there was one thing that would you reject this proposal, what would it be?” and we will be incorporating that here at MMU.
It’s implicit in the Freakonomics description, but I’d suggest a subtle change to your suggestion.
Instead of “assume this project wasn’t funded – tell me why”, use “It’s 6 months from now and you know this proposal was rejected. Explain what went wrong.”. There’s evidence that explicitly projecting yourself into the future and treating the failure as fact makes it easier to talk about.
I wish I could remember the source, but I can’t. Fundamentally, our brain isn’t rational in the mathematical sense. It has built-in biases that we can hack our way around. Useful reading – “Predictably Irrational” by Dan Ariely, and “Thinking Fast and Slow” by Daniel Kahneman.
I like the notion of “prospective hindsight”. I think a slightly more positive way to address this for grant writers might be: “assume this application is not funded, but you have the opportunity to re-apply next year. What could you have done better? What would you do differently?”.