December 2023: Response to REF 2028 consultation

UKRIO’s views on assessing people, culture, and environment (PCE) in the Research Excellence Framework 2028 (REF 2028).

Below is our submitted response for the 1st of December 2023 deadline, outlining UKRIO’s concerns about this element of REF 2028 and possible solutions to our identified challenges and concerns.

The UK Research Integrity Office (UKRIO) welcomes the increased emphasis on research environment and culture in the REF 2028 and looks forward to seeing the outcome of the commissioned work to develop appropriate indicators. We see benefits to this change, but also challenges and potential pitfalls, which we outline here. As a charity that has provided significant practical support for institutions and individuals in research integrity and culture for many years, we are keen to ensure that this development has a positive impact.  

We believe that the challenges involved in developing appropriate and comprehensive indicators of research culture should not be underestimated, especially considering the rich and varied research environments that we have in the UK. Work that has been carried out on this in the past, for example ‘Indicators of Research Integrity – An Initial Exploration of the landscape, opportunities and challenges’, highlights some of the difficulties ahead in the development of a comprehensive set of indicators that will work well for all institutions and disciplines, without placing undue burdens on researchers or inadvertently incentivising poor practice. 

Whilst a survey of staff may help with consistent approaches to the statements, and a focus on ‘demonstrable outcomes’ is reasonable, the process of developing a good research culture is about more than demonstrable outcomes. It is about the gradual development, over time of a positive, open and inclusive culture. There will be a need for both quantitative and qualitative indicators that fully take account of the range of institutions and disciplines, and that recognise that this is an area where change may be gradual. Indicators will need to be used to derive measures of investments in people, culture and environment relative to the size and/or budget of a higher education institution. In addition to establishing robust indicators, it will be helpful to learn from best practice and the experience of other organisations. 

We recognise that effort will be made to recognise research cultures of all shapes and sizes; there is a possibility however that the bigger, more well-resourced institutions will be in a better position to provide evidence – not because their cultures are necessarily always of a higher standard, but because they will have more resource to focus not only on research integrity but also on the collection of relevant data and evidence. They will also have more staff and resources in all areas of the institution, including research integrity/culture professionals. Just as an industry has grown around research impact after its introduction to the REF landscapes, a similar industry may grow around research culture/ integrity. Whilst this is not a problem – it is essential that institutions invest in support for researchers – the PCE must not become a tick box exercise where those institutions which are able to tick most of the boxes score most highly. 

We welcome the confirmation that panel members will receive training explicitly on how to recognise the positive features of a research culture. Nonetheless, it is likely that many panel members will come from larger research-intensive institutions and may expect to see the research environment that they are used to, rather than a range of shapes, sizes and stages in research development. In previous exercises, these have tended to score more highly in the research environment element. Single-disciplinary institutions may have a different approach to research culture and indeed may be more innovative within their specialism. It is important that this is recognised rather than compared to general across-the-board indicators. 

The process of attempting to assess research culture could change what institutions are doing and how they are doing it. UKRIO has observed great work in this area and the enhancing research culture funding that has been available in parts of the UK has supported this work, as have the activities of our charity and others working in this space. Unintended consequences to new developments are a feature of the REF, and quite apart from the nuts and bolts of how the assessment will be achieved, we believe there is a danger that the assessment process itself could lead to a change in mindset. There is a potential for a shift from being focussed solely on developing and gradually improving your research culture, to one where there is an awareness that the work being done will be judged, measured and a rating put on it, and that will impact the REF outcome for the institution and the funding resulting from it. That could change what people do and how they do it. As noted, the REF has a history of unintended consequences, and it is essential that steps are taken to mitigate this potential outcome and consideration given to how research culture can be assessed in a way that does not unduly impact the work already being done.  

The emphasis may shift to activities that can be more easily measured and that could result in a 4* research environment – ‘quick wins’ rather than deeper, longer-term improvements to environment, culture and systems. This could also bring in an element of competition to a space that has been mostly collaborative, as units attempt to demonstrate their uniqueness, potentially resulting in less sharing of good practice, resources and collaboration.  

Work on research integrity and research culture should be subject to review and impact measurement, but there is a difference between self-evaluation and improvement, and external assessment that impacts on future reputation and funding. Such incentives can have mixed effects. Also, with an outcome that is potentially reduced to a single grade for each element of the REF, this could have a discouraging impact on those who are not successful in achieving a high score. This well-intentioned initiative could become about key performance indicators and undertaking activities that can be measured, rather than about doing what needs to be done. In some circumstances, this could move the focus from the activities to their measurement.  

We note (paragraph 26) that assessment must be ‘wholly related to a unit or institution’s research activity’. We strongly believe that this is not desirable. For example, development activities may take place that are open to all staff, for example relating to diversity, unconscious bias, etc. but not necessarily in the research domain. These would aid in improving the research culture but would not be directly related to research activity. 

Another significant challenge is that information the indicators will not be available until late 2024. With the submission due in 2027, this will leave limited time to use the information available to demonstrate a strong/improved research culture (noting the medium- and long-term nature of many measures to implement sustained cultural change). Again, this could lead to ‘quick wins and box-ticking, rather than genuine improvements to environment and culture. 

As the UK’s most experienced research integrity organisation, UKRIO firmly believes in the importance of safeguarding and enhancing research culture, systems and environments for, as demonstrated by our longstanding work in this area and our extensive support for implementing key initiatives such as The Concordat to Support Research Integrity. We welcome the additional focus on people, culture and environment in REF2028 and the benefits that this could bring, but wish to highlight potential inadvertent incentives which could arise from the proposed approach. 

Solutions 

• It will be vital to work very closely with all disciplines, institution types and other stakeholders when establishing the indicators, including actively seeking out voices that may not otherwise be heard.  
• Panel members will need to be diverse in terms of the institutions and unit sizes that they represent, as well as demographically.  
• Include a published narrative outcome that accentuates the positive features as well as areas for improvement. The definitions of what a 4*, 3* outcome means in terms of PCE will need to be set out very clearly.  
• Consider a pilot exercise and/or a delay to the implementation of this area, to ensure it is implemented fairly and rigorously, and fully tested prior to implementation.

This response was written by Nicola Sainsbury, Research Integrity Manager, UKRIO and reviewed by UKRIO’s research integrity managers team, Chief Executive and Chair.

UKRIO would like to thank our Advisory Council for their comments on this response.