Research Excellence Framework - second consultation (REF2014)

The Royal Geographical Society (with The Institute of British Geographers) welcomes this opportunity to comment on the Research Excellence Framework second consultation on the assessment and funding of research.

The Society is the UK’s learned society and professional body for geography and geographers. It was founded in 1830 for the advancement of geographical science and has approximately 15,000 Fellows and members. The Society’s remit spans the breadth of geography and geographical research.

In preparing our response to this consultation, we requested comments from all UK Departments of Geography, the RGS-IBG research groups, members of the Society’s Research Committee, and the Society’s Fellows and members more widely. Many responses were received. While there is almost unanimity within the community as to the principles on which we seek geography to be assessed in the REF, and these are fully endorsed by the Society, there is less agreement as to exactly how best to achieve this.

We also had discussions with other learned societies and representatives of heads of department from some cognate disciplines. In this regard, it is very apparent that on a considerable number of issues, notably the concerns about panel structure (STEM - non-STEM divisions), geography’s concerns and principles as a subject community are very close to those expressed by some other areas whose subjects cross the natural and social sciences.

 

Consultation question 1

Do you agree with the proposed key features of the REF? If not, explain why.

 

In summary, the principles that all the community and the Society fully agree on are:

• Geography’s intellectual value and integrity as a discipline must continue to be recognised and retained within a single unit of assessment.

• Geography should not be weakened by the subject being split across different units of assessment, either explicitly or as a result of an institutional ‘follow the funding’ imperative.

• Geography as a discipline must be appropriately assessed and funded in ways that fully recognise the nature of the discipline and its 50% science base.

Thus, we strongly reiterate our formal request made in June that the physical and natural science research within Geography be afforded STEM protection, thus preserving the breadth of the environmental science research base, especially at a time when environmental issues that physical geographers are working on (for example, climate change, fluvial processes and flooding; Arctic/Antarctic dynamics and melt, sustainable development) are some of the most pressing ones facing society and government. We believe, and substantiate with data appended to this document, that a 50% ring-fenced allocation would be just. It is essential that the part-STEM designation is recognised from the outset of REF and put in place prior to the next assessment exercise and not left as a decision until after submissions have been made.

Reflecting the different circumstances in HEIs, there is a range of views among the community as to how these principles are best delivered in terms of the panel and unit of assessment structures of the REF. The Society has listened, discussed and given much thought to the different options and has decided to make two recommendations:

Our preference is for the panel structure itself to reflect that an important group of disciplines exist which do not fall within the rigid science, social science, and humanities framework, but cut across that framework. This would translate to the creation of an additional panel (E), within which Geography would be a unit of assessment. This would recognise fairly and equitably disciplines including Geography, Archaeology, Environmental Science, Environmental Studies, possibly Psychology, and others.

Our second option sees Geography as a unit of assessment being placed either fully within Panel B and funded fully in line with other units of assessment in Panel B; or with assured 50% part-STEM designation and funding within panel C.

We strongly support Geography and Environmental Studies remaining as a separate unit of assessment. Should the situation arise that changes are proposed to the subject content of the Unit of Assessment, we strongly ask that we are consulted about proposals. Our view is that we would only be prepared to consider any changes if they related to cognate discipline(s) that have a similar breadth of scope, that encompass research across the sciences, social sciences and humanities; and that have a tradition of science-based field and lab based research consistent with the part-STEM designation and funding requested here.

Thus, we support neither of the suggested scenarios stated within the consultation document, namely Geography and Environmental Studies in Panel C or Geography combined with Earth and Environmental Science in Panel B. The former is unacceptable in that it gives no funding recognition to the significant science base of geography; the latter will be especially difficult given the breadth of geography research in the social sciences and humanities

The other key points we would like to highlight are:

• There is strong support for the continued focus on research excellence and peer review. If panels are to be informed by bibliometric data, we recommend these be in raw form, as reasoned below (Q2).

• We recognise and support the position that impact will be an important feature of REF. However, we urge that impact be defined very broadly to fully recognise that disciplines will have impacts in widely different ways. We strongly support inclusion of impact embracing social, cultural, policy, and environmental arenas and not just economic impacts. In addition, we believe it is important to include impacts in terms of the individual, such as quality of life, well being, and the delivery of wider public learning and knowledge.

• We believe it important to recognise explicitly in the criteria on impact that this can apply to local, regional, national and international scales, with regional, national and international being of equal merit.

• Given the uncertainty as to how this will work in practice we urge sensible caution until the outcomes of the pilots are known and until the methodology has been tried and tested. A level of 15 – 20% for impact in REF 2012 seems appropriate; we do not support 25%

 

Consultation question 2

What comments do you have on the proposed approach to assessing outputs? If you disagree with any of these proposals please explain why.

We support the principle that institutions should select staff and outputs.

We strongly support the continued focus on research excellence and peer review. If panels are to be informed by metrics, we recommend the data be given to the panels in raw form. Given different citation patterns in sub-disciplines of Geography, and of different disciplines in single units of assessments, the discipline does not support the use of bibliometrics.

The community support the approach adopted by the RAE2008 panel in which all outputs were considered in detail. Overall, there is support that four outputs be assessed to reflect sustained research activity and to allow the exploration of quality across the breadth of research. However, if the burden on panels (due to fewer panels and increased numbers of submissions per panel) is likely to result in sampling of outputs, we strongly advocate a reduction from 4 to 3 outputs and sampling not be used.

We support the double counting of certain outputs – monographs for example. However, clear criteria are needed for those outputs to be assessed in this way and they must be specified by the submitting institutions at the outset not retrospectively by panels.

 

Consultation question 3

What comments do you have on the proposed approach to assessing impact? If you disagree with any of these proposals please explain why.

As a matter of principle, we urge that impact be defined very broadly to fully recognise that disciplines will have impacts in widely different ways.

For Geography, with research (and impact) across a broad array of fields, ranging from environmental change, climate science, natural resource management, regional governance, urban policy, leisure policy, planning, to education and more, a broad view of social, cultural, policy, service, environmental and economic impact must be adopted.

In addition, we believe it is important to include impacts in terms of the individual, such as quality of life, well being, and the delivery of wider public learning and knowledge.

We believe it important to recognise explicitly in the criteria on impact that this can apply to local, regional, national and international scales, with regional, national and international being of equal merit.

It is also fundamental to recognize that the impact of high quality research can be indirect. Much geographical research has an impact through the wider contributions that researchers make, for example, in advisory roles, through the media, and in raising wider public awareness.

The impact of research leadership in the discipline also needs to be recognised (formerly part of esteem) e.g. senior roles in learned societies; leading editorial roles etc. The current measures are too narrowly defined.

We support the notion that impact be assessed at the level of the department/unit through case studies. However, we recommend that HEFCE give serious consideration to increasing the number of case studies for smaller units to ensure the assessment of impact for a whole unit is not based on just one or two researchers.

We note, as do HEFCE, the very significant difficulties of measuring impact, such as time lags, attribution and corroboration. It is likely that there will be implications for those units working on new, emerging topics and areas, and for scholars at early stages in their careers (see recommendations in Q11).

Subject appropriate users and producers should be involved in assessment of impact. We recommend that academic members should sit alongside users on the impact panel and there be clear guidelines in reporting and liaison.

 For Geography, users need to be defined broadly to capture the impact of geographical research for business, government, schools, research communities (consultants), third sector, community groups, museums, heritage organizations, libraries, etc

Representatives of learned societies, particularly those which are also professional bodies, who work with academic and professional communities, may be in a very strong position to provide useful insight into disciplinary impact.

Given the uncertainty as to how this will work in practice, we urge sensible caution until the outcomes of the pilots are known and until the methodology has been tried and tested. A weighting of 15 – 20% for impact in REF 2012 seems appropriate; we do not support 25%

 

Consultation question 4

Do you have any comments on the proposed approach to assessing research environment?

Generally we support the approach. Though we note including public engagement as part of ‘environment’ may have the potential to result in double counting. Environment should measure inputs weighted appropriately to the size of the submission. It should not measure outputs.

 

Consultation question 5

Do you agree with our proposals for combining and weighting the output, impact and environment sub-profiles? If not please propose an alternative and explain why this is preferable.

We reiterate our comments from above that given the uncertainty of the assessment of impact, sensible caution must be exercised until the outcomes of the pilots are known and until the methodology has been tried and tested. A weighting of 15 – 20% for impact in REF 2012 seems appropriate; we do not support 25%.

 

Consultation question 6

What comments do you have on the panel configuration proposed at Annex E? Where suggesting alternative options for specific UOAs, please provide the reasons for this.

See comments above.

 

Consultation question 7

Do you agree with the proposed approach to ensuring consistency between panels?

We stress the need for effective moderation of assessments of outputs that are cross-referred between units of assessment and between main panels. Careful consideration must also be given to normalization of all profiles between units of assessment and panels.

Where multiple units are submitted to one unit of assessment, it is important that outputs are tagged to enable disciplinary profiles to be constructed

 

Consultation question 8

Do you have any suggested additions or amendments to the list of nominating bodies?

No response.

 

Consultation question 9

Do you agree that our proposed approach will ensure that interdisciplinary research is assessed on an equal footing with other types of research? Are there further measures we should consider to ensure that this is the case and that our approach is well understood?

The proposal relies on disciplinary structures aligned with sciences, social sciences and arts and humanities. It does not reflect those disciplines, nor those individual researchers, that transcend these structures. We reiterate our concerns about the special challenges this presents for the discipline of geography and other similarly structured disciplines.

 

Consultation question 10

Do you agree that our proposals for encouraging and supporting researcher mobility will have a positive effect; and are there other measures that should be taken within the REF to this end?

The principles are sound, but more details are needed.

 

Consultation question 11

Are there any further ways in which we could improve the measures to promote equalities and diversity?

We recommend procedures be put in place that monitor and report on the impact of key features of REF on equality and diversity; for example, of assessing impact, the shorter assessment period etc, on young researchers, those who take family related career breaks, and those who work in non-English language traditions

 

Consultation question 12

Do you have any comments about the proposed timetable?

The inclusion of impact for the first time, and the yet as unproven measures for its assessment, do raise significant concerns with respect to the proposed timetable. For this reason we reiterate that sensible caution must be exercised until the outcomes of the pilots are known and until the methodology has been tried and tested. We also suggest that the timeline be delayed by one year.

If further changes are proposed between this draft and the final outcome, we stress the need for additional consultation and discussion.

 

Consultation question 13

Are there any further areas in which we could reduce burden, without compromising the robustness of the process?

No comments

 

Consultation question 14

Do you have any other comments on the proposals?

Geography as a discipline must be appropriately assessed and funded in ways that fully recognise the nature of the discipline and its 50% science base. Thus, we strongly reiterate our formal request made in June 2009 that the physical and natural science research within Geography be afforded STEM protection, thus preserving the breadth of the environmental science research base, especially at a time when environmental issues that physical geographers are working on (for example, climate change, fluvial processes and flooding; Arctic/Antarctic dynamics and melt, sustainable development) are some of the most pressing ones facing society and government. We believe, and substantiate here with data, that a 50% ring-fenced allocation would be just.

 

Geography: The case for funding protection as a part-STEM subject

A significant fraction of geographical scholarship is scientific work of high quality

Scientific geographical research requires and attracts significant funding to ensure the necessary science-based infrastructure is in place, as evidenced by SRIF investments. This is on a par with SRIF funding to Earth and Environmental Sciences as defined in RAE2008.

50% of the research submitted to RAE2008 in UoA H-32 was defined as physical and natural sciences according to (i) the assessment of the RAE Sub-Panel; (ii) the journals that the research was published in; and (iii) the research councils and charitable funding sources that supported it. This further supports longstanding recognition of Geography as part-laboratory in terms of cost weights.

There were significant cross-referrals between E-17 and H-32 panels in RAE2008.

While the nature of the discipline and its science cost base is widely recognised (part STEM; part non-STEM), following RAE2008 Geography did not receive part-STEM funding. This in unjust and will have a severe and detrimental impact on the quality of the UK science base and on research in the discipline.

We do not seek 100% STEM protection, but do believe that it is right and proper that there be 50% STEM protection.

 

1. The scientific laboratory status and costs of Geography

Geography is well known as a subject that combines physical science with social science and humanities. Geographers work on some of the key environmental and societal issues challenging policy today: including climate change, its causes and effects, at local, regional and international scales; Arctic/Antarctic dynamics and melt; fluvial processes and flooding; and sustainable development. Specific examples include Geographers working at the core of international efforts to measure rates of sea level change (Durham, Plymouth), assessing environmental degradation in lake systems (UCL, Loughborough), developing polar and alpine ice mass models to understand how they link in to global environmental systems (Cambridge, Edinburgh, Aberystwyth), understanding rates of abrupt climate change and how environmental processes and humans respond (RHUL, UCL), interfacing with climate modellers to develop Earth System Science models (Bristol), and mapping how humans have altered vegetation patterns and cover (Oxford, Southampton).

Geography has long been regarded as a part-laboratory subject, a status justified largely on the grounds that almost all research active Departments of Geography maintain scientific research laboratories, often highly sophisticated, with technician support in order to deliver departmental research agendas and meet the needs of research staff in physical geography. These essential science infrastructure costs are further augmented by field equipment and by GIS and Remote Sensing computer-based needs required to maintain Geography research programmes. In addition, departments cite their science infrastructure support as necessary for them to compete for international grants and research talent.

Evidence of sustained, even increased, emphasis on the science base of Geography can be seen in the level of SRIF investment that occurred in the 2001-2008 period, and is explicitly referenced in the RA5a returns for the 2008 RAE. In total, this reveals at least £30.7m of SRIF investment in laboratory facilities and the scientific equipment needed as research infrastructure to support research-active staff working in Geography on environmental research. This figure is based on evidence provided in the 21 submissions reporting unambiguously on SRIF funding and its use; their average SRIF spend was £1.465m. In the RA5a returns to sub-panel E17 (Earth Systems and Environmental Science), for comparison, the total revealed SRIF investment was at least £34.4m, explicitly reported in 22 submissions where the average spend was £1.564. However, it is worthy of note that the highest individual SRIF spend in these E17 submissions was at Lancaster, where it was reported that "a further £4.25M (SRIF) funded re-location of Geography into purpose-built facilities."

 

2. Balance of science and social science/humanities research in Geography

Since the 1996 RAE, and following representation to HEFCE, Geography has been assigned a subject cost-weighting for research for a "part-laboratory" subject. At 1.3, this weighting is mid-way between that for humanities' subjects at 1.0, and science subjects at 1.6, and is therefore equivalent to the average of these two weights for a unit with equal amounts of science and humanities. (The same logic applies to the Geography allocation to teaching cost band C). It has been established and widely recognised therefore that Geography has a significant element of science in both its research and its teaching.

The precise balance of the science and humanities within Geography has been confirmed by the 2008 RAE, as is made clear in the Subject Overview report (http://www.rae.ac.uk/pubs/2009/ov/). Here, it is stated (#2.5) that "the number of outputs broadly classified as physical geography and environmental science was approximately 2,240, while those in human geography and social scientific environmental studies numbered about 2,380. This is a 50:50 balance within the margin of error in assigning outputs to the two categories.

 

3. Geography, Environmental Studies and Environmental Science in the RAE

For the 2008 RAE, Geography was combined with Environmental Studies to form sub-panel H-32. This was the first use of a panel with this title in an RAE. In RAE 02/2004, the initial HEFCE consultation on Panel Configuration and Recruitment for the 2008 RAE proposed a Main Panel H with Environment Sciences, Built Environment, Town and Country Planning and Geography. The Geography community responded with support for this Main Panel, since it approved of the subject range, embracing the sciences and social sciences. In particular the inclusion of Geography and Environmental Sciences was seen as positive; these being two similar subject areas with both natural and social science components.

However, the eventual outcome of the consultation saw Environmental Sciences moved to Main Panel E, where it combined with Earth System Sciences to form sub-panel E-17. At the same time, Geography was combined with Environmental Studies which was an outcome never considered and Geography was given no opportunity to respond.

During the 2008 RAE process, the similarity of the remit of E-17 (or at least, its Environmental Science component) and H-32 meant that there was considerable overlap between the two subpanels. This meant that a small minority of Geography submissions were split (physical geography to E-17, human geography to H-32). Some others were submitted directly to E-17 and cross-referred to H-32 (some human geography and the social science based studies in Environmental Science submissions). There was a high level of cross-referral activity between the two sub-panels; approximately 200 items were referred from E17 to H32, about half of which were in the social sciences, and about 150 were referred in the opposite direction, most being geology.

The outcome of the RAE was, in terms of the overall quality profiles, very similar between H-32 and E-17. In the submission to sub-panel H-32, the number of Category A staff was 1,166 (a head count figure, #2.3 of the Subject Overview Report). The HEFCE FTE count for 2009-10 is 989. This means that about 500 individual researchers within units submitting to H-32 are contributing to the UK's environmental science research effort, using the 50/50 split noted above. The combined submissions to E-17 give a HEFCE FTE count for 2009-10 of 812. Given that the majority of this is in Earth Sciences, it is very likely that there is more environmental science research being done in the submissions to the Geography and Environmental Studies sub-panel than in the submissions to E-17, with there being no evidence of any material difference in its quality.

 

4. Funding Consequences

However, the funding consequences for those submitting to the two sub-panels are radically different, largely because E-17 is afforded the STEM protection outlined in the March 2009/08 HEFCE paper Recurrent Grants for 2009-10. Given the degree of similarity of the two fields, and the level of joint assessment through cross-referral, this is an unjust outcome that seriously undermines the position of physical geography research, the ‘environmental’ science research base, the status of Geography as a whole and the integrity of the RAE. In addition, there is the matter of equity; similar fields should be afforded similar treatment.

There are also a series of specific anomalous outcomes. Social science work submitted to E-17 and assessed by H-32 is now funded at abnormally high levels; some geography that was submitted to E17 has been assessed as having a lower than average quality profile by H-32 standards, but is funded as well as a high-ranking submission to H-32; and some geology submitted to H-32 is much less well-funded than comparable geology submissions to E-17.

 

5. Summary

All of the evidence supports a case that Geography should be afforded STEM protection in a similar manner to Environmental Sciences. This would also be consistent with the evidence that the physical geography/environmental science conducted in Departments of Geography has the same basis in laboratory and field investigation, and their associated infrastructure costs, as that conducted in Environmental Science Departments (Section 1). Given the evidence in Section 2 that there is a 50:50 balance between research which is physical geography and human geography (including social science environmental studies), it would be appropriate to provide 50% STEM protection to Geography. The inequitable allocation (Section 4) has serious and unintended consequences. It should be recognised that even this position represents a concession, in that social science work submitted to sub-panel E-17 received STEM protection.

We formally request that the physical science research within Geography be afforded STEM protection, thus protecting the breadth of the environmental science research base, and especially at a time when the environmental issues that physical geographers are working on are some of the most pressing ones facing society and government. Moreover, we believe assurances of part-STEM designation and funding would serve to significantly reduce games-playing by institutions regarding which panels to submit to and to retain the integrity of the REF exericse.