Draft panel criteria and working methods (REF 2014)
1 Overall draft panel criteria and working methods
1a The generic and four main panel statements achieve an appropriate balance between consistency across the exercise and allowing for justifiable differences between the four main panels.
Disagree
1b Are there particular aspects of the criteria and working methods that should be more consistent across all the main panels? Are there differences between the disciplines that justify further differentiation between the main panel criteria? Where referring to particular main panels, please state which one(s).
Overall more consistency is needed. There are significant differences between the Main Panels which we believe go beyond justifiable (disciplinary) differences of the Panels; notably the descriptions of quality levels of outputs; reserves for double weighted outputs; and exemplification of impact. Our preferences on approaches are outlined below specifically with respect to Panel C.
Steps of 1%: We believe the idea of accuracy achievable to 1% in each division is unfounded and thus steps of 1% are inappropriate. Rankings may well end up depending on the score given to individual outputs and these will not be perfect. The previous scoring of RAE2008, to the closest 5%, was more acceptable.
2 Individual staff circumstances
2a The proposals for determining the number of outputs that may be reduced without penalty, for staff with a range of individual circumstances, are appropriate (Part 1, Tables 2 and 3).
Disagree
2b Please comment on these proposals. Respondents are also invited to comment specifically on: whether Tables 2 and 3 are set at appropriate levels; the proposed options for taking account of pregnancy and maternity (Part 1, paragraph 62)whether a consistent approach across the exercise is appropriate, or whether there are any specific differences in the nature of research that justify differences in the approach between UOAs or main panels; If commenting in respect of particular panels or disciplines, please state which.
The existence of an Equality and Diversity Panel that covers all Main Panels and sub-Panels is welcomed to ensure consistency of approach.
Special circumstances may have a greater impact on disciplines (such as geography) that involve fieldwork and/or laboratory work.
The proposals for determining the number of outputs without penalty for early career researchers seem appropriate.
In terms of maternity leave, we strongly prefer the proposal outlined in paragraph 62 - that the number of outputs be reduced by one for each discrete period of maternity leave. Further clarification on paternity leave, shared parental leave, and single carer responsibilities is needed.
We are concerned that the tariffs for other special circumstances (14 months/1 output) are too stringent.
We request that decisions be made as early as possible as to whether special cases are to be accepted or not.
3 Main panel criteria and working methods
3a The main panel statement achieves an appropriate balance between consistency and allowing for discipline-based differences between the sub-panels.
Agree.
3b Please comment on the balance between consistency and allowing for discipline-based differences between the sub-panels within this main panel. Please state the UOA(s) on which you are commenting.
The main panel statement achieves an appropriate balance between consistency while allowing for disciplinary differences.
4 Submissions and units of assessment (Section 1)
4a Submissions and units of assessment (Section 1) Do the UOA descriptor and boundary statements provide a clear and appropriate description of the disciplines covered by the UOAs? Please include any suggestions for refining the descriptors and state which UOA(s) you are commenting on.
The varied needs of the different subject domains within sub-Panel 17 (Geography, Environmental Studies, and Archaeology) have been met appropriately.
The descriptors for Geography are seen to be comprehensive and we agree they should not attempt to be exhaustive.
Omissions that have been highlighted by some in the geographical community include: environmental geography; ecology; climate change; climatology from the list of descriptors; a lack of clarity in the boundary between development studies (issue-driven research, with a strong practice element) and development geography (driven by geographical research agendas); and those who work specifically on techniques (GIS/RS) question whether it is appropriate only to mention physical and human geography.
4b Please comment on the main panel's criteria in relation to multiple submissions in its UOAs.
Most Geography and Archaeology departments are discrete entities with distinct research and staffing strategies, infrastructure and facilities, and cultures. Thus we believe that multiple submissions will be the norm for C17 - we estimate 90% of the cases and that these will fulfil the generic criteria. We would recommend that this be even more explicitly acknowledged in the documentation; i.e. to indicate this is expected.
October 2012/January 2013 is too late for decisions to be made about multiple submissions if for any reason they are denied. We strongly urge HEFCE to make this process as simple as possible and expedite the process so decisions are made earlier.
5 Assessment criteria: outputs
5a Assessment criteria: outputs (Section 2) Overall, the main panel criteria relating to outputs are clear and appropriate.
Disagree.
5b Please comment on the criteria in Section 2, in particular on where further clarification is required or where refinements could be made.
Quality descriptors:
The assessment criteria for outputs are clear and generally familiar from RAE2008, but ensuring consistency between Panels will be crucial.
The descriptors for sub-Panel C17 seem to favour conceptual and methodological innovation. It is also important to recognise the sustained value of various forms of empirical work.
Multiple submissions of co-authored work:
There are significant inconsistencies between panels about co-authored work from the same unit (Panel A not expecting any; Panel B not permitting it; Panel C allowing a maximum of two authors; Panel D permitting it without restriction). For Panel C there is an inconsistency for multiple submissions within a unit (limit of two) and from multiple institutions (no limit).
For Panel C, we ask that an explicit statement be made as to whether there are any limits to submitting a co-authored output against more than one member of staff in the same institution but different units of assessment.
While it is clear that the output is to be assessed not the author's contribution, it is not clear how the rationale (text provided) for multiple submissions of one piece of work will be assessed. More details are needed.
The approach to the double weighting of outputs, specifically around whether reserves are, or are not, allowed is different between Panels C and D. The approach should be consistent. We favour the approach of Panel D. Also, more details are needed of the criteria by which outputs will be assessed as to whether they will, or will not, be double weighted.
Citation data:
Overall the geographical community believe strongly that citation data should not be used and that to make a distinction between human and physical geography is inappropriate and problematic. Thus we recommend that citation data not be used. However, if the absence of such data negatively impacts the evaluation of an output cross-referred to panels/sub-panels that use citation data (e.g Physical Geography papers referred to sub-panel B17; Economic Geography papers referred to the Economics sub-panel) then citation data should be provided for that output for use by the sub-panel cross-referred to.
Criteria definitions:
A minor point, the 'originality' text needs to be rephrased as it currently suggests that all of the descriptors that follow the colon have to be fulfilled (and that is not what is intended).
6 Assessment criteria: impact
6a Assessment criteria: impact (Section 3) Overall, the main panel's criteria relating to impact are appropriate and helpful to institutions in preparing submissions.
Disagree.
C6b Please comment on the criteria in Section 3, in particular on where further clarification is required or where refinements could be made.
Impact
We recognise that by not providing long lists Panel C is not being prescriptive. We strongly support this given the diversity and breadth of impact possible from geographical research. However, given impact is new to REF, institutions/disciplines do need more guidance about impact that will be valued, about evidence, etc. This could be provided in appendices to the General or to the Panel guidance – it does not have to be embedded in the main body of the guidance document. Any such exemplification should be comprehensive, though not exhaustive, and should draw on examples developed by all of the Panels. We strongly urge panel C and sub-panel C17 to develop such guidance soon.
Guidance is also needed on how much users can contribute to making the case. Users would benefit from guidance on this too.
How the panels are to evaluate reach and significance and the relative balance of these also needs more exemplification.
More clarity is also needed on the relative roles of the user members and academic panel members in assessing impact and of the Panel calibration exercise.
7 Assessment criteria: environment
7a Assessment criteria: environment (Section 4) Overall, the main panel criteria relating to environment are clear and appropriate.
Neither agree or disagree.
7b Please comment on the criteria in Section 4, in particular on where further clarification is required or where refinements could be made.
Panel C is the only Panel not to give a weighting for the different elements under Research Environment --- an indicative weighting would be helpful.
Additional guidance is also needed on what is required in the overview (this is given by Panels B and D).
8 Working methods
8a Working methods (Section 5) Overall, the working methods of the main panel and its sub-panels are clear and appropriate.
Neither agree or disagree.
8b Please comment on the working methods, in particular on where further clarification is required or where refinements could be made.
Cross referrals:
Panel C has stated quite clearly that they will attempt to reduce the number of cross-referrals between sub-panels within Panel C by appointing external assessors (and in special circumstances appointing assessors who can sit on more than one panel). Clarity is needed though about cross-referrals to other main Panels - of specific interest for geography are the possible cross-referrals between B7 and C17 (in both directions). More details would be helpful and again we ask for consistency in the working methods of Panels.
Appointment external assessors and users:
We strongly support the encouragement for HEIs to be detailed in their responses to submission intentions so that appropriate expertise (additional assessors) can be appointed to the panel. If, however, at the time of actual submission further additional expertise is identified, we urge the Panel to allow additional appointments to the sub-Panel.
Given the diversity and representation of the membership of sub-panel C-17, as additional external assessors are appointed we ask this be taken into account (particularly in terms of the gender balance of the sub-Panel).