Fulfilling our potential: teaching excellence, social mobility and student choice
1: What are your views on the potential equality impacts of the proposals and other plans in this consultation?
The proposed Teaching Excellence Framework (TEF) would use metrics, set in context by qualitative judgements made by independent review panels. This qualitative element is crucial to ensure that metrics performance does not incentivise behaviour that undermines the equality agenda. We note that parental income, social networks and student backgrounds are strongly linked to access to higher education and to employment outcomes, Metrics used in TEF need to recognise this appropriately to mitigate unintended consequences, whether in terms of recruitment of disadvantaged students, or disadvantages to institutions more effective in the recruitment of such students. TEF needs to look beyond teaching alone and considering the broader elements of learning environment and assessment processes that have direct impact on issues of equality and access.
2: How can information from the TEF be used to better inform student and employer decision making? Please quantify these benefits as far as you can.
Teaching and learning in higher education are of the utmost importance. There is an underlying assumption in the document that teaching is not valued as much as research, but evidence is not given to support this. In practice most university geography academics teach and this endeavour is valued in time, energy and effort.
Research and teaching quality are not independent. Teaching of geography in universities is for the most part research-led and benefits greatly from this. The best teaching and innovative curriculum design often follows the best research. It is a false dichotomy to pit research ‘against’ teaching.
There is no single measure that students use when making their choice and high quality courses can take many forms. TEF should be flexible enough to identify quality in different forms and places. Students will value information on specific subjects.
Metrics need to recognise differences in students - their backgrounds, experience, expectations and desired outcomes from higher education. Individual students will value different aspects of their degree experience – face-to-face contact hours, strong employability-focus, proximity to the best researchers, library facilities, etc. This may also vary between disciplines. A degree of flexibility in any metrics used to assess teaching quality must be present.
Employers are a heterogeneous group and their needs also are diverse. The employment destinations of graduates also are diverse - some students will pursue graduate careers ‘in’ their disciplines, others will draw on their transferable skills and find employment in a broad range of sectors and roles. Very careful attention needs to be given to engagement with employers (inclusive of large organisations and SMEs) and metrics used to document quality and success of graduates from their perspectives.
3: Do you agree that the ambition for TEF should be that it is open to all HE providers, all disciplines, all modes of delivery and all levels?
Yes. TEF assessments will need to acknowledge the diversity of the sector with appropriate levels of contextualisation. TEF at an institutional level will mask unevenness across subjects. Moreover, students will value information on specific subjects. However, devising a framework that allows for fair and robust comparison across providers, disciplines, modes and levels will be highly challenging and potentially burden-some.
4: Where relevant, should an approved Access Agreement be a pre-requisite for a TEF award? What other mechanism might be used for different types of providers?
Yes. We fully support commitments to increasing access and the achievement of students from disadvantaged backgrounds and under-represented groups.
5: Do you agree with the proposals on:
a. what would constitute a ‘successful’ QA review
Yes. QAA reviews provide a credible and established baseline for TEF.
b. the incentives that should be open to alternative providers for the first year of the TEF
No answer.
c. the proposal to move to differentiated levels of TEF from year two?
Yes. We support the proposals for differentiated levels of TEF. However, sufficient time must be taken to ensure the evidence base, metrics in particular, are robust and reliable across institutions. Full engagement with all sector stakeholders is needed in this process.
6: Do you agree with the proposed approach to TEF assessments on
Timing - Yes.
Assessment panels - Yes.
Process - Yes.
Five year awards seem appropriate, with institutions applying in annual windows (i.e. not all at the same time). Every attention must be directed to the ‘burden’ of any new processes. Criteria need to be clear and transparent and there needs to be effective moderation processes to ensure consistency across panels
Independent experts – academic peers students, and employers – should make up the assessment panels. It is critical that TEF is credible to the public and the higher education community.
7: How can we minimise any administrative burdens on institutions? Please provide any evidence relating to the potential administrative costs and benefits to institutions of the proposals set out in this document.
To establish a robust process, and the trust of the academic and teaching community, a new system could soon grow to the scale and cost of the equivalent Research Excellence Framework. Attention to costs, and reporting these, must be constantly kept under review to ensure the cost does not become too great relative to the potential benefit.
Applications and re-assessments should be maintained at a frequency that maximises ease of application and minimises complexity. TEF processes and metrics should be aligned with other assessments of quality (e.g. QAA or equivalent). TEF processes and metrics should take account of processes and data already in use in the day-to-day oversight of teaching and learning in HEIs.
8: Do you agree with the proposed approach to differentiation and award as TEF develops over time?
Yes. We believe over time discipline/subject level scrutiny should be introduced.
9: Do you agree with the proposed approach to incentives for the different types of provider?
No. The primary goal of a TEF must be to raise quality across the board and encourage innovative and effective practice, not to raise fees. It is important the link between the two is severed. Linking teaching quality to a system of funding could result in game-playing by institutions. The risk is that effort will be placed not on actually improving quality, nor sharing best practice (as encouraged in the current QAA system), but on securing greater fees by adapting to the system and achieving particular metrics.
10: Do you agree with the focus on teaching quality, learning environment, student outcomes and learning gain?
Yes. We support the proposed focus on teaching quality, learning environment and student outcomes. However, we question whether robust and reliable measurements of learning gain, across all subjects and students, can be developed in the near-future. See our further responses to Q11, relevant to this question too.
11: Do you agree with the proposed approach to the evidence used to make TEF assessments - common metrics derived from the national databases supported by evidence from the provider?
Not sure.
The Green Paper does not specify in detail what will be measured in TEF. The extent to which the TEF will be regarded as an indicator of quality will depend on the robustness of the metrics and the methodology to be determined. We note:
Effective metrics must be valid, robust, comprehensive, credible and current. In a teaching system, attention needs to be directed to the quality of the inputs (staff expertise, training and accreditation, links to research, contact time (recognising this varies by subject) etc); the quality of the outputs (degree levels, completion rates, satisfaction etc); and the added value for students (learning environment, employability and learning gain). The current proposal does not adequately include all three elements.
We strongly encourage that evidence of the effectiveness of the TQA some 20 years ago is sought out where possible. If the evidence suggests it had no discernible impact on teaching quality, then we feel the current approach should be questioned given the time and money that will be involved.
The metrics that will be used to measure teaching excellence need much more attention and it needs to be communicated, very clearly, that these will be at best proxies. Any system needs flexibility and a degree of discipline specificity. For example in geography, fieldwork is a key element of the learning experience and metrics used must recognise this.
There is evidence that NSS, student satisfaction, is not positively linked with good learning outcomes and DLHE is linked to background and social networks and strength of regional economies. Benchmarking of any metrics needs to be transparent and fair.
A very important element of teaching and learning in higher education, particularly in the social sciences, is challenging students and exposing them to alternative perspectives and different ways of thinking about the world. This involves using a diverse range of teaching practices (seminars, labs, field courses etc). This can unsettle students, there are no ‘right’ answers and students are expected to be active participants in their learning. This learning experience may be as important as the learning outcome. Metrics need to recognise this.
There is little attention in the document for the need for students to be encouraged to ‘fulfil their potential’ by actively engaging with, and participating in, their learning/degree. It is not a one-way ‘customer contract’, but rather a mutual agreement between two parties, either of which might fail in their commitment.
Peer review of individual teaching performance is essential and interpretation either of existing metrics or of newly-designed and measured metrics should be carried out by expert panels who can interpret them in context.
It is important to ensure there are no unintended consequences in terms of the emphasis (p.27) on consumer protection law. This could stifle innovation as the course content and structure has to be set 18 months before students arrive and sustained until they leave (4.5 years, 5.5 years on a 4 year degrees). For a course like geography although core concepts remain, areas of interest change as the world changes. Such an approach could dull the possibility of new modules being developed and also, if there are personnel changes, mean that staff are not teaching to their strengths.
TEF will only apply to England. The higher education landscape is UK wide. Where will this leave Welsh, Scottish and NI higher education institutions?
Given the limitations of the metrics, it is important for providers to have the opportunity to explain the context in which they demonstrate excellence and to be assessed by their peers.
12a: Do you agree with the proposals to further improve access and success for students from disadvantaged backgrounds and black and minority ethnic (BME) backgrounds?
Yes. Widening access to higher education is essential.
There needs to be careful attention to ensure that those institutions who offer the greatest degree of access and opportunity do not, unintentionally, perform less well on metrics used in TEF. It is important to understand what allows students to thrive and to consider how these should be integrated into TEF (e.g. extra learning support).
It is unfortunate that there have been cuts to the student opportunity funding for the poorest and disabled students within the overall teaching budget.
12b: Do you agree that the Office for Students should have the power to set targets where providers are failing to make progress?
Not sure. Targets should be set by the providers, based on their mission and context. It is appropriate for the Office for Students to review and work with providers to approve these targets.
12c: What other groups or measures should the Government consider?
Part-time and mature students to ensure the metrics are fully appropriate
13a: What potential benefits for decision and policy making in relation to improving access might arise from additional data being available?
Better understanding of the common barriers and challenges to individuals from disadvantaged and under-represented groups to target initiatives to address these.
13b: What additional administrative burdens might this place on organisations? If additional costs are expected to be associated with this, please quantify them?
Any additional administrative burden will depend on the data to be made available. The burden may not be uniform. Data that can serve multiple purposes are desirable.
14: Do you agree with the proposed single route into the higher education sector? Please give reasons for your answer, including information quantifying how the potential cost of entry would change as a result of these proposals.
Yes. All Higher Education providers operating in the field should enter the system through a single route.
15a: Do you agree with the proposed risk-based approach to eligibility for degree awarding powers (DAPs) and university title?
Not sure.
15b: What are your views on the options identified for validation of courses delivered by providers who do not hold DAPs?
The bar to entry must be high to protect the excellent global reputation of UK higher education. Teaching in higher-education should be research-led.
16: Do you agree with the proposed immediate actions intended to speed up entry?
Not sure.
17: Do you agree with the proposal to introduce a requirement for all providers to have contingency arrangements to support students in the event that their course cannot be completed?
Yes.
18a: Do you agree with the proposed changes to the higher education architecture?
No option selected.
The legal framework and architecture must keep pace with the changing environment for higher education provision.
The architecture also needs to support postgraduate training (vital for the future health of research).
18b: To what extent should the Office for Students (OfS) have the power to contract out its functions to separate bodies?
No option selected.
The OfS should rely on existing UK bodies with appropriate expertise.
18c: If you agree, which functions should the OfS be able to contract out?
It is important that expertise currently within HEFCE be captured in a new regulatory environment.
18d: What are your views on the proposed options for allocating Teaching Grant?
Option 1: BIS Ministers set strategic priorities and BIS officials determine formula
No answer.
Option 2: BIS Minister sets strategic priorities and allocation responsibilities divested to OfS
Agree.
We stress the importance of support for the full range of disciplines and adequate funding for those that have important high-cost elements. While the part-STEM nature of geography (along with psychology) has been recognised, teaching funding does not follow.
It is important that expertise currently within HEFCE be captured in a new regulatory environment.
19: Do you agree with the proposal for a single, transparent and light touch regulatory framework for every higher education provider?
Yes.
21a: Do you agree with the proposed duties and powers of the Office for Students?
Not sure. We support the OfS as an arms-length public body, with a duty to promote students’ interests. We have questions about its proposed powers and stress the need for independence between assessments of quality and funding.
24: In light of the proposed changes to the institutional framework for higher education, and the forthcoming Nurse Review, what are your views on the future design of the institutional research landscape?
Research UK needs to respect the autonomy of the subject areas as currently recognised by the Research Councils and how they are configured. Strong disciplines are at the basis of cross-disciplinary research necessary to address global challenges.
25a: What safeguards would you want to see in place in the event that dual funding was operated within a single organisation:
We strongly endorse the current balance of research funding, with QR allocations from HEFCE providing a baseline funding across qualifying institutions on the basis of REF assessments, and funding on a peer-reviewed proposal basis through the Research Councils. These two streams serve two different and important purposes and each does it well.
If dual funding is operated in a single organisation (e.g. RUK), the independence and separate administration of the two funding streams needs to be embedded in a robust legal framework. It is vital that the REF be managed at arm’s length from the department. QR block grant funding should be allocated on the basis of an independent process underpinned by peer review, supporting excellence wherever it is found. The detail of this proposal should be the subject of further consultation. Moreover, in line with government commitments to 2015 levels of financial support for science being retained in real terms, the QR percentage of the total science funding should also be sustained at the 2015 level.
As a discipline, geography spans the natural sciences, social sciences and arts and humanities. Thus we recognise the essential importance of funding streams across broad areas and the significant contributions the social sciences, arts and humanities make to both pure and applied research. These must not be marginalised. Many of the nation’s and the world’s ‘grand challenges’, such as climate change, natural resource security and migration etc, demand an integrated understanding of the scientific processes, of human and behavioural (social science) perspectives, and of spatial differences and interconnections.
Research funding should not lose sight of the importance of funding high quality ‘blue skies thinking’ across all research areas, including STEM and the social sciences. There must be a balance between research tailored to assist with government policy priorities and the need for blue skies research.
25b. Would you favour a degree of hypothecation to ensure that dual funding streams, along with their distinctive characteristics, could not be changed by that organisation?
Yes. See responses above.
26: What are the benefits of the REF to a) your institution and b) to the wider sector? How can we ensure they are preserved?
The Research Excellence Framework demonstrates the world-class excellence of UK research, both in terms of academic quality and broader societal impact. This is very well illustrated in geography. Formal recognition of this supports the wider international uptake of UK geography through publishing, conferences, training of international students etc. Some of these activities directly support the RGS-IBG.
The REF is an important element in the distribution of funding under the dual-support system by all four of the UK higher education funding bodies.
27: How would you suggest the burden of REF exercises is reduced?
We urge universities to review their costs to reduce bureaucracy and burden (given estimates suggest that 95% costs are incurred in universities).
The responses of the RGS-IBG to consultations dating back to 2008 have been critical of the use of metrics in research assessment, with a strong preference for peer review. There is no evidence we are aware of for metrics, established or new, that can adequately capture originality, significance and rigour of academic outputs. We do not believe citation data, or other variants, quantify ‘quality’ independent of peer review (see Analysis and evidence to this effect is presented in Richards, K. et al. (2009). The nature of publishing and assessment in Geography and Environmental Studies: evidence from the Research Assessment Exercise 2008. Area, 41(3), 231-243). More recently, Wilsdon has shown that automation of research assessment would fail to adequately assess quality. We continue to have concerns about issues of equality – particularly for young/emerging scholars. We know from the citation and readership data for papers in our own journals, ‘established’ ‘recognised’ disciplinary figures are privileged.
The elements of the REF that are robust are the peer assessment of outputs and the impact case studies. We would recommend that the more ‘creative’ writing elements - research environment and impact template - be dropped.
Normalising sub-panel grade-profiles would obviate the need for elaborate calibration procedures between sub- and main panels.. In place of the textual research environment, some level of quantitative data on grants and PhDs would offer useful indicators, as long as the burden for collecting (and validating) the data can be centralised away from individual institutions.
In terms of research grants metrics, however, it must be noted that the bias is heavily in favour of capital intensive disciplines. In many parts of the social sciences large research grants are not appropriate; research relies on data collected by others (the census is a classic example).
Our overall recommendation is to simplify, rather than make more complex the process. HEFCE has tended to make the procedures more complicated and time consuming in every successive exercise, by introducing new elements. This is often because the sector asks for something different or new, and HEFCE. REF was much more demanding than RAE because of the impact agenda, and the (otherwise laudable) integration of output assessors.
Do you have any other comments that might aid the consultation process as a whole?
RUK, as proposed by the Nurse review, must be robust in protecting the Research Councils’ encouragement of discovery and ‘blue skies’ research across the breadth of disciplines currently supported.
The existing Research Councils serve as experts and advocates for their respective disciplines, providing disciplinary leadership, agility and connections with communities. Whatever system evolves, these elements must be maintained, with support across the breadth of disciplines and for cross-cutting research. Efficiencies should come from reduction of administrative burden and harmonization and transparency of processes.
The impact of the splitting of research and teaching policy needs to be considered most carefully. There is a risk of duplication of effort and engagement, and loss of strategic oversight that has been provided by HEFCE of teaching and research funding, quality and capital, as well as wider institutional concerns such as financial sustainability. Effective oversight of the sector as a whole must be maintained in some form. UK higher education, across research and teaching, is internationally leading. This must not be undermined