REF, TEF and Bobtail and a 'Bill of Rights': Analysing the Government's response to the TEF consultation
Two very important reports emerged this week. Firstly, the government's consultation response about the Teaching Excellence and Student Outcomes Framework – although renamed earlier this year, it is still known 'popularly' as TEF [1]. This was followed by the shocking findings in the report, 'Is Britain Fairer' from the Equality and Human Rights Commission (EHRC) [2]. Although some may see both as somewhat unconnected, the juxtaposition of the reports is important at this time since they shed some light onto what should be the real priorities.
The TEF Consultation response.
The government's cunning plan was to win over the so called ‘providers’ to their well worked proposals on the design of subject-level TEF. So far so good it seemed. The position of the government is clear enough since they see that "the movement to subject-level as an important development in TEF, ensuring that prospective students have information about a provider’s teaching excellence and student outcomes in the subject they are looking to study.” It seems that TEF will move to subject level assessments come hell or high water. But how to do this effectively?
Two clashes emerged from the report. The first was the idea of including a ‘grade inflation metric’ that was met with a degree of sceptical reporting in the press [3]. The Telegraph noted that the minister, Sam Gyimah, warned that if assessors found universities to be deliberately inflating grades, their rating would inevitably suffer and their “reputation will also take a hit” [4]. This is truly aggressive stuff and the idea of grades being 'inflated' has clearly rattled the government. But surely it is a situation that is entirely of their own making and must have been anticipated from the outset.
The second contentious point was ‘teaching intensity’ and it is fair to say that many 'providers' blew a cold chill over this idea. The report concluded that, “a measure of teaching intensity in the TEF would not be an effective way to capture teaching intensity. It would not help the assessment panel identify excellent teaching or influence a provider’s rating”. Instead, it is being passed over to the Office for Students (OfS) to sort something out with the student’s panel. This decision is also a thinly veiled admission that the TEF does not measure teaching quality in any way. Something we all figured out a long time ago.
What is grade inflation and what are the causes?
It seems that those in government are being pressed to explain why the number of first class degrees being awarded across the universities has increased during their time in government. The idea that students might be working harder, or that they expect to get more for their money, does not seem to have dawned on them. Using Higher Education
Statistical Agency (HESA) data [5] it can be seen that the numbers have indeed been rising (Figure 1). The question might arise that some institutions are deliberately inflating their grades as Sam Gyimah has suggested. Figure 2 shows the classes of degrees awarded in 2017 across 132 institutions. These are plotted as percentages along the Y-axis and total numbers along the Z- axis which are shown as relative sizes of the spheres. They are plotted against the mean entry grades - as Tariff Points - for each university taken from the 2018 data of the Complete University Guide [6]. It gives a good overall picture of the latest situation. To spare their blushes, the type of institution and their identities are not shown here. However, the entry grades for Cambridge, Oxford and Imperial College are more obvious outliers, then come the Russell Group and pre-92 universities. Overlapping with these are the post-92 institutions.
The trend is obvious in that all institutions, regardless of the entry tariff points, are awarding a significant number of First and Upper Second degrees. Although the percentages are similar, the larger numbers at some institutions may be fuelling the concerns. What is also clear is that the higher tariff students are less likely to get a lower second or third degree. In contrast, the other institutions taking lower tariff point students are awarding many more of these degree classes. Two major points emerge. Firstly, institutions taking lower tariff qualified students are setting a clear standard, as evidenced by them not being averse to awarding lower class degrees. Secondly, many students with lower tariff entry qualifications are achieving more than they might have expected. The grade boundaries might have shifted, but there is still a standard in operation and the increase in first class degrees might be down to hard work, better teaching and simply more students.
Lurking in the report, 'Is Britain Fairer' from the Equality and Human Rights Commission (EHRC)[2], is a pertinent observation amidst the general conclusion of a “two-speed society” that in just three years “alarming backward steps” have left disabled people, ethnic minorities and children from poorer backgrounds struggling to make headway in a society where "significant barriers still remain”. It is noted with regard to universities that "In the period 2006 to 2016, entry rates for disadvantaged students increased by almost four percentage points at less selective universities in the UK but by just over one percentage point at the most selective". Perhaps the 'grade inflation' amongst such students doing well is the real target of the government angst as their cryptic 'vision' of the UK in the 21st Century emerges.
On teaching quality, intensity and standards.
If the grade boundaries have shifted, then this should not be confused with the quality of teaching. This must be addressed as a separate issue. One cause of grade boundary drift must be obvious to anyone in a university, and anyone who has acted as an external examiner over that time. My experience has been in examining across several Russell Group universities so I am unable to comment on the others. However, I would suspect they have adopted similar procedures. In the past, to achieve a first or upper second degree the marks had to be clearly over 70% or 60%. It was a simple boundary and rarely would a student be 'uplifted' to a higher class from a 'near miss'. To stave off more appeals by students, that were paying more but achieving marks that were 'near misses', an automatic mechanism of compensation started to spread across universities. An example might be marks above 68% or 58% being looked at again. If the majority of work had just about achieved over 70% or 60%, then a first or upper second class degree is awarded automatically despite a mean of 68% or 58%. This also applied at the lower class degree boundaries and essentially moved the boundary down for many students. As this practice drifted across the universities, the number of higher degree classes also drifted up. More 2.2s emerged at one end and were offset by more 1sts at the other end. The timing closely matches the trends in Figure 1 and there was a general shift up as a result.
Then comes the separate issue of standards and quality. 'Teaching intensity', that includes contact time for teaching in courses, appears to be a sensible measure. Every course has a timetable and list of assignments so the basic data is easy to gather across subjects. However, it did not go down too well. It seems that Universities UK might be protesting too much with its response to the consultation, "The introduction of any teaching intensity measure risks undermining institutional and academic autonomy to choose the correct pedagogical approach for their unique mix of subject and student cohort. It would lead to homogenisation across the sector, stifle innovation and damage student success”. One wonders what might be their underlying motivation. An answer could be in the conclusion, "The burden of collecting the data was also felt to be disproportionate, particularly as not all providers have systems in place for capturing this type of data". This may be avoiding the issue or perhaps they don't want too much scrutiny of the contact time being slowly eroded away. It would be interesting to see which institutions showed the most concern.
The idea that teaching by the most experienced staff would be an indication of quality was also challenged with an alarming conclusion. "Option 3 (the GTQ weighted by qualification/seniority of teacher) was the most strongly opposed option, with the main concern being that staff seniority would not be a good proxy for teaching quality". It possibly shows that many universities do not show much respect for their senior and most experienced staff when it comes to teaching. This is probably complete nonsense. It might be that in some cases that staff are under pressure and avoid teaching to further their own more lucrative research careers. More likely though is that managers want to shield them from teaching that is becoming a function of more and more temporary and 'casualised' labour. The managers can 'sweat their assets' by pushing successful research professors more into earning research grants and less into teaching. The REF forced this consequence onto universities that compete 'head to head' with each other. Meanwhile, teaching is relegated to the younger academics on temporary contracts. Surely students will wonder where their money is really going at some point.
But none of this is new.
The expansion of Higher Education in the 1960s was also met with worries about resources at a time of severe financial constraint. Then, the standards in the new Polytechnics were all scrutinised by the Council for National Academic Awards (CNAA) since the government of the time recognised that quality and standards could be problematical. When this ceased, along with Polytechnics that became universities after 1992, it might have been expected to lead to changes. Since the 1980s the numbers of students has increased greatly whilst the unit of resource has diminished. The increased pressure on staff was terrifying then and still is. Managers now see some merit in seeking extortionate pay for themselves by squeezing as much as possible out of as few staff as possible. Eventually quality must be sacrificed at the altar of efficiency as limits are reached. Yet government fails to see that TEF is not likely to improve the situation.
Ensuring quality.
There is an underlying problem in the whole system that has been allowed to grow in a poorly coordinated manner. An analogy is what I call 'the British problem' exemplified by Heathrow Airport. Starting life as a small airfield with a 'wooden hut' it has grown piecemeal over the decades. Before one development was completed it was already too small and another was added and so on. The result is a mess of distant terminals, connecting tunnels and corridors and confusion for passengers. Somewhere in the middle, the 'wooden hut' is probably still standing. Planning further into the future and building in some redundancy as a contingency and planning for expansion were ideas too far it seems. Similarly, the university system has become a monstrous competing 'mess' of institutions forced to compete for students; many of whom are lost in the maze designed by REF, TEF and Bobtail. The regulation seems to be an afterthought in response to wholly predictable crisis.
Ensuring quality and, above all fairness, should be at the heart of the system for both the students and the prosperity of the country. Yet neither seems to be addressed logically. One bad idea is 'improved' by bolting on more and more complex arrangements to compensate for the bad idea in the first place. The grade inflation was inevitable as a result of making universities a market place. Then TEF is bolted on later to somehow compensate for the bad idea. Clearer forward thinking at the outset might have helped.
At this point it is best to ignore grade inflation since most employers have already reacted to its presence. Instead, the root of quality control should be addressed. A minimum standard for each subject should be set in place. This includes adequate contact time and assignments. Like any sensible investor, the government should expect to see a forward plan and defined outcomes. The current external examiner system should be 'beefed up' considerably to ensure that the plans are adhered to for the benefit of both students and society.
A student 'Bill of Rights'.
In return for the universities ensuring even more higher standards, students should expect to be challenged and always helped along the way. But it should be fair and equitable for all. The student charters go some way to helping with this idea, but do not go far enough. A 'Bill of Rights' that all universities must sign up to might include a right to attend lectures by the best researchers. Also, to expect a supportive level of tutoring and contact time to back up their experience. Fair assignments and examinations, where every student is given the same time to complete or prepare work, are essential. Students must each be able to devote enough time to their studies and those in financial stress must be supported for an equal chance to study alongside their peers. Tutorials and practical sessions should be designed for effective interaction and where there are not too many students per session. These and other ideas must lie at the core of any contract where the government and the students fund the endeavour.
It should become a right and not a mere aspiration.
The TEF Consultation response.
The government's cunning plan was to win over the so called ‘providers’ to their well worked proposals on the design of subject-level TEF. So far so good it seemed. The position of the government is clear enough since they see that "the movement to subject-level as an important development in TEF, ensuring that prospective students have information about a provider’s teaching excellence and student outcomes in the subject they are looking to study.” It seems that TEF will move to subject level assessments come hell or high water. But how to do this effectively?
Two clashes emerged from the report. The first was the idea of including a ‘grade inflation metric’ that was met with a degree of sceptical reporting in the press [3]. The Telegraph noted that the minister, Sam Gyimah, warned that if assessors found universities to be deliberately inflating grades, their rating would inevitably suffer and their “reputation will also take a hit” [4]. This is truly aggressive stuff and the idea of grades being 'inflated' has clearly rattled the government. But surely it is a situation that is entirely of their own making and must have been anticipated from the outset.
The second contentious point was ‘teaching intensity’ and it is fair to say that many 'providers' blew a cold chill over this idea. The report concluded that, “a measure of teaching intensity in the TEF would not be an effective way to capture teaching intensity. It would not help the assessment panel identify excellent teaching or influence a provider’s rating”. Instead, it is being passed over to the Office for Students (OfS) to sort something out with the student’s panel. This decision is also a thinly veiled admission that the TEF does not measure teaching quality in any way. Something we all figured out a long time ago.
What is grade inflation and what are the causes?
It seems that those in government are being pressed to explain why the number of first class degrees being awarded across the universities has increased during their time in government. The idea that students might be working harder, or that they expect to get more for their money, does not seem to have dawned on them. Using Higher Education
 |
Source HESA [5]
|
 |
Source the Complete University Guide [6]
and HESA [5]
|
Lurking in the report, 'Is Britain Fairer' from the Equality and Human Rights Commission (EHRC)[2], is a pertinent observation amidst the general conclusion of a “two-speed society” that in just three years “alarming backward steps” have left disabled people, ethnic minorities and children from poorer backgrounds struggling to make headway in a society where "significant barriers still remain”. It is noted with regard to universities that "In the period 2006 to 2016, entry rates for disadvantaged students increased by almost four percentage points at less selective universities in the UK but by just over one percentage point at the most selective". Perhaps the 'grade inflation' amongst such students doing well is the real target of the government angst as their cryptic 'vision' of the UK in the 21st Century emerges.
On teaching quality, intensity and standards.
If the grade boundaries have shifted, then this should not be confused with the quality of teaching. This must be addressed as a separate issue. One cause of grade boundary drift must be obvious to anyone in a university, and anyone who has acted as an external examiner over that time. My experience has been in examining across several Russell Group universities so I am unable to comment on the others. However, I would suspect they have adopted similar procedures. In the past, to achieve a first or upper second degree the marks had to be clearly over 70% or 60%. It was a simple boundary and rarely would a student be 'uplifted' to a higher class from a 'near miss'. To stave off more appeals by students, that were paying more but achieving marks that were 'near misses', an automatic mechanism of compensation started to spread across universities. An example might be marks above 68% or 58% being looked at again. If the majority of work had just about achieved over 70% or 60%, then a first or upper second class degree is awarded automatically despite a mean of 68% or 58%. This also applied at the lower class degree boundaries and essentially moved the boundary down for many students. As this practice drifted across the universities, the number of higher degree classes also drifted up. More 2.2s emerged at one end and were offset by more 1sts at the other end. The timing closely matches the trends in Figure 1 and there was a general shift up as a result.
Then comes the separate issue of standards and quality. 'Teaching intensity', that includes contact time for teaching in courses, appears to be a sensible measure. Every course has a timetable and list of assignments so the basic data is easy to gather across subjects. However, it did not go down too well. It seems that Universities UK might be protesting too much with its response to the consultation, "The introduction of any teaching intensity measure risks undermining institutional and academic autonomy to choose the correct pedagogical approach for their unique mix of subject and student cohort. It would lead to homogenisation across the sector, stifle innovation and damage student success”. One wonders what might be their underlying motivation. An answer could be in the conclusion, "The burden of collecting the data was also felt to be disproportionate, particularly as not all providers have systems in place for capturing this type of data". This may be avoiding the issue or perhaps they don't want too much scrutiny of the contact time being slowly eroded away. It would be interesting to see which institutions showed the most concern.
The idea that teaching by the most experienced staff would be an indication of quality was also challenged with an alarming conclusion. "Option 3 (the GTQ weighted by qualification/seniority of teacher) was the most strongly opposed option, with the main concern being that staff seniority would not be a good proxy for teaching quality". It possibly shows that many universities do not show much respect for their senior and most experienced staff when it comes to teaching. This is probably complete nonsense. It might be that in some cases that staff are under pressure and avoid teaching to further their own more lucrative research careers. More likely though is that managers want to shield them from teaching that is becoming a function of more and more temporary and 'casualised' labour. The managers can 'sweat their assets' by pushing successful research professors more into earning research grants and less into teaching. The REF forced this consequence onto universities that compete 'head to head' with each other. Meanwhile, teaching is relegated to the younger academics on temporary contracts. Surely students will wonder where their money is really going at some point.
But none of this is new.
The expansion of Higher Education in the 1960s was also met with worries about resources at a time of severe financial constraint. Then, the standards in the new Polytechnics were all scrutinised by the Council for National Academic Awards (CNAA) since the government of the time recognised that quality and standards could be problematical. When this ceased, along with Polytechnics that became universities after 1992, it might have been expected to lead to changes. Since the 1980s the numbers of students has increased greatly whilst the unit of resource has diminished. The increased pressure on staff was terrifying then and still is. Managers now see some merit in seeking extortionate pay for themselves by squeezing as much as possible out of as few staff as possible. Eventually quality must be sacrificed at the altar of efficiency as limits are reached. Yet government fails to see that TEF is not likely to improve the situation.
Ensuring quality.
There is an underlying problem in the whole system that has been allowed to grow in a poorly coordinated manner. An analogy is what I call 'the British problem' exemplified by Heathrow Airport. Starting life as a small airfield with a 'wooden hut' it has grown piecemeal over the decades. Before one development was completed it was already too small and another was added and so on. The result is a mess of distant terminals, connecting tunnels and corridors and confusion for passengers. Somewhere in the middle, the 'wooden hut' is probably still standing. Planning further into the future and building in some redundancy as a contingency and planning for expansion were ideas too far it seems. Similarly, the university system has become a monstrous competing 'mess' of institutions forced to compete for students; many of whom are lost in the maze designed by REF, TEF and Bobtail. The regulation seems to be an afterthought in response to wholly predictable crisis.
Ensuring quality and, above all fairness, should be at the heart of the system for both the students and the prosperity of the country. Yet neither seems to be addressed logically. One bad idea is 'improved' by bolting on more and more complex arrangements to compensate for the bad idea in the first place. The grade inflation was inevitable as a result of making universities a market place. Then TEF is bolted on later to somehow compensate for the bad idea. Clearer forward thinking at the outset might have helped.
At this point it is best to ignore grade inflation since most employers have already reacted to its presence. Instead, the root of quality control should be addressed. A minimum standard for each subject should be set in place. This includes adequate contact time and assignments. Like any sensible investor, the government should expect to see a forward plan and defined outcomes. The current external examiner system should be 'beefed up' considerably to ensure that the plans are adhered to for the benefit of both students and society.
A student 'Bill of Rights'.
In return for the universities ensuring even more higher standards, students should expect to be challenged and always helped along the way. But it should be fair and equitable for all. The student charters go some way to helping with this idea, but do not go far enough. A 'Bill of Rights' that all universities must sign up to might include a right to attend lectures by the best researchers. Also, to expect a supportive level of tutoring and contact time to back up their experience. Fair assignments and examinations, where every student is given the same time to complete or prepare work, are essential. Students must each be able to devote enough time to their studies and those in financial stress must be supported for an equal chance to study alongside their peers. Tutorials and practical sessions should be designed for effective interaction and where there are not too many students per session. These and other ideas must lie at the core of any contract where the government and the students fund the endeavour.
It should become a right and not a mere aspiration.
[1] Teaching Excellence and Student Outcomes Framework: subject level Government consultation response October 2018. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/750411/TEF_government_response.pdf
[2] Is Britain Fairer? (2018) https://www.equalityhumanrights.com/en/publication-download/britain-fairer-2018
[3] UK universities face grade inflation crackdown The Guardian 22 Oct 2018. https://www.theguardian.com/education/2018/oct/22/uk-universities-face-grade-inflation-crackdown
[4] Universities found guilty of grade inflation will be penalised in rankings under Government plans The telegraph 22 October 2018 https://www.telegraph.co.uk/politics/2018/10/21/universities-found-guilty-grade-inflation-will-penalised-rankings/
[5] HESA Student outcomes https://www.hesa.ac.uk/data-and-analysis/students/outcomes
[6] The complete university Guide https://www.thecompleteuniversityguide.co.uk/league-tables/rankings
Comments
Post a Comment
Constructive comments please.
Did you have a free Higher Education?