A-Levels bake off: UPDATE. Government confirms SNAFU

A-Levels bake off: 

Government confirms SNAFU

UPDATE 17th August 2020.

The whole ‘house of cards’ that constituted the examination system across the UK finally collapsed at 4pm this afternoon. It was expected as soon as the Scottish Government realised their mistake last week and reversed the decision to use algorithm calculated grades based upon the past performance of schools and colleges. Sure enough, once Ofqual let us know on Thursday that they had used similar methods, then the governments in Wales, Northern Ireland and England got cold feet. With Northern Ireland (first GCSEs and later A-levels) then Wales making their moves earlier today, the Education Secretary, Gavin Williamson, was left with no choice but to complete the demolition later in the day. There was an apology from the Chair of Ofqual, Roger Taylor, but it should have been from Williamson first. It has been a new low in how the UK conducts itself and the abject failure of government to connect with, or comprehend, the complex reality of their policies. World news outlets have yet to report on this debacle, but they surely will. The government must accept that they are fully responsible for the construction of the entire dodgy edifice. 

The news today, that all examinations grades in in the UK are now to be decided by teacher, or centre assessed grades (CAGs), has sent shock waves across the whole education sector. Many of the ‘TEN suggestions to reverse out of the exams mess’, that TEFS offered last week, in advance of the results, are being met. But there is still some way to go. A full review and the release of data should also happen to get to the root causes of such a terrible policy miscalculation.

Meanwhile, university admission departments are already at full stretch filling places. Many students were allocated grades that led to universities excluding them. As a result, some universities will have filled their places up to the cap of +5% on last year’s numbers as directed by the government. Now they are faced with the dilemma of many more students matching their offer grades in retrospect. These might come to substantial numbers suddenly flooding the system. Some of these will be higher than those of the students already accepted onto their filled courses. Others will have moved on, as University Minister, Michelle Donelan advised an age ago last week. She seemed to anticipate the problems last Tuesday and was reported (BBC News 11th August 2020) as asking on universities to show "flexibility" and to hold open the places for students whose "grade may change as the result of an appeal". Now the flood-gates have been opened suddenly.

But universities have a contract with students they offered places to that they must honour. The next step must be to lift the cap on numbers to match the moral imperative faced by each university to resolve the mess.

The BTEC problem.

Hiding in the wings is resolving how BTEC results are to be calculated or adjusted. These qualifications are administered by private provider, Pearson. Again, some students have results, and these may have to be reassessed. But it turns out that there are many students without results due to delays that Pearson are trying to reconcile (BBC News today ‘BTec missing results: 'We've been forgotten about'). Now the rules have changed, and Pearson is in a race against time. Have they been forgotten about? TEFS reminded everyone that a sizeable number of students, in excess of 45,000 in 2019, enter universities with all BTEC or a mixture of BTEC and A-levels (see TEFS 28th April 2020 ‘To BTEC or not to BTEC, that is the question’). Ofqual were fast off the mark to explain arrangements for A-levels, but were very slow indeed to realise they needed to do the same for BTEC qualifications. It seems the same is happening now and it is not right.

The problems for students will not recede.

This sorry episode has stressed students and their families beyond anything acceptable. In the middle of a major economic crisis, when many are losing jobs, livelihoods, and even family members to COVID-19, it was a double blow. Now students and their families will be coming to terms with a collapse of family finances. They will be calculating if they can really afford a university place and are looking at applying for better maintenance loan provision. Many part-time jobs that students rely on to continue were blown away in the COVID storm. They government must act quicker on this issue. Universities will be gearing up their hardship funds against a backdrop of not really knowing the extent of the problem that they will face from new and existing students. See TEFS 19th June 2020 ‘Open Letter to UK Government Ministers - taskforce on student support urgently needed’ and the Guardian 19th June 2020 'University students who work part-time need support – or they will drop out'.

It is hoped that this sorry episode can end with a new generation of students totally determined to change the way they have been treated. They will not forgive of forget. The best way for the governments across the UK to apologise would be come together to act fast to make sure students can continue with their education and not end up struggling and dropping out. The ad hoc hardship funding system in place in our universities was not designed to cope in a timely manner with a lot of pressure. Universities UK and the Office for Students have been strangely quiet so far. They must now work with the government to put sustained, equal, and robust support in place for students across the system.

Mike Larkin, retired from Queen's University Belfast after 37 years teaching Microbiology, Biochemistry and Genetics. He has served on the Senate and Finance and planning committee of a Russell Group University.

Original post ‘A-Level bake off: You couldn’t make it up, could you?’

The fury surrounding the A-level results has continued well into the weekend. The president of the NUS said "inequality was baked in". The demand is for a proper inquiry into the way the results were estimated, and why they differed so much from  teacher estimates, termed Centre Assessed Grades or CAGs. The writing was on the wall for some time since the government demanded that standards were maintained as the top priority. Ofqal worked diligently, and out of sight of prying eyes, to satisfy that demand with some modicum of fairness. It was always going to be a compromise and eventually turned out to be a very complex mess. The reasoning behind dividing the assessment of individuals between the teachers and estimates based on past school or college performance remains obscure. The inherent unfairness in the fact that colleges and independent schools tended to fall either side of an arbitrary cohort-size line seems to have eluded the government. This was an obvious outcome from the outset, the only uncertainty was how bad it would become. The jury is in and the verdict is swinging toward using only CAGs. The differences between different schools and cohort sizes is looked at in detail from 2019 data and further reinforces this conclusion.

Ofqual published a complex 319-page document, ‘Awarding GCSE, AS, A level, advanced extension awards and extended project qualifications in summer 2020: interim report’, on Thursday that attracted considerable attention. Its complexity will take more time to sink in as the full horror of what they were doing becomes clearer. Within the copious statistical analyses, it emerged that they tried various and different models to assign grades to students in the absence of examinations. The first thing obvious was grades would be very difficult to predict accurately from previous cohorts of students if the numbers were very low. A cut-off at 5 (below which only CAGS would be used) and then 15 (below which teacher assessments would be used to some extent) was decided. With cohort sizes increasing above 15, there would be increasing confidence in predicting a fair grade, but this would still be uncertain. The method used for the larger cohorts relied upon centres and teachers ranking each student. Ofqual could then allocate a grade to each depending on the past record of the centre. One thing for sure, the outstanding students risked being marked down using the methodology. This was always going to be a compromise. However, their justification on the grounds of accuracy is tenuous. 

The Fischer Family Trust (FFT) Education Datalab has provided some of the most accessible analyses of the situation with ‘A-Level results 2020: How have grades been calculated?’ and ‘A-Level results 2020: Why independent schools have done well out of this year’s awarding process’. 

Ofqual analysed 2019 results to test their models for what they call ‘predictive accuracy’. The unsurprising conclusion is that accuracy rises with numbers of students. With high numbers, Ofqual decided that they could be over 90% accurate “within one grade”. So, they assumed this would also be the case in 2020. 

But the accuracy prediction method has one flaw that could not be avoided. George Constantinides, Chair of Digital Computation at Imperial College explains this flaw very well in his posting today ‘A-Levels and GCSEs in 2020’. He spotted that, with no teacher rankings available last year, “the actual rank order within the centre based on the marks achieved in 2019 were used as a replacement“, i.e. they back-fitted 2019 marks to rank orders”. The issues raised by Constantinides are echoed by a stern letter from the Royal Statistical Society (RSS) . This follows considerable concerns they raised in their ‘Statement on grade adjustment in UK exams in 2020’ and also raised with the House of Commons Education Committee who demanded that Ofqual “must urgently publish the evidence thresholds for proving bias or discrimination…. This must be communicated to parents and pupils in advance of results day” on the 11th July with ‘Getting the grades they’ve earned: Covid-19: the cancellation of exams and ‘calculated’ grades’. The RSS question the approach of Ofqual with regard to ‘trustworthiness, quality and value’. The word ‘Trust’ does not appear in the Ofqual report and on ‘fairness’ they state “Without standardisation there was the potential for students to be unfairly advantaged or disadvantaged, depending on the school or college they attended and the approach they took”. Many think they have fallen well short of this goal. 

Trust and the Blame Game. 

The Telegraph yesterday reported that Ofqual was retaliating by blaming the teachers with ‘Teachers accused of submitting 'implausibly high' predicted grades as A-level results row grows’. Although contentious, even inflammatory, the Telegraph was correct in highlighting the attitude of Ofqual. The approach all along appears to be one of little trust in the schools, colleges, and their teachers. This means that Ofqual can no longer command the trust of those educating in the front-line and it’s days are numbered. A new system is needed. Other observers are coming to the conclusion that reverting to CAGs is the only fair way to go as was concluded in Scotland last week (see TEFS 4th August 2020 ‘SQA cements the inequality ‘goal posts’ into the ground’). However, the Scottish government had the added incentive of an upcoming election in 2021 with 16 year-old students registered to vote. Nicola Sturgeon may be many things to voters, but she is never indecisive. 

What happened to Ofqual this week was seen coming a long way off. The Guardian reported yesterday that IT consultant, Huy Duong, had “long warned 39% of grades between A* and D would be lower than teacher assessments”. Ofqual were caught red handed some time ago but did not own up. The simple fact is that the decision to standardise on old data was a political one taken by the government. The root of the current problem arises from the original letter sent by the Education Secretary, Gavin Williamson, to Ofqual back on 31st March 2020. Without mentioning ‘fairness’ or ‘trust’ he demanded “Ofqual should ensure, as far as is possible, that qualification standards are maintained and the distribution of grades follows a similar profile to that in previous years”. This they did and the consequences are on the faces of astonished teachers and disappointed and confused students. Earlier this week, TEFS considered this was indeed a deliberate move to maintain existing inequalities and even make things worse. 

Size matters.

Once the political decision was made to prioritise ‘standards’ above all else, Ofqual had little choice but to plough on. However, it is very likely that some of the professionals within their organisation voiced concerns about differences between students in different exam cohorts. If the government was made aware of this, then it was a wilful decision to carry on. The statement within the Ofqual report on setting thresholds in cohort sizes offers some idea of the dilemma in which they found themselves, “to strike the balance between statistical defensibility, the potential consequences for outcomes and commonality of handling across centres, the values of 𝑛thresh and 𝑛small were set to 5 and 15, respectively”. This set off series of events that have yet to be resolved. 

The Ofqual interim report revealed the type of anomaly that fed the idea that this was deliberate. Overall grades and passes had shown a slight increase on those of 2019. However, this turned out to be more pronounced for candidates from the independent schools. Table 9.10 from the report is reproduced below and the uplift in A and A* grades in independent schools is striking.

The reasons for this might lie in the class sizes and thus the size of the cohorts from these schools. OECD data from 2017 indicates that state schools in the UK (designated as lower secondary education in public educational institutions) had a mean class size of 24.3. In contrast, for similar students in ‘independent private educational institutions’ the mean is only 12.4. This does not necessarily indicate that cohort sizes will vary, but I decided to take a closer look at the large Schools performance data set for 2019. It seems the same data will be withheld in 2020 so a similar analysis will be difficult. In a written statement by Gavin Williamson in House of Lords on 23rd March 2020, ‘Impact of Covid-19 on Summer Exams:Written statement - HCWS176’, he tagged on the end that "The Government will not publish any school or college level educational performance data based on tests, assessments or exams for 2020".

From 2019, the first thing apparent at one end of the spectrum is that independent schools tend to have smaller cohorts entering A-level examinations in many subjects.

Figure 1 shows in simple terms the proportion of examinations with cohort numbers greater that fifteen, the cut off 𝑛small set by Ofqual. Those above this were awarded grades based on teacher rankings and past school performance only. The second striking thing is the large proportion of exams with cohorts over 15 arising from colleges. I lumped together all sixth-form colleges and further education colleges in this analysis. All secondary schools included ‘community schools’ or comprehensives and all voluntary aided schools. All academy schools and free schools, funded directly by the government were considered in the last category. 

The distribution of cohort sizes was then a consideration. Figure 2A shows how the cohort numbers are distributed across the four types of school. Again, independent schools fair well with a sizeable proportion of their subjects assessed by their own teachers. Colleges are at the other end of the deal since they have much larger cohort sizes. In a few cases, these exceeded over 200 entries. However, as is generally the case, the percentages can hide the reality of the actual numbers. It is the total numbers of students added up in each cohort ranges that reveals why there is major trouble brewing. These numbers are displayed in Figure 2B (numbers for <5 are estimates using a mean of 2.5 as actual numbers are suppressed in the data). For example, a small number of subject entries with cohort sizes greater that 105 translates to a much larger number of individual exam entries. This reveals an alarming picture and why so many stories of disappointment are emerging. 

What next? 

Ofqual are not out of the woods yet. They are only half way there with GCSE results coming out next week. The Guardian yesterday launched another fierce attack on the standardisation system with ‘GCSEs: 2 million results set to be downgraded, researchers warn’. The cat is now out of the bag and there will be even more anger next week. 

Interestingly, the Ofqual interim report has a strange quirk that should be explained. They say that “The predictions include all students (of the appropriate age) that are matched to their prior attainment, except for GCSE where students from independent and selective schools are excluded”. They go on with “This is because evidence suggests that students from independent and selective schools perform differently to the overall cohort……. We do not think that this would be fair, hence these students are routinely excluded from the predictions”. Could it be that they observed that students from independent schools might get lower GCSE grades in a prediction and this is to be avoided? What about the rest? 

Most observers are now thinking that the CAGs approach was the best and only way to go after all. Schools could have been given clearer guidance at the outset to assign grades. Instead of the energy expended on the contorted statistics and logic that emerged, effort should have been diverted into a way to check what schools were doing. This might have involved groups of schools checking each other’s approach with the expectation that there would be spot-checks and oversight to ensure rigour. Outstanding students would be clearly identified, and ‘real’ live data would be used. In the end, not one would be sacrificed at the altar of ‘standardisation’.

Mike Larkin, retired from Queen's University Belfast after 37 years teaching Microbiology, Biochemistry and Genetics. He has served on the Senate and Finance and planning committee of a Russell Group University.