Qfqual builds a concrete wall: UPDATED

UPDATE 8th August 2020

Things are moving fast today with severe criticism mounting about Ofqual and SQA, and urgent action is needed. TEFS has laid out ten points that should be considered to reverse out of the crumbling mess. Fairness should replace 'maintaining standards' as the primary objective. The government must cease trying to defend a system that acts as a barrier to the less advantaged.

Since posting yesterday, things have been moving fast. Today the Guardian put the examinations issue in large print on its front page with ‘Nearly 40% of A-level result predictions to be downgraded in England’. This conclusion came about after some great detective work by former medical statistician, Huy Duong, who analysed the data available and reconciled this with the Ofqual announcement that there could have been a 12% inflation in higher grades. It seems that Ofqual have been caught red handed and "Duong’s findings were privately confirmed to the Guardian by exam officials”. This is a terrible embarrassment for both Ofqual and the government who tried to keep their methodology under wraps. Duong has proved too clever for them in his forensic analysis and they must now come clean and admit their folly.

Further analysis was expected from another relentless investigator of the Ofqual methods, Dennis Sherwood. Sure enough, he pitched in this morning with a new post for the Higher Education Policy Institute ‘Something important is missing from Ofqual’s 2020/21 Corporate Plan’. The things missing all along are the qualities of “reliable and trustworthy”. Both must be in place before it can be called ‘fair’. Yet School’s Minister, Nick Gibb (Independent school and Law graduate from Durham University) was on Radio 4 this morning to say they had ensured it was 'fair'. 

Sherwood also notes that past restrictions on the grounds for appeal and re-marking had caused a large decline in appeals that were rising due to an inherent unreliability of marking that was “fuzzy” at the grade boundaries. Surely appeals must be allowed and dealt with quickly. The SQA have already denied to the Scottish Sun yesterday that ‘SQA deny Scots pupils will need to wait NINE MONTHS for appeal grades after calls for Swinney to resign’. But they must produce a timetable quickly outside of their usual procedures and surely might have anticipated this. 

Meanwhile, the somewhat insipid response to the criticism in the Times Educational Supplement (Tes) report yesterday, ‘Exclusive: Teacher grades ignored in most GCSE results’ does not inspire confidence. In their ‘Response to TES story about centre assessment grades’ they simply promise that “We will publish full details of the moderation process, including details for low entries, next week.” It gives them a few days grace at least.

Where did it all go wrong?

George Best was once asked this when he was in his hotel room with Miss World, a bottle of Champagne and thousands in cash from his winnings earlier. It may have been asked with some degree of irony, but Ofqual are not in a position to laugh of their predicament. Unlike the students of 2020, George Best was not judged by the past record of the teams he played for. When Manchester United were declining in the 1970s, he was still judged for his ability that stood above that of the team. I was lucky to see him play several times closeup. He was phenomenally quick and that ability was not diminished by the team failing to keep up with him.

It seems that the government has called the shots to date and this arises  from the original letter sent by the Education Secretary, Gavin Williamson to Ofqual back on 31st March 2020. This initial approach laid the foundation for what has become a very shaky edifice today. Unwise moves years ago by Michael Gove on examination reform (see The Student Room ‘Five ways Michael Gove has changed GCSEs and A-levels’) only made the scope for a future mess much greater.

It is astounding that the words ‘reliable’, ‘trustworthy’ or ‘fair’ do not appear in the instructions in the Williamson letter. These are three of the most important imperatives in any examination process. But the omission says a lot about the mindset of the Minister and that of the government. The emphasis is all on ‘risk to standards’ with “In order to mitigate the risk to standards as far as possible, the approach should be standardised across centres.” Never mind fairness then.  Trust departed a longtime ago.

No doubt blame for the current mess will fall onto Ofqual's shoulders. But they should not be blamed. Mistakes in Government policy for some time have led to this situation. However, Ofqual must admit that they pressed on with the government priority as instructed. They might have kicked back along the way, but we may never see if this happened unless there is a whistle blower in the wings or an independent investigation. 

What to do now?

Firstly, the mess will not go away. The whole edifice of the process is very shaky and unstable because of a  weak foundation. The only thing to do now is make it safe – in terms of fairness and trustworthiness – and then demolish it carefully before building something more reliable and substantial. In organising examinations and assessments for hundreds of university students for many years, it was always the case that the individual was the only focus of any alternative assessment. If  examinations were missed for a reason not the fault of the student, then their individual performance to date was considered. It would take a particularly stupid university examinations board to assess a student on the basis of the performance of others in the previous year as the only guide. The excuse of saying they may be very good but they might make the standard look bad is not valid. Why then should Ofqual sacrifice fairness at the altar of maintaining standards? Could it be that fairness is not the objective?

Here are ten suggestions (not necessarily to be carved in stone) that must be considered in order to reverse out of the mess, make it safe in terms of fairness and then start to rebuild trust.

1. Go ahead with the current grades as it is too late to stop now with UCAS and clearing in the wings.
2. Reveal how the grades were calculated in detail with data available openly
3. Report the awarded grades to the candidates, UCAS and the universities along with the predicted grades and the past performance of the school in each subject. This must include n, standard deviation and range.
4. UCAS must set out guidance on how to use the data on each candidate as an individual. This includes a fair way to make contextual decisions that reflect the work and determination of the candidate along with potential they offer.
5. All universities, including the elite research ones, should take context into account when awarding places this year. They should go further and pilot this as a way forward from next year.
6. Appeals should be stopped at once. It will only advantage those from better off families that push harder and will be socially divisive and unfair. 
7. The entire process should be put under independent review to reassess all grades across every school and centre. It must be very robust and Qfqual and the government should not be allowed to hide in the corner.
8. Resits in the Autumn might go ahead, but this must be uncertain as some will be impeded by more lock-downs by then. Students already in their first term at university should be given the same chance to improve since A-Level results will still be important in their later job applications. 
9. No student should have their results downgraded because of the review exercise. But if there are systemic errors and mistakes evident, then they should be openly acknowledged, and lessons learned.
10. Regrading checks must be completed by the summer of 2020, and before the completion of the next round of examinations, to expedite university decisions for those trying again.

In the midst of mounting criticism and anger, fuelled by realisation that the whole system is inherently unfair, it seems that a new approach is needed. There is no evidence of ‘levelling up’ at the moment, only a determination to build layers of ‘concrete’ barriers in the system that make it harder for students who have less opportunity through no fault of their own.

The government has hidden behind the smokescreen of an unfair system, that they constructed, for too long and favoured those who,  like themselves, had advantages. Its time they moved over and allowed others to spread their wings to improve the lot of the whole country.

Mike Larkin, retired from Queen's University Belfast after 37 years teaching Microbiology, Biochemistry and Genetics. He has served on the Senate and Finance and planning committee of a Russell Group University.

EARLIER BLOG post is below.

Friday 7th August 2020

This is an update on the earlier posting this week about the SQA results that came out on Tuesday (TEFS 4th August 2020 'Scottish Qualifications Authority cements the inequality ‘goal posts’ into the ground: UPDATED'). Now it seems the inherent inequalities that exist in the system are to be further exacerbated by Ofqual in England. 

Yesterday Ofqual followed the lead of SQA in Scotland and is beginning to loosen the grounds for students to appeal their exam results due out next week (‘Appeal arrangements for AS, A levels and GCSEs’). This appears to have been the result of mounting pressure on Ofqual and a storm of criticism after the Scottish results were released on Tuesday. The possibility of numerous mistakes must be looming. However, other revelations earlier today about how the results have been calculated in England are very disturbing. It seems that Ofqual in England are building a concrete wall around the goals in addition to cementing the posts in the ground.

Back on 15th May the Ofqual ‘Exceptional arrangements for exam grading and assessment in 2020’ indicated that the Secretary of State had directed that “students should have access to a right of appeal if they believe the process was not followed correctly”. Also, that “the appeals process should focus on whether the right data was used and correctly applied, rather than on teachers’ professional judgement”. This has largely stayed the same as proposed in May. It is confirmed the students can opt to take the examination in the Autumn, however the possibility of another more serious lock down is looming. 

There is a fear that “if one student successfully appealed against their position in the rank order, it would have negative implications for other students who would, in turn, need to be given an opportunity to appeal.” This sounds like a ‘red herring’. It is hard to see why this could not be dealt with effectively by promising to maintain the position of all students in the case of such an appeal succeeding. Indeed, there is a such a possibility possible if a student appeals on the grounds of a genuine mistake in the ranking and “the centre itself made an error when submitting a centre assessment grade or rank order information”. The whole idea is logically inconsistent.

In the earlier proposals in May, Ofqual concluded that “we do not believe it would be meaningful or appropriate for students to appeal on the basis of their centre’s judgement of their likely performance in the exams……or on their position in the centre’s rank order”. 

At the end of June, in ‘Consultation on statutory guidance in relation to the GQCovid regulatory framework’, Ofqual retreated a bit by opening up the possibility of appeals on the grounds of “malpractice” or students “have concerns about bias, discrimination or any other factor that suggests that a centre did not behave with care or integrity when determining the centre assessment grade and/or rank order”. This could open the floodgates driven by angry parents who do not trust the teachers. No doubt those with more resources will game the system if they spot a gap in a teacher’s defence. However, most schools will have taken the wise precaution of cross checks and collective decisions in anticipation of this likelihood.

The guidance today goes a little further by explaining of how errors in data sets will be addressed upon appeal. Schools can challenge the decisions “if they can evidence grades are lower than expected because previous cohorts are not sufficiently representative of this year’s students”. 

Now there’s the rub.

Times Educational Supplement (Tes) reported today with ‘Exclusive: Teacher grades ignored in most GCSE results’ how cohort sizes will determine the precise methodology used. It also explains the earlier need to acquire teachers grade predictions alongside ranking of students. Tes have gathered that any cohort size above 15 will have grades determined entirely by the historical record of the school. Any cohort size below 5 will have to rely on the teacher or school predictions entirely. It is not clear what will happen for cohort numbers between 5 and 15. I am assuming that these are cohort numbers for individual subjects. Thus, subjects will small numbers will have the advantage. Larger cohorts for more popular subjects will fall back on the school history in that subject. Tes has found out that “one exam board has told Tes that in large-entry subjects around 60 per cent of grades will be based purely on statistical modelling”. This is a staggering revelation that is bound to reverberate around the school system for a long time.

Smaller class sizes in independent schools may afford them a major advantage in being able to use teacher/school assessments while their friends in state schools are lumbered with their historical performance only. Add the further possibility that state schools in disadvantaged areas may have even larger class sizes and things will look even worse. OECD data from 2017 indicates that state schools in the UK (designated as lower secondary education in public educational institutions) had a mean class size of 24.3. Of course this may not reflect the cohort size in any one subject if there are duplicated classes. In contrast, the OECD data shows that for similar students in ‘independent private educational institutions’ the mean is only 12.4. It doesn’t take a ‘rocket surgeon’ to conclude that independent schools are more likely to fall below the cohort size cut-off set at 15 in any one subject. It will take a more rigorous analysis of the equality bias that could emerge, but it can only reveal further the inequalities in our educational system.

Dennis Sherwood, who runs the consultancy ‘Silver Bullet Machine’ and has been a consultant for Ofqual in the past on grade inflation, raised the horrible spectre last year that ‘1 school exam grade in 4 is wrong. Does this matter?’ He has been very critical of the current system in 2020 in more postings for the Higher Education Policy Institute (HEPi) with ‘Hindsight is a wonderful thing: Ofqual, gradings and appeals by Dennis Sherwood’ in July and ‘No test is better than a bad test’ in June. TEFS has been in correspondence with Dennis on these points and I value his judgement. His current concerns will have raised some alarm bells in Ofqual that they might have heard much earlier. We look forward to reading his considered analysis of the current system when all the details are finally revealed.

There is a major storm brewing that may breach concrete walls.

It seems the possibility of appeals on the grounds of faults in the whole system is not considered. This could bring about three outcomes for the government and Ofqual. Firstly, a demand to see how it is all worked to be made more open. Secondly, this will prompt statistical experts to look very carefully to determine to what extent the results were somewhat random or biased and not reasonable. Then the coup de grâce, a class action court case. 

When making decisions about students in a university setting, particularly where there are genuine mitigating circumstances for a student, I always reminded everyone that what we must ‘pass the barrister test’. That is asking, could we defend our decisions in court? If there are any doubts, then we look again. I hope Ofqual, directed by the government, has heeded the same advice.

Mike Larkin, retired from Queen's University Belfast after 37 years teaching Microbiology, Biochemistry and Genetics. He has served on the Senate and Finance and planning committee of a Russell Group University.