Do Not Use Algorithms To Mark Exams!

POSTED BY   Michele Baker
6th July 2018

Recently, I made the controversial decision to move my daughter from a state primary to an independent school. My instinct was that she was failing to thrive at her current school. Being an exceptionally creative and intelligent person, she had reached the limit of what her teachers were able to offer her in terms of academic advancement. This is not the teachers’ fault. We all know how strapped they are, for numerous reasons.

The fact, however, is that the majority of the work of a teacher in a state primary is to bring students who struggle up to an acceptable level to pass government-issued testing criteria. Those with excess ability have trouble travelling far beyond that acceptable level, as the attention is centred on helping those less well off academically. It is great that our teachers work so hard on those who find learning in a school environment taxing, but the regulation SATS and so on mean that all students are being steered to learn in the way it takes to pass those tests.

One of the things I was told when visiting the independent school I eventually chose for my daughter, which cemented my decision, was this: “Oh, we’ve done away with SATS. We found they limited creativity and lateral thinking by focusing too hard on technicalities.” Independent schools are permitted to do this. Is it elitist? Of course it is. And make no bones about it, it’s not going to be easy to keep her there.

Why am I telling you this? Well, to illustrate, before I launch into this piece, my views on standardised testing, my aversion to box-ticking, and my insistence on a creative, holistic education for all children at all levels.

So, do I think it’s a good idea for educational institutions to use algorithms to mark pupils’ essays and exams? It’s not a flat ‘no’, but I’m far from enthusiastic.

The Malicious Marking Machine

In the news this week, I read that some US states are using ‘robo-graders’ to mark students’ work. Sure, it’s a brisk task-saving device. But it’s fraught with problems.

Perhaps using an algorithm to mark maths and science papers could be beneficial. Anything where there’s a factual, quantifiable answer, I see little reason why a machine could not grade effectively. Providing, that is, that the paper is also checked over by a human marker too.

Other subjects in the curriculum simply cannot be effectively graded by a computer, and it is outrageous to think otherwise. My views are shared by two English teachers who gave quotes for the article in NPR:

“An art form, a form of expression being evaluated by an algorithm, is patently ridiculous,” says Kelly Henderson, a high school English teacher from Boston, MA. “Is it going to reward some vapid drivel that happens to be structurally sound?”

Robyn Marder, another teacher, went on to say, “What about original ideas? Where is the creativity of expression? A computer is going to miss all that.”

It’s not just English papers that it is ludicrous to mark with an algorithm. History, foreign languages, law, business studies, theology, classical civilisations. These are all subjects where the quality of a paper cannot be reduced to mere calculations.

MIT research affiliate, Les Perelman, demonstrated the ridiculousness of the idea with an algorithm of his own, shrewdly called The Babel Generator (Basic Automatic BS Essay Language). The programme collects together particular words and phrases that tend to perform well in papers, and uses them to generate a high-scoring paragraph of its own. This is one that achieved a 100% mark in the GRE automated scoring system:

“History by mimic has not, and presumably never will be precipitously but blithely ensconced. Society will always encompass imaginativeness; many of scrutinizations but a few for an amanuensis. The perjured imaginativeness lies in the area of theory of knowledge but also the field of literature. Instead of enthralling the analysis, grounds constitutes both a disparaging quip and a diligent explanation.”


What a load of twaddle. And yet, a student who wrote this would pass their exam with flying colours. The GRE defines full marks scores like this as “present[ing] a cogent, well-articulated analysis of the issue and conveys[ing] meaning skillfully.”

Machines work off numbers. So they can quite happily mark the answer to 2 x 2 = 4 with no issue. The same, however, cannot be said of the numbers involved in dates. “You can write that the War of 1812 began in 1945,” says Perelman. “and that wouldn’t count against you at all.” In short, the algorithm cannot understand meaning.

Students have already demonstrated an aptitude at gaming the system. They work out what the system likes and do that, regardless of whether it makes any sense or if they even really understand the question. To me, this doesn’t sound a far cry from how we expect students to prepare for exams anyway. But it sucks.

Nitin Madnani, senior research scientist at ETS, the company behind the GRE automated scoring system, actually thinks that it’s great how students are gaming the system.

“If someone is smart enough to pay attention to all the things that an automated system pays attention to, and to incorporate them in their writing, that’s no longer gaming, that’s good writing,” he says. “So you kind of do want to give them a good grade.”

No, Madnani, that is not good writing.

I hold the view that the education system as it stands is committed more to churning out worker drones than developing the potential of individuals. This cannot continue, and not just because it’s a bit gross.

The workplace of the future will be completely different from the one we have now. Automation is already beginning to take over the mundane parts of our jobs, and soon the majority of tasks that can be automated will be automated. What does that leave us with?

Well, you could become a computer scientist and help build the machines that automate ever more tasks. But there’s a shelf life on that. We’ve already seen software that can create software. Robots can easily create the hardware that the software sits within, so before long, all but the very highest of roles in computer science will be eliminated.

The safe haven for us remains in the one thing we have that machines do not. In our creativity, our interpersonal skills, our understanding of the world. An algorithm will never be able to match these elements of humanity, probably not even when or if we create a ‘consciousness’ code for them. As Dr Joanna Bryson put it at her CogX 2018 talk, “We will never build something from chips and wires that shares our visceral experience as much as cows do”.

Mark Cuban also recently spoke out to say that he believes philosophy degrees will become more valuable than computer science ones. That’s right. All of those jokes you hear about arts grads ending up in call centres could not be more wrong as the wave of automation rolls on.

If we try to quantify creative work, reduce it to spelling, punctuation and high register vocab, we will only be propagating the same bullshit education we’re haplessly dishing out to the next generation as it is.

Our children deserve better. We need to be preparing them for the future they’ll grow into, not the drudgery and boredom of the workplace we and our forebears have been stuck with for generations. Things will be different for them, and they need to use their human minds to their full potential.

So, by all means, use a machine to check spelling and grammar, to mark basic maths, but please, please leave it to the teachers to decide what makes a good piece of work.

Don’t worry, we’ll never share details with anyone.

Do Not Use Algorithms To Mark Exams!

Michele Baker

Michele Baker is the Senior Content Strategist at TDMB. She began her journey into tech marketing via a Masters in Creative Writing, evolving from a prize-winning poet and short story writer to a futuristic content guru. Michele now writes endlessly about all aspects of technology, hosts the TDMB Presents… tech podcast, and speaks at numerous tech and marketing events.

Get in Touch With Michele Baker

01306 632 854

You may also like

Surviving AI: November’s TechTech Book Club
The Question Of Autonomous Vehicles and Morality
Our Pick Of The Best Robotics and Artificial Intelligence Events 2019