The Dystopian Shadow of Citizen Scoring

POSTED BY   Michele Baker
27th October 2017
The Dystopian Shadow of Citizen Scoring | TDMB Tech

On Wednesday night, I had this brief WhatsApp exchange with my best friend:

 

 

 

 

That’s when I started thinking about what I’d been reading about China a bit more seriously. As if by magic, I then stumbled upon Wired’s article from 21st October on precisely that.

If you haven’t yet been acquainted with the unsettling idea from the East, let me briefly explain. In the State Council of China’s “Planning Outline for the Construction of a Social Credit System”, published back in 2014, the idea of a Citizen Score was proposed. By compiling all the data available on a person (and extending the types of data and ways it can be collected), the idea is to boil it all down to a single number, born out of assigning all behaviours a positive or negative value. Your number is your Citizen Score and denotes how trustworthy a person you are. That number will determine your eligibility for work, for finance (including a mortgage), your ability to rent a home, even where you can and cannot travel.

It sounds like one of those dystopian nightmares that will never come true, right? Well, no. The bad news is that the Social Credit System is already being rolled out, slowly but surely, across China’s 1.3 billion citizens. The Chinese government clearly think it’s a great idea, stating:

“It will forge a public opinion environment where keeping trust is glorious. It will strengthen sincerity in government affairs, commercial sincerity, social sincerity and the construction of judicial credibility.”

It’s taking a softly-softly approach right now. It’s voluntary to offer yourself up to the Social Credit System, presumably with the aim of encouraging complicity and demonstrating the benign and beneficial qualities of the strategy. But by 2020, it will be mandatory.

Softly-softly as the roll-out may be, however, people are still being coaxed in on the promise of rewards and special privileges for those who can demonstrate a good score.

High Score/Low Score?

One of the Big Data companies behind the initiative is Sesame Credit, who – Wired reports – measure individuals on a score of between 350 and 950, taking into account several categories for calculation. The first is, predictably, credit score. Next is ‘fulfilment capacity’, i.e. how able a person is to fulfil their contractual obligations. Personal info, such as phone number and address details are next. But then, there’s the fourth category, behaviour and preference.

Citizens are thus judged by the products that they buy. “Someone who plays video games for ten hours a day, for example, would be considered an idle person,” says Li Yingyun, Sesame’s Technology Director. “Someone who frequently buys diapers would be considered as probably a parent, who on balance is more likely to have a sense of responsibility.” 

The plot thickens still further when we look at the fifth category: interpersonal relationships.

If somebody were to assess you based on your online friends and your interactions with one another, what would they think of you? What judgments might they make of the sort of person you are? Would you be an ideal citizen?

I mean, I’d probably be okay because the worst thing I have ever done is romp in fields of wheat, with my virtuous conservative friends, of course. But what about you? What did you do last summer?

Even if you are the pinnacle of virtue amidst a writing mass of depraved buddies, your Citizen Score would still be negatively affected. All it takes is for that old uni pal of yours who sits around smoking weed and moaning about the world’s ills to post a rant online that the government disapproves of, and your own score will sink quicker than a grand piano full of cement.

Moreover, your score may block access to potential relationships – your score affects who you can see on Baihe, the Chinese version of Tinder.

You may think this could never happen in the West, and nor did I. But then, my beloved bezzie sent me this link on Thursday this week:

 

 

 

 

 

 

Just for the record, I just want to say that my friend is far from a conspiracy theorist. She’s into gin and unicorns mainly. But the way she’d picked up the overlap between these two bits of news interested me.

The link she sent me was to a Mashable article about Google’s Sentiment Analysis API, which Motherboard reported this week as holding some serious biases, notably against ethnic and sexual minorities. Biases in AI are nothing new, of course. It’s an issue that’s causing much anguish in the tech and academic communities at present. But when it comes to the use of sentiment analysis and its potential applications in a Chinese style social control experiment, I start to get a bit angsty.

Google’s Cloud Natural Language machine learning API launched in July this year, as a way to “easily reveal the structure and meaning of your text in a variety of languages.” Sounds innocuous enough, right? Well, that’s until you discover what ‘sentiment’ it attributes to statements such as ‘I’m gay’ and ‘I’m black’. Or, indeed, “I’m a gay black woman”. This has echoes of the news about that AI that claims to distinguish gay from straight people from photos alone, another development this year that has troubling undertones.

The API assigns a sentiment score in + or -, with 0 as neutral. The higher the number above zero, the more positive the statement is deemed to be. Conversely, if the statement returns a score below zero, it’s deemed a negative statement to make. Check this out:

Google API via Motherboard

If that doesn’t prove the point, how about this?

Google API via Motherboard

China seems far away, and quite a different culture to that in which we live in the West. But how much less eager would we be to sign up to some Citizen Scoring program? Well, we already hand over all the data that our own governments need. Who are we handing it over to? Well, all the Silicon Valley megaminds, that’s who. Including, yes, Google. That’s right – the company behind the above algorithm.

Surely what’s happening in China would never happen here, right? I mean, it’s not as if we make dumb political choices in the West. It’s not as though we allow those in power to make shocking decisions or to manipulate us with the aid of media and technology, do we?

What is the worst that we do in dissent of political actions that we disagree with? Post a rant on Facebook or share a stupid meme? Shake our heads and cluck our tongues over a nice cup of tea?

I’m not saying we should panic, but… perhaps it’s time to consider how we register our dissent, and to pay attention to things we might be dissenting of. Technology can be an amazing thing – I, for one, am all for it – but it’s a double-edged sword. As citizens, we mustn’t let ourselves sleepwalk into an Orwell novel.

 



RECEIVE OUR PICK OF THE GLOBAL NEWS ON AI AT 9.30 AM EVERY SUNDAY
Don’t worry, we’ll never share details with anyone.

1

Michele Baker

Michele Baker is the Senior Content Strategist at TDMB. She began her journey into tech marketing via a Masters in Creative Writing, evolving from a prize-winning poet and short story writer to a futuristic content guru. Michele now writes endlessly about all aspects of technology, hosts the TDMB Presents… tech podcast, and speaks at numerous tech and marketing events.

You may also like

Coca-Cola: Big Data Pioneers
21st September 2017
Big Data
The People’s Bank of China’s New Digital Currency
29th June 2017
Big Data

One comment

  1. Matt

    Fascinating article and certainly thought provoking. Would be interested to read any updates you have for this
    Michele.