TL;DR - I'm listening to audiobooks, some reviews below, and I would love to get some recommendations from you.
This is the 3rd part of a series of (audio) book reviews Here are the previous posts:
Part 1
Part 2
Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, Cathy O'neil:
Short summary: Computer based decisions are every bit as biased as people, and less transparent. They should not be blindly trusted, should be used cautiously, and must be constantly monitored.
What I have to say: I find this book a must-read (or hear, apparently) for anyone who's taking part in software manufacturing, acquisition or regulation. It's probably a good idea to listen to this book if you are only using software. It's not that the book is presenting a revolutionary idea, or that it masterfully narrated (although I did find it quite enjoyable) - it is the way it makes something we are all aware of to some degree very explicit, and shows how prevalent is the problem it discusses. In short, the title of the book lays it all quite clearly - There are very harmful algorithms out there, and they pose a significant threat to the society. That's why they are named Weapons of Math Destruction (WMD, for short).
But, putting aside they hyperbolic phrasing, what is a WMD? and why do we care?
a WMD is a software using some sort of mathematical algorithm to achieve a task, which have the following three properties:
- It's pervasive. It doesn't matter how nefarious is the algorithm I use to manage my local neighbourhood book-club, it's not a WMD unless it affects a large amount of people.
- The algorithm is opaque. Visible algorithms are regularly scrutinized (or at least, can be scrutinized), and they lay out the rules quite clearly - so anyone affected by the algorithms can assess the expected outcome and act to change it. Or, if the system is measuring completely the wrong things, they can be challenged easily enough.
- Damage. Some algorithms are using bad math, some of those are scaling up rapidly, but only some of those are causing significant damage to people under their influence.
A bit abstract, right? Most of the book is dedicated to discussing some of such algorithms, and showing the types of damage they create. Some of the most damaging algorithms are created with the best intentions in mind, and that is the main problem: The people using them are thinking they are actually doing good. Two examples that stuck in my mind are the teachers grading algorithms, and some criminal risk assessment programs used to help judges decide on the length of imprisonment.
The teachers grading algorithm is simpler, since it has one main flaw - it uses lousy math (in this case, trying to draw statistical conclusions based on the achievements of 20-60 students). From the examples in the book, it is quite evident that this model has nothing to do with reality. So this algorithm is used, because it seems "objective" and "fact based", where in reality it is pretty much a random number generator that should not have been used at all.
The second WMD is a bit more intricate and complicated. The problem is that the software seems to be pretty much benign: it helps a judge assess the level of danger a convict poses in order to determine their punishment, or to assess their chance of recidivism when considering parole. The reasoning behind it is simple: more of a risk a person presents to society, longer should this person be detained, imprisoned or at least carefully watched. That way, minor offenders could get out, leaving the law enforcement mechanism to deal with bigger problems. Chances are, that the algorithm predictions are fairly accurate, too - the company selling it has an interest in keeping it accurate, or seemingly accurate to sell it to the next state and fend off its competition. There are, however, some caveats: First, the algorithm; being the competitive advantage on the competition, is secret. Normally, a judge must explain the motives behind a given verdict, and those reasons can be challenged or limited. No judge today would say "I decided for a stricter punishment since the convict is poor, and therefore is more likely to commit crime again", and yet - the statistical model might do exactly that. There is a correlation between poverty and crime, and between poor neighbourhoods and criminal opportunities, so the model, measured for "correctness" will be more effective to use that. Even if we won't provide the income level of a person, there are a lot of proxy measurements that are highly relevant: Area of residence, whether the convict has a home or a job to go back to, even how many times was this person arrested in the past has some correlation to their financial situation, as wealthy people tend to get arrested less for minor misdemeanors.
On top using discriminatory elements, there's another risk for this WMD: it creates what the author calls "pernicious feedback loop". Meaning, the algorithm results are actually creating the reality it attempts to predict.
Imagine that: Two people are detained for drunk driving. One of them gets a low recidivism score and therefore is released with a warning. The other gets a high score, and so the judge chooses a more deterring punishment and sends him for 6 months in jail. Six months later, when getting out of jail, this person finds that it is more difficult to find a job with a criminal record (and the longer one was sentenced, the harder it becomes), and he got to know some "real" criminals while in jail, so when things will get rough, the route to crime will be ever more tempting. Point for our recidivism algorithm! The one marked as a likely felon was indeed the one who returned to crime. What did you say? it was only because of the first score he was given? Naaa, this is science, and science is always right, isn't it?
So we got an algorithm that discriminates weak populations, and then actually harms their lives and makes it harder for them to make their way in the world. Fun, isn't it?
Unlike the teachers assessment program, the recidivism model can be used for good, since wheher or not we like it, there's no denying that it is possible to correlate life circumstances with chance of recidivism. People without steady income, or with criminal family members do return to crime more often than people with a decent job who know no criminals. However, imagine what would happen if this algorithm would be used to determine whom to target with rehabilitation programs, or whom to support more closely upon release - In such a case, the algorithm ceases to be a WMD, since it improves the chances of its targets. Instead of deepening the chasms between rich & poor it would help level the playground by providing help for those who need it most. Any recidivist from the "safe" group? this feedback would return to the system and improve the algorithm.
I got a bit carried away, but I hope that I managed to show why I think this book is important for anyone involved in anything remotely technological: It raises some interesting points on the potential damage of careless or malicious use of big-data algorithms (I skipped the malicious ones, but think targeted marketing) and mentions that sometimes, a perfectly valid algorithm is becoming a WMD only because the way it is used, so take care to ensure your software is being used for good, or at least, does no harm.
The second WMD is a bit more intricate and complicated. The problem is that the software seems to be pretty much benign: it helps a judge assess the level of danger a convict poses in order to determine their punishment, or to assess their chance of recidivism when considering parole. The reasoning behind it is simple: more of a risk a person presents to society, longer should this person be detained, imprisoned or at least carefully watched. That way, minor offenders could get out, leaving the law enforcement mechanism to deal with bigger problems. Chances are, that the algorithm predictions are fairly accurate, too - the company selling it has an interest in keeping it accurate, or seemingly accurate to sell it to the next state and fend off its competition. There are, however, some caveats: First, the algorithm; being the competitive advantage on the competition, is secret. Normally, a judge must explain the motives behind a given verdict, and those reasons can be challenged or limited. No judge today would say "I decided for a stricter punishment since the convict is poor, and therefore is more likely to commit crime again", and yet - the statistical model might do exactly that. There is a correlation between poverty and crime, and between poor neighbourhoods and criminal opportunities, so the model, measured for "correctness" will be more effective to use that. Even if we won't provide the income level of a person, there are a lot of proxy measurements that are highly relevant: Area of residence, whether the convict has a home or a job to go back to, even how many times was this person arrested in the past has some correlation to their financial situation, as wealthy people tend to get arrested less for minor misdemeanors.
On top using discriminatory elements, there's another risk for this WMD: it creates what the author calls "pernicious feedback loop". Meaning, the algorithm results are actually creating the reality it attempts to predict.
Imagine that: Two people are detained for drunk driving. One of them gets a low recidivism score and therefore is released with a warning. The other gets a high score, and so the judge chooses a more deterring punishment and sends him for 6 months in jail. Six months later, when getting out of jail, this person finds that it is more difficult to find a job with a criminal record (and the longer one was sentenced, the harder it becomes), and he got to know some "real" criminals while in jail, so when things will get rough, the route to crime will be ever more tempting. Point for our recidivism algorithm! The one marked as a likely felon was indeed the one who returned to crime. What did you say? it was only because of the first score he was given? Naaa, this is science, and science is always right, isn't it?
So we got an algorithm that discriminates weak populations, and then actually harms their lives and makes it harder for them to make their way in the world. Fun, isn't it?
Unlike the teachers assessment program, the recidivism model can be used for good, since wheher or not we like it, there's no denying that it is possible to correlate life circumstances with chance of recidivism. People without steady income, or with criminal family members do return to crime more often than people with a decent job who know no criminals. However, imagine what would happen if this algorithm would be used to determine whom to target with rehabilitation programs, or whom to support more closely upon release - In such a case, the algorithm ceases to be a WMD, since it improves the chances of its targets. Instead of deepening the chasms between rich & poor it would help level the playground by providing help for those who need it most. Any recidivist from the "safe" group? this feedback would return to the system and improve the algorithm.
I got a bit carried away, but I hope that I managed to show why I think this book is important for anyone involved in anything remotely technological: It raises some interesting points on the potential damage of careless or malicious use of big-data algorithms (I skipped the malicious ones, but think targeted marketing) and mentions that sometimes, a perfectly valid algorithm is becoming a WMD only because the way it is used, so take care to ensure your software is being used for good, or at least, does no harm.
No comments:
Post a Comment