The trouble is that our models reflect our society. For example, when police use "AI" to determine where patrols should go, they rely on the data the police have already collected. If the police spend more time in minority neighborhoods, or make more arrests there, the model will tell them to keep doing what they're doing. The same goes when a model is used to review resumes. If biased decisions were made by the company in the past, the model will continue to make biased decisions, with a false veneer of objectivity. Garbage in, garbage out.There are also base rate problems that come into play. When a company says their facial recognition system is 99.9% accurate with only 0.1% false positive rate that sounds pretty good. But if there are 10,000 people that pass by the city cameras, it would be expected to falsely identify 10 people as criminals. Even if there were 10 suspects in those 10,000 people, half the people targeted would be innocent. This isn't hypothetical - in the UK they tried something like this and found significantly more false positives than actual suspects.

The trouble is that our models reflect our society. For example, when police use "AI" to determine where patrols should go, they rely on the data the police have already collected. If the police spend more time in minority neighborhoods, or make more arrests there, the model will tell them to keep doing what they're doing. The same goes when a model is used to review resumes. If biased decisions were made by the company in the past, the model will continue to make biased decisions, with a false veneer of objectivity. Garbage in, garbage out.There are also base rate problems that come into play. When a company says their facial recognition system is 99.9% accurate with only 0.1% false positive rate that sounds pretty good. But if there are 10,000 people that pass by the city cameras, it would be expected to falsely identify 10 people as criminals. Even if there were 10 suspects in those 10,000 people, half the people targeted would be innocent. This isn't hypothetical - in the UK they tried something like this and found significantly more false positives than actual suspects.

user
yes
assistant