- AI algorithms are susceptible to the biases of their human creators
- Error rate isn’t a good method for checking whether an algorithm produces biased results
- Shadow algorithms can test other algorithms, helping to control for different biases
For all of the hype, in practice, artificial intelligence today is still a very human pursuit.
Machine‑learning algorithms, the basic building blocks of AI, are sets of rules coded by human programmers to generate predictions. They’re “trained” using data sets that are handpicked by people. They learn how to make better predictions by analyzing human feedback on the accuracy of their predictions.
It’s no surprise, then, that the data scientists creating AI algorithms today unwittingly introduce occasional biases and mistaken assumptions into their formulas. Sometimes these biases don’t have much of an impact on machine‑made decisions. But other times, they skew the results of decisions that impact lives: credit scores, loan approvals, job applications, college admissions, even policing and criminal sentencing.
As AI expands its influence in business, many companies face the thorny issue of how to prevent AI‑powered discrimination in their products, services, and hiring practices—not just to reduce legal exposure on sensitive issues such as race or gender, but to make truly objective decisions no matter what AI application is in play.
The nascent field of AI bias prevention, or algorithmic auditing, is complex. It wrestles with questions of objectivity of data but has roots in ethics and political philosophy, says Nicolas Economou, chairman and CEO of legal technology company H5.
“Can we trust software engineers to make decisions that relate to domains they have little expertise in, and that emanate from fundamental issues of public policy?” Economou says. “Entrusting developers with the task of de‑biasing AI is a technological version of Juvenal’s conundrum: ‘Who will guard the guards themselves?’”
While the practice of algorithmic auditing is new, some companies are already exploring how to take the process out of human hands—by training so‑called “shadow algorithms” to detect and fix their own hidden biases. Microsoft, Facebook, and IBM, among others, are all working on different algorithmic solutions to AI bias.
“For a long time, it was good enough to judge the quality of machine learning applications by overall error rates,” says Briana Brownell, CEO of AI‑powered analytics firm Pure Strategy. “Now we know that not fully understanding the ways in which a model makes prediction errors can result in all kinds of unintended consequences.” Dealing with bias is becoming a necessity for data scientists, she adds.
Putting data to the bias test
Despite ethical and philosophical questions, training algorithms to detect and eliminate bias is possible, says Jason Odden, director at tech consulting firm Cask. One strategy is to deploy the same machine‑learning tools used to code algorithms to find biased “blind spots” buried in data.
“Reverse engineering of these techniques can expose bias within the dataset itself,” Odden explains.
In 2015, for example, reports revealed that searches for “gorilla” on Google’s Photo app returned photos of African‑Americans in their results. Through reverse engineering, programmers can create an algorithm to detect and scrub the race‑related data tags and prevent discriminatory data from infecting future outcomes.
But what if the bias is more subtle? In a classic 2003 study, researchers found that job candidates with the Caucasian‑sounding names “Emily” and “Greg” were 50 percent more likely to land an interview than those with the stereotypically African‑American names “Lakisha” and “Jamal.” Data scientists can easily reverse‑engineer an algorithm to prevent AI applications from unfairly downgrading candidates based solely on names.
More difficult, however, is mitigating bias in a hiring algorithm trained to favor candidates with profiles similar to “successful” workers already at the company, such as those with longer tenures and a track record of promotions. Those are logical criteria for hiring—unless the company tends to value and promote men over women, or white workers over ethnically diverse ones.
Other approaches can be used to train algorithms to detect bias, Odden says. One technique involves feeding so‑called adversarial data samples—inputs intended to produce a mistake—into an algorithm to uncover data distortion, a method commonly used in spam filtering. A potential adversary may try to pollute an AI system with bad data. By peremptorily factoring similar data into the machine‑learning system, an AI user can eliminate biases.
The “black box” conundrum
AI systems are designed to be self‑reinforcing as they “learn,” so even the most subtle types of bias can become more pervasive and entrenched in just a brief amount of time.
The most complex sets of algorithms used in AI applications—so‑called “black box” systems—present the biggest problems in managing bias.
Insurance companies, for example, rely heavily on “black box” algorithms to set auto insurance premiums using data not just from a customer’s driving record, but from a host of other sources including credit scores and zip codes. The formulas are opaque and highly proprietary, and can often result in discriminatory outcomes.
A 2015 Consumer Reports study of two million rate quotes from 700 insurance companies found that drivers with pristine driving records but spotty credit history paid substantially higher premiums than terrible drivers with higher credit scores. Another recent study reported that drivers from mostly African‑American neighborhoods were quoted rates that were, on average, 70 percent higher than those of drivers from predominantly white areas.
Could programmers create an algorithm sophisticated enough to shine a light on bias in “black box” algorithms? Eventually, yes. For now, the most pressing challenge is a human one: establishing basic standards that can help mitigate bias before the AI watchdogs enter the picture.