Why these risk management strategies should be your focus in 2022

But some are raising red flags. Greene says that because AI relies on human-inputted data to make decisions, there’s the potential for bias to infect the algorithms. Removing humans from decision-making doesn’t ensure less descrimination or bias. When humans are unaware of how their biases impact their decision-making, researchers like Cappelli and Greene say they are concerned that AI tools will simply mirror those flaws.

Greene says there isn’t a lot of science underpinning hiring platforms like HireVue, for example, which records an interview and uses AI to evaluate a candidate’s voice or expression to score them on attributes like friendliness and assertiveness. “This, unfortunately, has led to the resurrection of physiognomy,” says Greene, referring to the pseudoscientific and often racist practice of assessing someone based on physical characteristics such as their facial features.

Other AI-powered recruiting tools rely on past hiring data to identify optimal candidates, so if companies have mostly hired men, says Greene, the algorithm will continue to choose men for available jobs. “Bias in past hiring might cause bias in current hiring,” he says. An AI-powered recruiting tool for Amazon, for instance, made headlines in 2008 for teaching itself to systematically favor male job candidates over female candidates based on past hiring practices.

Governments are stepping in to address such transparency and fairness concerns. Over the past few years, Illinois and Washington passed laws forcing employers to inform potential hires if AI is used in the decision-making process, and to create transparency whenever AI is marketed to consumers. New York City passed a law last year that will force local employers to reveal the use of AI in hiring and to make sure AI systems undergo annual bias auditing when it goes into effect in 2023.

Some researchers think government oversight doesn’t go far enough. In a 2019 paper co-written by Wharton’s Cappelli, the authors argue that organizations building or buying automated HR tools should test any system they plan to buy for bias, invite employee input on uses of these tools, and create an appeals process that would force a human to enter the decision-making process.