Trust is the first and perhaps most important driver of interest in XAI.
Artificial intelligence is changing how companies do business. But it still has trouble showing its work.
Nine out of 10 businesses using AI today report a critical need for better explanations about what’s happening inside AI’s “black box,” according to a 2021 IBM report. More than three-quarters of IT professionals in the survey say it’s critical they can trust that their AI’s output is fair, safe, and reliable.
Designers of AI systems know what data goes in and the answers that come out. Too often, what happens in between remains a mystery. Hidden biases in the data can deliver results that are inaccurate, unethical, and, in some cases, illegal. Faced with increased regulatory pressure—and growing demand for transparency from customers, employees, and investors—organizations need to explain the reasoning of their AI systems to show they are delivering accurate results and operating within ethical boundaries.
“These systems are much easier to troubleshoot when you understand the reasoning process,” Rudin says.
Aperio Consulting Group, a company that explores the intersection between behavioral science and AI, developed an AI app that analyzes psychological factors in entrepreneurs. The target audience: Investors who want to know how to guide startup founders toward success. Explainability was designed into the tool from the start, says Bryce Murray, Aperio’s director of technology and data science.
For example, a black-box AI tool might give a single score of an entrepreneur’s potential. Aperio’s explainable system provides a more detailed explanation of the factors that influence the score—such as a low level of perseverance—making it possible to outline more targeted coaching guidance to help the entrepreneur “tune up” that quality.
“We’re working with people data, so if we were using unexplainable AI, it wouldn’t work,” Murray says. “We have to explain beyond just throwing lots of statistics and numbers out there.”
Despite the recent progress, explainability is still part of a “continuous improvement journey,” says Grace Abuhamad, who heads up trustworthy AI research at ServiceNow.
“Explainability is still an area of active research,” she says. “There are a lot of companies that are trying to do work in that area, and a lot of startups are launching themselves as experts in this space, but overall it’s not a solved question.”