Get Started
When the technology that’s all around just works, the world works.

Q&A | APRIL 22, 2022

Most machine learning projects needlessly fail

A conversation with author and machine learning expert Eric Siegel

Nearly 9 in 10 enterprises increased annual spending on machine learning and AI projects in 2021, according to a DataRobot report. But their potential ROI is far lower than it could be, says Eric Siegel, a machine learning expert and author of “Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die.”

Most projects stall out before deploying machine learning models, according to Siegel’s January 2022 survey of data scientists.

In a recent interview with Workflow, Siegel explained why leadership, not technology, is to blame for the high failure rate, and what CIOs and other senior executives can do to reverse the trend.

Because they’re typically viewed as technology projects when they should be seen as organizational ones. The point is to improve a company’s operations, not to flex high-tech muscles.

There’s a fetishization that happens with machine learning, and that applies to even the most hardcore technical experts. They’re trained to download the software and start making models without stopping to think, “Wait, this is a business endeavor; what is this project really supposed to achieve?” They may consult with their business counterparts, but not nearly to the depth that is required.

These should be treated as projects meant to improve operational performance, where machine learning is but one key component. Ultimately, it’s up to executives to identify the business goals and work with the data scientists on the best ways to achieve them.

<20%

Percentage of machine learning models that a majority of data scientists say are ever deployed.

More semi-technical understanding may be necessary, but it’s actually a myth that machine learning is hard to understand. It’s just a probability calculator: “Which customer is more likely to do X if you do Y?”

It’s like driving a car. You don’t need to know how an internal combustion engine works, but you need to know how to operate a steering wheel, the concepts of momentum and friction, and have a destination in mind.

You need to start by defining your business objectives. But you also need to be very specific about what your ML models are designed to predict.

It’s not enough to ask which customers are likely to cancel. Instead, you should ask which customers who’ve been around for three years will decrease their spend by a specific amount over the next four months. The business objective can be general, but the prediction goal and deployment particulars have to be much more specific. And you need to agree on the metrics you’ll use to measure success.

Once you’ve gotten those predictions, what are you going to do differently as a result? What operational change will you embrace that will make an impact? For example, if you’ve got a team of auditors that have been doing fraud detection the same way for 20 years, how are you going to convince them to do things a new way? They may not be aware of how data-driven probabilities work.

It’s not horribly technical, but they may be set in their ways.

One big reason why ML projects don’t get deployed is because the line of business managers are used to doing things a certain way and don’t really want to change.

That depends on the organization. Some projects are led bottom-up, some are top-down, some a combination of the two. Change happens in all sorts of funny ways. The existence of a CIO, CTO, chief analytics officer, or chief data officer certainly speaks to the idea of someone leading this process, but I can’t say there are that many people in those roles who are really getting in the trenches and making sure models are delivering the kind of impact they’re looking for.

For each initiative, you’ve got to clear a path from the start that will lead to the integration of a predictive model. This requires a socialization of buy-in: Line of business leaders and managers must agree to make a real change to operations. They must learn what a predictive model does for them and must be willing to put their faith in it.

Then, even after this informed and detailed buy-in, business-side leaders and line-of-business managers must continue to remain involved with each stage of the ML project, including preparing the data, generating the model, deploying it, and measuring its performance.

They’re typically viewed as technology projects when they should be seen as organizational ones.

Financial services has a long track record of successfully deploying this technology with credit scoring and fraud detection. They’ve carefully defined their goals and changed their internal processes. An ML project may simply be an incremental update to things they’re already doing.

In logistics, UPS uses ML to predict where packages will need to be delivered tomorrow so they can start loading trucks in an optimal way the night before. Big tech companies like Google and Facebook had language translation models for years, but when they introduced deep learning algorithms, the translation quality improved overnight. They could do this because they already had much of the technology in place. It wasn’t a huge change to operations, as it would be for companies whose ML practices are less mature.

Where you’ll find the greatest potential to improve efficiency is in large-scale, legacy enterprise systems that have only just started to experiment with ML. That’s where change management needs to play a huge role in making sure these projects ultimately get deployed.

Trending articles

Want to Get Ahead and Stay There? Rethink IT Service Delivery.

Subscribe

Related articles

Upgrading the production line
RESEARCH
Upgrading the production line

Optimized manufacturing pairs highly skilled workers with advanced technologies

How AI can drive business results
CRASH COURSE
How AI can drive business results

Strategies for extracting more value from machine learning models and applications

Supply chains get a digital makeover
CRASH COURSE
Supply chains get a digital makeover

Digital workflows can help ease what ails global logistics

From the lab to the platform
Q&A
From the lab to the platform

Valérie Bécaert relies on her scientific background to translate AI research breakthroughs into enterprise software products