NTT Docomo rolls out zero-touch service operations

Automating remote network maintenance to free up skilled staff
Founded in 1992, Japanese company NTT Docomo has been providing high-quality mobile communication services for 30 years. It is a subsidiary of Fortune Global 500 telecommunications company, NTT Group. NTT Docomo services range from voice calls and data communications to IoT. The company began providing new network services, such as 5G, in March 2020. Since then, the volume of remote maintenance work to fix issues and outages without an engineer physically being on site has been steadily increasing.

Japan’s mobile communications network is one of the most important pieces of infrastructure supporting modern society, and it is crucial that system failures do not lead to long-term or extended outages. This requires an ever-increasing volume of work, and to manage it, NTT Docomo launched zero-touch operations to fix issues remotely without the need for human input.

By automating these processes, the company aimed to speed up recovery times and significantly reduce the number of staff involved in maintenance, while increasing the quality of work carried out and eliminating the risk of human error.

Launching zero-touch operations for remote maintenance of roaming services
As the volume of work increased for networking staff, the demand for high-quality remote assistance grew in tandem.

“Calls, emails, and SMSs aren’t just a means of communication, they’re an integral part of people’s lives and the economy. They highlight the need for high-performing social infrastructure,” explains, Takaaki Okami, Lead of International Service Operations, International Services, Service Operations Department, Network Division at NTT Docomo. “If there’s an outage, we need to get service up and running again quickly. We decided to leverage automation to speed up identifying and resolving issues.”

Automating this process would let the company reallocate remote networking technicians to more complex cases. “Telecommunication services make up a significant part of our business and revenue,” says Yuki Nagaguro, Senior Associate of International Service Technologies in the Services Operations Department, Network Division. “The challenge we’re facing is how to grow other areas of the business. If we can redeploy personnel by automating certain tasks, they can actively contribute to more strategic transformation initiatives.”

In October 2020, the company launched a proof of concept to introduce zero-touch operations for remote maintenance work, starting with international roaming services that let Docomo customers to make phone calls and use data when they’re travelling abroad.

Automating fixes for routine failures
NTT Docomo defined a small set of processes to be automated as part of its zero-touch proof of concept.

Communication services and network failures are either classified as routine failures, which have established recovery procedures and are simple to automate, or non-routine failures. Non-routine failures need to be resolved individually and require staff with specific skills and experience, therefore making them more challenging to automate completely.

“If maintenance involves going through standard procedures, letting the system handle it will be much faster than allocating it to a member of the support team and it eliminates the risk of human error,” says Kiyotatsu Suto, Senior Associate of International Service Technologies in the Services Operations Department, Network Division. “Automating routine failures was the obvious choice.”

Routine issues also amount for approximately 80% of all failures, so automating fixes would significantly reduce the maintenance team’s workload. 

“We decided to start small and focus on international roaming services, using the knowledge we gained to then roll out the solution to domestic services, which is a larger part of the business,” says Okami.

Finding the best SaaS solution with ServiceNow
NTT Docomo decided to use a SaaS solution as the foundation for zero-touch operations. As Nagaguro explains, “SaaS means there’s no need to build new infrastructure for new projects, which reduces the lead time for installation. We’re continuously adding new features to our environment, so we needed a flexible solution and the ability to scale system resources such as CPU and memory easily.”

NTT Docomo went to market and selected ServiceNow IT Service Management. “IT Service Management complies with Information Technology Infrastructure Library (ITIL) best practices and comes with features such as web portals, automated digital workflows, and visualization tools as standard,” comments Nagaguro. “The inter-instance connectivity and the extensive range of APIs mean we can easily connect the platform to other technologies such as our communication tool, Slack, and the UiPath automation system.” 

ServiceNow also supports no-code/low-code development and can be managed in-house with minimal customization.

Redefining scope to stay on budget
During the proof of concept, there were significant challenges to resolve before the solution could be advanced to production. For example, identifying all the tasks to be automated from a list of 300 processes was an enormous amount of work.

“Deciding how to reduce the list of potential tasks to automate was a challenge. If we automated everything, costs would be four times higher than the budget and cause a large deficit,” says Okami. “We held meetings with 40 relevant teams from both inside and outside the company and asked them to consider how urgent tasks were and whether they could be run less often or outsourced to another team or partner.”

This significantly reduced the list of processes from 300 to a more manageable figure, cutting development costs by 82% in the process. With the project back on track and profitable, the team developed zero-touch monitoring and maintenance for international roaming services and went live in April 2022.

Speeding up system recovery by up to 75%
Today, incident tickets can be raised by Docomo users or automatically when the monitoring system detects an issue. This then triggers a workflow to run normality tests to determine where the issue is and to carry out the standard recovery process when it’s identified.

“Previously, users had to call our contact center to notify an agent of a failure. The agent then escalated the ticket to remote maintenance staff who would manually go through system tests and fix the issue to restore service,” said Suto. “The entire process, which was like a ‘bucket relay’, has now been automated with ServiceNow.”

This has reduced the time between receiving a failure notification to service recovery by up to 75%, demonstrating that using automated workflows for routine issues is much faster than routing them to a member of staff. The company has freed up 30% more time for the remote maintenance team, who can now focus on more valuable strategic activities. 

With plans already in the pipeline to expand zero-touch monitoring to domestic services, NTT Docomo is leading the way for the wider NTT Group. It has agreed to share its findings with 14 other NTT organizations and ultimately hopes to roll the solution out across the whole group.

Download PDF