Balanced metrics: The key to success in DevSecOps transformation

When measuring the success of large-scale transformations—particularly in the technology space—it’s natural to look at hard metrics, such as cycle time, mean time to recovery (MTTR), and so on. In IT, for example, hard metrics are what we do all day long.

But within any organization, change is ultimately personal. In my experience, relying exclusively on hard numbers often leads you to ignore the human side of transformation, and sometimes even action the wrong things. That’s why we use both hard and soft metrics to guide our DevSecOps transformation at ServiceNow.

Measuring progress in 2 ways

In our transformation journey, we look at more than 20 data points, with equal weight given to soft numbers and hard numbers:

• Hard metrics are quantifiable, such as the number of releases flowing through the pipeline, the frequency of deployment, and the number of teams using the pipelines.

• Soft metrics are qualitative, subjective values, including Net Promoter Score (NPS) and Adoption Readiness surveys of our program leads, that assesses the engineering community’s satisfaction with the transformation. We often follow these surveys with retrospectives where engineers can share their thoughts on changes from a people (culture), process, and technology perspective.

To help us transform insight into action, we use a combination of both metric types to consider the human side of a project via satisfaction and feedback surveys. Including soft metrics brings team members into the transformation process, giving them the chance to voice their views and influence change.

Teams that show great self-reported metrics, such as satisfaction scores, and hard metrics, such as deployment frequency, are the ones that make steady progress and find new ways to improve week over week.

On the flip side, teams that rely only on the rigor of hard metrics and overlook user satisfaction often hit a plateau quite quickly. This results in frustration—pushing hard but not reaping rewards.

I must admit that although I love to see high satisfaction scores, I welcome the low ones because they yield a goldmine of feedback that will help us improve how we work.

Gaining insights from divergence

Hard and soft metrics often track with one another; good progress against hard metrics often results in high satisfaction. Where it gets interesting is when the two metrics diverge.

For example, one of our teams’ high satisfaction scores showed they were very happy with their new process. Yet, they were used to their previous release schedule and saw no great need to increase their deployment frequency. By considering both metrics, I knew we needed to create encouraging conditions for them to deploy more frequently and experience the benefits themselves.

At first blush, it may seem like low satisfaction scores are purely a symptom of cultural challenges, but a deeper look can reveal whether it’s the process, technology, or culture that needs fine-tuning.

As an example, one of our teams had low satisfaction scores—not because they resisted change, but because their unique setup wasn’t supported by our technology. They listed their complaints, and the DevSecOps tooling team enabled features to accommodate them. As a result, the releases going through the pipeline and satisfaction increased not only for this team, but also for other teams that had this problem but not as severely. (Read more about using metrics to gain visibility and achieve desired outcomes.)

Transformation is a journey

Transformation is an ongoing process, from crawl to walk to run to sprint. Because of that, attitudes and metrics are going to change. Be sure to take the pulse of your community periodically—on both hard and soft metrics—to assess your progress and make any necessary course corrections to achieve your goals.

Find out more about how ServiceNow uses its own products.

© 2021 ServiceNow, Inc. All rights reserved. ServiceNow, the ServiceNow logo, Now, and other ServiceNow marks are trademarks and/or registered trademarks of ServiceNow, Inc. in the United States and/or other countries. Other company names, product names, and logos may be trademarks of the respective companies with which they are associated.