As unexpected change rolls in keep technical debt from freezing you in place

[Michael Hansen]

By Michael Hansen and Lisa Wolfe

Airlines, social media providers, software giants, and critical public sector institutions continue to make news headlines almost daily with unplanned outages. Often the root cause is simply being unable to move fast enough. As unexpected waves of change come crashing down, organizations must pivot quickly. Think back on the pandemic to companies that were able to move their workforce offsite overnight and keep on running the business as normal versus others that were just paralyzed, not prepared to change business as usual. In today's crises, media pundits and even decision-makers in the affected organizations are loudly calling out technical debt as the key issue that will determine if a company can move quickly to safety or must sit frozen like a bronze statue.

In one of the recent cases, a well-respected airline had a holiday operations meltdown.  Major news outlets US quickly attributed the problems to the company's outdated scheduling software system. The cascading failures which began with a relatively limited weather outage eventually canceled over 15,000 flights.  The technical debt issues were an open secret in the industry with the flight attendants union even writing a public letter that placed updating the airline’s software above its members' demands for increased pay.  Now airline executives find themselves under intense public scrutiny of their own compensation.

If managing technical debt was easy or straightforward, corporate leadership would have likely tackled it straight on, long before it reached this point.  Afflicted companies do not spend significantly less for technology than other carriers, they just get ever-decreasing returns on the dollars they spend.   Technical debt is not easy to quantify in a simple dashboard or action plan.  Before any organization can effectively address technical debt, you must have good answers to the following questions.

First,  do our company’s technology components map to a clearly understood service architecture?

No one wants to see journalists or armchair CIOs publicly second-guessing their priorities and investments in the newspapers after a major operations crisis, supply chain failure, or security breach. Prior to the airline meltdown, the company had many indications that their scheduling software was becoming inflexible in the form of smaller problems and inconveniences.  It is vital to recognize that every electronic purchase transaction, every purchase order, and every customer logon is a test of your technical debt.   Small failures cannot be grouped together, their severity understood, nor root causes diagnosed unless and until they are mapped to the overarching services your organization provides.  This skill can be difficult in organizations that naturally drift to technology silos.

In the case of the airline, the problem was not a database problem, a Linux administration problem, nor a help desk problem. It was not even a software development problem strictly speaking.    It was a “crew scheduling” problem that could not be properly understood nor addressed until it was seen holistically in the frame of the business service the airline's technology systems were providing.

If you cannot map the bit parts of your technology to the vital business services they are providing, you are effectively flying blind.  Fortunately, it has never been easier to automate the Discovery of technology, even complicated technology like cloud containers, microservices, and APIs, using automated service mapping and observability solutions.  It has never been easier to group the outputs of different IT systems dynamically and in near real-time using connectors that graph each data element to the service it provides.

Second,  do we manage our company’s daily technology operations with awareness of the services they are providing?

Consider the following service operations scenario.  A cloud application server housed on a legacy mainframe is taking too long to respond, causing sporadic outages of an important customer-facing application.  All of the functions performed by that mainframe are now available via next-generation cloud servers or microservices.  But support staff are uncomfortable with these new configurations since the application has always been hosted on-site.  How can a tiger team tasked with eliminating this technical debt best demonstrate that the new configuration will work, that the cloud instances are stable and reliable, and that the application will respond adequately in every geographic region? The answer likely depends on what the mainframe was doing for the application, how the application is structured, and the impact of an outage or slow response on the business.  One of the real advantages of having more robust mappings and observability of your technology is that these architectural assets already provide a considerable amount of the context needed to make the right decisions.  Context comes in the form of detailed records of each node in the current and proposed system and visual relationships that show how all the components fit together.

Day-to-day operations also benefit immensely.  Not only do human beings make much better decisions when we have the full context, but Predictive AIOps can also mine this contextualized data in the form and metrics, events, and health logs to predict issues and their likely severity so operators can address potential problems before they occur.  The risk of a change to hosting technology is far easier to assess when teams understand how the application interfaces with other systems and what external dependencies may exist.   Each scheduled change is therefore less likely to produce an outage and easier to trace to the offending component should an unexpected outage occur. 

Third,  are we able to group systems with similar functions to realize efficiencies and plan for manageable levels of future technical debt?

Technical debt is often the flip side of technical bloat.  The reason companies struggle to replace overcomplicated systems is that they do not have confidence that they can effectively match the size and capacity of a new solution to an existing system that is mysteriously configured and poorly documented.  Planning gets even more complicated (and the benefits of technical debt management get even larger) when companies have employed more than one system or technology to address adjacent or even identical business requirements. 

Application portfolio management is a long-standing and well-established discipline that allows technology leaders to group systems providing similar functions together into transparent and logically similar blocks so each software function—like flight and crew scheduling--can be evaluated together based on cost and function.  Security systems, workflow software, and/or ERP software are other common categories.

When equipped with well-informed application portfolios, technology leaders can consolidate duplicate systems or systems laden with technical debt into more responsive and robust solutions. Most importantly they can ensure that their tech dollars are spent to prioritize projects and applications that will keep them out of the newspapers, flying high above their competition.

These are not the be-all and end-all to resolve technical debt – technical debt is a hard problem for any organization-but staying ahead of it and managing it is vital to preventing unplanned outages and being ready for the next unexpected change.

Learn more: 

Predictive AIOps

IT Operations Management

Application Portfolio Management

Strategic Portfolio Management

Service Operations