Introduction
Over the last three decades working with hundreds of customers, I’ve yet to find an organisation who isn’t committed to improving their Service Assurance strategy and delivering excellent service to their customers. Despite extensive investment in tools to help build, deploy and operate their applications, they still face significant challenges in understanding the what, where and why of service delivery issues. Understanding the ICT delivery chain (aka service map) for application services provides invaluable insight and helps expedite triage and service restoration. This is a core part of modernising Service Assurance processes.
ServiceNow’s Service Mapping product is a very innovative and powerful technology. As a techie who's worked in the ITOM field for a considerable time, it's certainly the most automated, comprehensive and configurable solution that I've seen. Through analysis of the actual config data driving your applications, it automates mapping of application service chains, maintains the CMDB and seamlessly integrates with ITSM and ITOM processes. However, when considering a technology like Service Mapping, IT organisations are sometimes concerned about its ability to deliver. Over the past decades, many vendors have tried to deliver on the “automated mapping” challenge – legacy-type solutions often under-delivering or requiring excessive manual intervention to derive value.
In this blog article I want to talk about some of the realities of deploying ServiceNow's Service Mapping solution (or SM as I’ll call it) and some of the new capabilities recently released to make it even easier to use. I encourage anyone who is interested in the challenge of diagnosing service delivery issues to take the time to investigate SM further. If you get the opportunity, attend one of ServiceNow’s SM hands-on experience events – this is a great opportunity to play with SM for yourself and understand how easy it is to use and maintain.
And don’t forget, we’re talking about solving one of IT’s most enduring and difficult problems. Unpicking the multiple layers of an organisation’s ICT spaghetti to understand which components deliver which services is a truly challenging task. Decades of growth, the constant push to implement the next “next-gen platform” and the many custom-built apps and Band-Aids make every organisation’s ICT landscape unique. At the end of the day, it’s all about deriving valuable insight into your business operations from a trusted source of truth (i.e. a CMDB). That insight can significantly reduce operational risk (e.g. of Change) and provide valuable guidance to help your business improve application service delivery. Given that most organisations are in the software delivery business today and excellent service quality is the expected norm from consumers, it’s a challenge which must be solved.
How many patterns are enough patterns?
Fundamentally, SM relies on predefined “patterns” to understand how each component in your application’s service delivery chain operates, connects to and depends on other components. Patterns execute as scripts, but you rarely ever need to (or would want to) see that side of them. You build patterns as a series of logical business rules to help SM identify the relevant ICT asset, locate its configuration data and understand what part of that data tells us something useful (like what other ICT the app talks to).
When I talk to architects, engineers or dev folks, they sometimes ask “How many patterns does SN ship with”? For the record, today there are ~400 (discovery and mapping), with new ones being regularly released via the SN store. But I don’t think that’s the most important question to be asked. What should be asked is “How much work is required to extend mapping to recognise my custom apps”? It is a given that every ICT organisation will operate some of their own custom-built applications. Whether locally built, contracted in, or purchased to satisfy a specific business need, these are all special-case applications, for which it’s very unlikely any vendor will supply an out-of-the-box "pattern" for.
Ultimately, what’s most important is understanding how much time, effort and skills are needed to build a custom pattern in SM – or extend one that is shipped out-of-the-box but needs to be tailored to cater for the peculiarities of your environment. ServiceNow's ITOM product team has focused on making this process as fast, easy and code-free as possible. That's what makes SN different from other tools you may be familiar with.
Who should manage patterns?
A pattern is linked to a specific device type or piece of software. Whether a load balancer, web server, app server, database or custom process supporting your application, an SM pattern needs to recognise, interrogate and analyse that hardware, software or cloud resource. When it knows how to successfully recognise and categorise one instance of the ICT asset, it will be able to recognise it wherever it is found across your enterprise. So, whilst there may be more up-front effort required to tailor some patterns, over time the value of SM increases whilst maintenance effort decreases.
Given the wide variety of software running across an average enterprise, it’s difficult to find one "super-engineer" who understands how each of your hardware and software products is configured and operates. Setting up a "Centre of Excellence" for handling all required SM pattern extensions isn’t usually the best approach. I recommend looking to your existing ICT support engineers who already know their hardware and software products inside-out. They’re best placed to know what changes need to be made to patterns to map their infrastructure. So, send them on the SM training class and include them in the project. They’ll be prepared to enhance patterns if required and can ensure a rapid roll-out of SM to deliver value to the business.
Writing and debugging patterns
It’s not that hard to jump straight into the SM pattern editor and start writing patterns (it's right there in the platform UI). However, you’ll be much more productive and find it much easier to design and test your pattern by using the interactive "pattern debugger".
The key here is the word “interactive”. This means that whilst designing each step in your pattern logic, you’re doing this via a real-time connection to the target device. You can run commands against the device and see the results, pull live config data and test each step to ensure it does exactly what you want. There is no offline coding, compilation or testing needed. Getting immediate results are a much more rewarding and expeditious approach to building patterns.
Following is a walk-through of a simple example from a hands-on lab class SN runs. This helps to demonstrate how patterns can be easily extended to cater for your custom applications. Here's the back-story:
- Mapping starts from the application’s entry-point - in this case a URL. After having discovered and chained together all the components and dependencies in the application, the service map appears to be complete. However, when reviewed by the application owner, it is rejected because an important security component of the application was missing. Refer to Tony Branton’s earlier blog article for further details on the in-built map certification process for collaboration.
- The application code which is running in Tomcat was correctly mapped by the supplied Tomcat patterns, but mapping didn’t know that the app has a dependency on an in-house built security module. The Tomcat app refers to a config file (call_secsvcs.txt) to tell it where the custom security service is hosted and how to call it.
- The purpose of the hands-on lab is to build the additional pattern logic to map the connection from the Tomcat app to the custom (node.js) security module and complete the service map. This is the task which most customers need to perform to ensure their custom software components are correctly mapped.
Following are the basic steps to extend the pattern and make this happen:
- Edit the Tomcat WAR pattern (supplied by ServiceNow) and add a new "connection section". This is where we will be adding our logic – but to start it’s just an empty placeholder. When the app is remapped, the new pattern will be executed. Whilst it won't do anything useful yet, it will be available for use in the real-time debugger!
- After remapping the application service, a debugging session can now be initiated against the new pattern. This is where we’ll write our logic to build a connection to the security module.
- Now we can write each logic step and test it in real-time. Via the debugger, you can see the data (e.g. config files) pulled directly from the application server and test the results (e.g. see new dependencies) which would be pushed into the CMDB. This real-time construction and debugging capability shouldn’t be underestimated. It dramatically shortens the dev cycle and makes it easy for prototyping. The following screenshot shows us opening the app’s config file and parsing out info about how it calls the security service. You can see the data, parse it and then move on to the next step.
- The second (and final) step builds the dependency records in the CMDB – effectively completing the service dependency map. It uses the connectivity information dynamically extracted in the previous step. In the lab use case, this is to ensure the security service is shown as a dependency from the Tomcat application component.
- Now when you rerun mapping, the new pattern will execute, and the logic will update the map. Those 2 steps in a pattern extension have incorporated the custom-built security service into the application service map.
Making sense of generic applications
As you can see from the updated service map above, the connection to the security service has been correctly built – it’s the new component called “node.exe”. It’s good that it’s now there, but the component shown on the map isn’t very informative. The person who wrote the pattern logic and the SME looking after the business application may well know what "node.exe" represents, but it’s not informative enough for a service desk, NOC operator or service delivery manager. During triage, the service map, overlayed with monitoring events is the primary source of insight into the root cause of a service delivery incident. To expedite the triage process, it's important that the service map clearly indicates where the fault lies so that the correct engineers can be assigned for remediation.
What we really want is for Discovery and Service Mapping to understand what that piece of software is doing for the application - i.e. it's the "security service". We’ll likely also need to know other information about it – like its version, deployment date, config files, etc. If we want to fully and correctly manage this software as a CI in the CMDB, we need to do the following tasks:
- Create a CMDB class to record instances of this software. This is usually extended from the Application class.
- Create a CMDB identification rule to ensure we identify and classify the software's executing process correctly and uniquely.
- Build a process recognition rule to understand “node.exe” and run the correct discovery/mapping pattern.
- Write a pattern to extract useful data about the component and tailor the CI record as required (e.g. set a meaningful name).
- Create relationship links to the Host it runs on.
- Create the CMDB self-health monitoring rules to govern the data’s quality.
In earlier SM releases, these steps would need to be done manually and whilst not necessarily complicated, would take some time and require knowledge of the CMDB. Since the Kingston release, a new wizard has been included to automate all these steps, making it even faster and easier to use. This is a very useful capability – making the process of rolling out patterns and recognising your software so much easier and faster.
Here’s how you use the wizard:
From the service map, right click on the generic application component which has been added by your pattern and select the option “Create pattern from generic application”. Define the component’s name and choose an icon. That’s it! It will do all the steps described above - preparing the CMDB and the pattern. The service will also be automatically remapped for you. You’re done!
This is the updated and now complete service map of our application service. The whole mapping process is dynamic, meaning that every time it is remapped (e.g. after a change window), any changes to the application structure or dependencies will be automatically detected and the service map adjusted accordingly.
Porting patterns between Instances
Now that the engineer has finished tweaking the pattern, it’s been tested, and the app owner has verified it, we’re ready to deploy it into production. Before London this wasn’t the easiest to do – various techniques existed including manually copying the NDL (generated script) between instances.
In London, this process is now made much simpler and patterns can be exported in XML format. From the pattern table, select the patterns of interest and invoke the export action. The export is effectively an update set although the standard update set process can’t be directly used. It isn’t capable of extracting all the required components into the update set. Please check the doc on this function as there are a couple of options to control how many related CMDB objects are included in the export.
Final thoughts
ServiceNow continues to invest heavily in the CMDB and ITOM products to automate discovery and mapping of your critical application services. Having moved to a monthly release cycle for new patterns (available through the ServiceNow store) means we can release new support as soon as possible. Some of the recent additions include AWS/Azure gateway, Lambda and Kubernetes.
Maintaining accurate and up-to-date service maps brings valuable insight into your application and business operations. This insight positively impacts the daily lives of many staff across the organisation who rely on core operational processes like Change, Incident, Security Ops, Problem and Event Management. Ultimately this helps to modernise your Service Assurance processes - enabling IT to provide an improved level of service delivery to the business.
I hope this blog article has helped to demystify Service Mapping and show that the value it brings is well worth the investment in teaching it about your custom applications. Please reach out to your local ServiceNow ITOM team if you’d like further information or to get some hands-on experience with the ITOM technologies.
