Think twice before integrating your EA platform into ServiceNow

[Mark Bodman]

Many ServiceNow customers try to integrate their Enterprise Architecture tools with ServiceNow. However most find that the complexity of such an integration increases dramatically as EA practices mature, and the use of ServiceNow expands. Here is how I came to realize that it may be an unachievable long-term, based on my own experiences over the past 20 years. I suggest that you should instead take a native, on platform approach.

My backstory

In 2005, I became an Enterprise Architect at Dell, having spent my first 5 years there leading the design and implementation of several large IT systems used to provide loans and leases to Dell customers. One of my first tasks as an EA was to implement tooling to perform our technology standards governance processes and app portfolio management activities. Our EA team was rather small at the time. After about 2 years of research and vetting, we settled on purchasing our EA tool from Troux. 

After about 2 years of implementing Troux to tackle our technology standards processes and application portfolio management, we started to integrate our EA tool with our CMDB.  The objective was to obtain the operational data so we could better manage both planning and operational activities.  This effort proved very fruitful in the beginning. We pushed our EA data such as Business Applications into our CMDB and brought operational data into our EA tooling. 

After another 2 years, cracks began to emerge in how sustainable this was as our data and workflow needs grew in scope and complexity.  It was at about that time I left Dell to join Troux, to begin my journey as Product Manager.

While at Troux, I added lots of great features, but the power of integrating the Troux EA platform with CMDBs was a common need many customers asked about.  To address this, Troux partnered with HP software and soon we were working together on a joint effort to create a fully supported, robust integration between the HP uCMDB and the Troux EA platform. This went on for several years, with successes and customers achieving their desired results. Analyst firms even published papers on this topic, predicting that most EA platforms and CMDBs would be integrated in just a few years. This looked promising; however, the vision didn’t come to pass as predicted. 

After about 4 years with Troux, I left to join HP Software as an EA in our pre-sales team. While at HP I learned about the full spectrum of tools that HP had acquired over many years and endeavored to explain how they all worked together to our customers.  I helped in creating a framework to communicate this, and we initially called the effort ERP-4-IT.  We eventually donated the IP to The Open Group, which then became the IT4IT standard.

Near the end of my time at HP, now re-named to HPE, I realize that one main customer challenge centered around integrating all IT tools following the IT4IT standard. While the intent of IT4IT was created to explain this, having our customers purchase and use all HPE products acquired over the years so that they work seamlessly together was another matter altogether. After 4 years with HPE, I was then contacted by ServiceNow.

Enter the ServiceNow platform difference

There were 3 attempts to contact me about a new product manager role at ServiceNow. I didn’t answer those first 2 attempts, thinking that ServiceNow just didn’t make sense for me in my current career. I wasn’t a service management specific talent, I was more an EA, and heavily involved with the IT4IT forum by then. ServiceNow was just a Service Management company, right?  So, I finally decided to respond on that 3^rd attempt to contact me given their persistence, mainly out of curiosity. Why was ServiceNow was so persistent in wanting to talk to me? In that initial conversation I found out that ServiceNow was at its core, so much more than a ticketing tool. I was intrigued.

During my interview, I learned that ServiceNow was not created for Service Management specifically.  Instead, our founder Fred Luddy’s dream was to create a SaaS platform that allows any knowledge worker to define and automate their work, without needing development skills. The first set of workflows that were created to get the ball rolling happened to be for IT Service Management. Our explosive success on that first use case led to ServiceNow being known only as an ITSM company, instead of the powerful platform core that we really are. 

The product ServiceNow asked me to manage was Application Portfolio Management (APM). I had the perfect experience for that.  This is a product I knew, and I knew well. But I knew the EA practices, tooling, and market well too.  Managing another APM tool was rather easy, but this isn’t what got me excited about the prospect of working at ServiceNow.  It was the opportunity to evolve APM into a more comprehensive, integrated EA function. To have seamless EA capabilities on the same platform with all the other powerful IT management solutions working together was compelling. The prospect of having all that operational data at my fingertips, enabling a real-time planning with the rest of the processes without the hassles of integration was exhilarating. I told my hiring manager: APM is just the start, we can do so much more here than we realize.

Within six months, I learned a hard reality. Even though we had many products like APM on our platform, there was no guarantee of consistent data structure that my product could take advantage of to fully connect to the spectrum of products we provide. Additionally, most APM tables were not available out-of-the-box, meaning customers were storing APM related data in other places.  They then had to migrate their unique tables into our APM structure after purchase. Although each product could benefit from common data, there was little coordination or guidance that described how to achieve this. After collaborating with my ITOM and ITSM counterparts, we decided to tackle this. CSDM (Common Service Data Model) was born.

Getting our first edition of CSDM published was a difficult feat. We needed a hybrid data model that consisted of EA, operations, and service management concepts all rolled into one. We were on to something which was unique in the market, and it struck a chord with many customers. Given the raving success of CSDM, I switched focus to primarily product manage CSDM and CMDB.  We could now prove that we were effective in bringing different IT management practices data into one succinct meta-model. CSDM contains concepts core to EA, Service Management, and Operations, and continues to evolve as we expand our products and shared data elements year over year.

A Pattern of EA and ServiceNow integration attempts

As I reflect on what started me on this journey within IT, integrating EA platforms with a CMDB’s, I realize that has been a consistent pattern of challenges for most organizations attempting to do this.  Below is a summary of the types of challenges that I saw repeatedly that you should watch out for. These are challenges ServiceNow is now poised to resolve elegantly, thanks to CSDM.

There are 4 key lessons that emerged as I reflect on the many organizations have seen attempt to integrate EA tools with their CMDBs.

Before I summarize the main 4 issues that most will encounter, I found that that organizations go through a 3 specific implementation phases before these issues become glaring apparent, and likely insurmountable by the 3^rd. The big takeaway is that a successful integration is likely to become a multiyear effort with a distinct scope and activities.

I have also noticed that each phase takes about two years to compete on average. In many situations, success was not achieved for 4 reasons outlined below.  The last phase is the turning point, one I recommend you consider before beginning the journey to integrate EA tools with ServiceNow at all.

These implementation phases are:

• The Independent Phase: Internal EA and operations teams establish their respective platforms independently, with limited or no initial integration. There is typically little to no overarching data model or semantics connecting these independent platforms. Often, the out-of-the-box data models may be changed in ways that make integrations down the line, more complex.

• The Honeymoon Phase: An initial integration is undertaken and seems to work well. This first-cut effort is usually enriched over time with more data flows, supported by increasingly complex processes and automations to integrate the EA and CMDB.

Here are some resources on how you can start your integration journey:

• CSDM V3 to ArchiMate framework mapping video
• White Paper on mapping CSDM V4 to ArchiMate

• The Ugly Reckoning Phase: As integrations become increasingly complex, cracks start to appear, revealing serious data quality, consistency. and other challenges as outlined in the 4 specific topics below. Increasingly complex interoperability of data and processes result in untrusted data and unexpected results in each system that erode trust.  Stakeholders find hard limit on realizing value from each, and teams begin to question how to best proceed.

More advanced features on ServiceNow using Strategic Portfolio Management, or Technology Portfolio Management as summarized here, can remain out of reach.

Making different choices in the first two phases can avoid many problems that will arise in the Ugly Reckoning Phase and make the integration more manageable but still unsuccessful long-term. However often there is no clear path forward as data overlaps grow, and processes become unmanageably complex.

Common Integration Challenges

In the quest for a robust integration, four different types of challenges become glaringly apparent by the third phase. These are often the root-cause to the problematic and frustrating results that begin to occur in the Ugly Reckoning phase of implementation.  The types of challenges are:

• Chicken-and-Egg: Which system will originate what data? This challenge will grow over time as the information needs in each evolve, and overlap. Knowing what entity is created in each source system is essential for defining the direction of information flow between them. A typical example is the problem of mapping data placed in the CMDB by discovery tools with the corresponding objects in the EA tool repository. Or identifying in what solution a Business Service or Application may be created first.

For example, the EA team may create a Business Application, or a new Business Application may be created operationally through the APM onboarding process where a new ID is provided.  Given this new data entity can originate in either system, what is  the “true” source and direction of data flow you should choose?

• Pulling the thread on the sweater: As the number of elements needed in each system grows, more information needs to traverse the integration between the CMDB and the EA tool.

For example, the scope of an initial integration may only be the Application entities, but it quickly becomes clear that business unit hierarchy is also needed, then locations, then vendor product details, and so on. The scope continues to grow and eventually requires a large portion of the data model in both systems to be synchronized. At this point teams begin to realize most data is now being replicated.

• Semantic Mismatches: The semantics and data modeling approach of ServiceNow and EA frameworks were initially designed for different purposes and were not inherently compatible.

For example, CSDM depends heavily on an Application Services concept to articulate a logical construct for differentiating environments (Dev, QA, Prod), Microservices, and platform host / platform concepts. EA frameworks and tools however don't support an Application Service concept in the same way.

• Relationship source: The above issues are compounded by the need to understand where and how the relationships between entities will be or should be managed. Each such relationship is itself is data that must be managed in one system and mirrored in the other to complete the model.

How should a new entity proposed from an EA perspective, be aligned to the CMDB and governed in ServiceNow? This becomes challenging for every case in which the full ServiceNow model of the current state is not also in the EA tool. Omitting relationships may lead to disconnected entities that were not considered in the EA tools, or visa-versa. If each system were not updated with the correct relationship (creation or deletion) at the right time, there will be a growing difference between them.

These issues can become so severe that a robust integration is simply unachievable as originally intended. Similarly, we struggled in trying to productize an effective integration while I was a PM at Troux working with HP.

A different approach

In my opinion: The best way to integrate an EA tool with operational data stores like ServiceNow is to avoid the traditional two-body approach entirely. Is it possible? What do you lose in a single-body approach?

I realized when joining service now that a better long-term solution would be to pursue a fully native platform approach in which all EA and operational data reside side-by-side on a single platform. ServiceNow has proven to be a winning one-stop replacement strategy almost all integrated IT management tools.  We have evolved our data model, to cover s robust IT data model addressing needs from the Enterprise Architect through operations.

Completely avoiding the inevitable 4 challenges outlined above, ServiceNow products such Application Portfolio Management provides functionality that EA’s need. This including Technology Reference model, Technology Portfolio Management, Information Object modeling, and Capability based planning.  There is also a Lucid integration now available. In addition, we have partners who have products that operate within the platform available in our store that function such as Ins-Pi’s Designer modeling tool.

Conclusion

Enterprise Architects may argue that ServiceNow doesn’t have all the EA capabilities out-of-the-box that other EA tools offer, yet. That ServiceNow does not currently appear in the Analyst reports that covers Enterprise Architecture, so we shouldn’t consider it. This is like making an argument that using a dedicated camera over the one built-into your phone is better. True for photographers with the skills and need, however the utility of a built-in phone is far more useful to the masses, and natively integrates, allows anyone with a phone to be a proficient photographer.  The same can be said about the Enterprise Architecture capabilities on the ServiceNow platform.

Over the years, ServiceNow has steadily invested to strengthen our Common Service Data Model and CMDB capabilities to provide a native ability to support almost all IT management data modeling needs in a single platform, including Enterprise Architecture. ServiceNow products and ecosystem of partner products keeps getting get better to support EA requirements natively. Accommodating all the needs that the typical EA needs to function is now well within reach on-platform, completely avoiding the integration challenges faced as you integrate separate EA tools with ServiceNow.

Your ServiceNow EA journey starts here: Application Portfolio Management

[Donabet]

Thank you for sharing your thoughts, Mark.

It resonates very well with me as it depicts what I call Pragmatic Excellence.

Why trying to (maybe sometimes even forcefully) bridge these taxonomies, semantics, relationships, etc in an ever faster evolving Technology environment when one can maybe gain benefits from within an existing journey.

If for example your software spend, and/or compliances are on a critical path and you're already embarking on a managed SAM path with servicenow being the SAM/ITAM platform that supports you; yes, in alignment with your smartphone camera analogy, then servicenow APM could be potential first choice to look into.

However, make no mistake, it's not the platform or tool that's in the driver seat but your clearly articulated priorities/objectives to gain transparency about your technology/software/application landscape, and if and how it enables and supports your business capabilities and strategy.

Very often already incremental steps lead to a significant positive impact. Be clear about the intent and be pragmatic. Rest I leave to the evangelists.

That's again for the diversity of thinking.