As of the 1.49 version (released September 2023) of the CMDB CI Models app on the ServiceNow store, there is a new data model for collecting API inventory and related information. While there are many different types of APIs (REST, SOAP, gRPC, etc) and many technologies supporting APIs (gateways, meshes, observability, CASB, etc), this new data model in CMDB is meant to be a normalized view across these various types and technologies.
Of the 10 new CI classes (and 6 related non-CMDB tables), some customers may find it sufficient to populate just a single table, while other integrations may target 9 or more of the tables. The point is that the model is designed to be flexible based on an organization’s data sources for such inventory. This article is intended to describe the data model in smaller segments and supplement the official ServiceNow documentation.
These are the new CI classes:
- API (cmdb_ci_api)
- API Component (cmdb_ci_api_component)
- API Frontend (cmdb_ci_api_frontend)
- API Backend (cmdb_ci_api_backend)
- API Gateway (cmdb_ci_api_gateway)
- Managed API (cmdb_ci_managed_api)
- Kong Gateway (cmdb_ci_kong_gateway)
- Kong Load Balancer (cmdb_ci_kong_lb)
- Kong Target (cmdb_ci_kong_target)
- Unmatched API Endpoint (cmdb_ci_unmatched_api_endpoint)
These are the new non-CMDB related tables:
- API Consumer (api_consumer)
- API Policy (api_policy)
- API Deployment (api_deployment)
- API Header (api_header)
- Kong Workspace (kong_workspace)
- API Endpoint Discovered (api_endpoint_discovered)
The Core Data Model
Let's start by looking at the first 2 new CI classes, API and API Component.
Core of the new API data model
The API class (cmdb_ci_api) is designed to store information about the top-level API like the name, version, and base URL. The API Component (cmdb_ci_api_component) is designed to represent the various resources of the API. These will include unique method and path combinations, varying authentication requirements, ports used, and more. Both of these classes are independent, but the ideal state would have the API Components be related to an API with the “Uses::Used By” relationship.
The API Deployment related item is meant to represent where an API is “running”. The basic information is just a “Name”, but if an existing configuration item can be identified, then the “Configuration Item” attribute may be manually populated as it is a reference to the cmdb_ci base class. An API can, for example, be deployed to multiple servers or possibly to SaaS-based serverless services. APIs are also commonly deployed within a set of containers running in a Kubernetes cluster.
The API Header related item for components is meant to capture a header schema. The “Name” attribute will store a header key, like “Authorization” or “Content-Type”.
An example of how this might look can come directly from ServiceNow if we look at the REST API Explorer.
ServiceNow Example
In this example, we are looking at the CMDB Instance API which would presumably be stored in the base API class (cmdb_ci_api). The various resources below that like "GET Query records for a CMDB class" would represent the different API components (cmdb_ci_api_component) related to the API.
Expanding the Data Model to API Gateways
API gateways have very structured representations of APIs and their components and are an ideal source of API inventory data. In addition to the two new classes described above, there are a number of classes surrounding these two which are used when populating from API Gateway sources. Expanding on the core model, we continue to build out the data model as follows:
Expanded data model for API Gateway data
A couple of initial points to notice are that the API class now has a new child class named "Managed API" (cmdb_ci_managed_api). This class has a dependency on the API Gateway class which the parent API class does not have.
Likewise, the API Component class has two new child classes, API Frontend (cmdb_ci_api_frontend) and API Backend (cmdb_ci_api_backend). These two new child classes are used to store common components found in API Gateways which represent the client facing frontend (request) endpoint and the request fulfilling backend service which provides the response.
We can look at an example of this using the Azure API Management service.
Azure API Management Service example
While the table "cmdb_ci_azure_gateway" does not currently exist, hopefully you can extrapolate how this will look in the future with that being a child class of the API Gateway class introduced in this new data model. In the picture of the Azure API Management service, you can see a number of APIs. When choosing the "Echo API" we see what Azure calls the various operations of the API. These are what we will normalize into the data model as API Components. However, we will not store them in that parent class, but use the additional data of this gateway to populate the API Frontend class with the "frontend" values like "POST /resource" and the API Backend class with the "backend" values like "http://echoapi.cloudapp.net/api" and related data about that resource.
We do not plan to replicate everything in the gateway into the ServiceNow CMDB, but rather, our intent is to capture the relevant data for achieving outcomes in the following use cases.
Example Data
Below is an example of how a gateway would populate the managed API dependent class which is a first level relationship to the gateway (denoted by the “L1”). You can also see the Frontend and Backend components below which map to the respective managed API and are second level relationships to the gateway (denoted by the “L2).
Kong Gateway Example
Below is a dependency view of the same gateway with related APIs and components.
Example dependency view
From this point, alerts can be bound to these CI's, dynamic CI groups can be created for service views, incidents and changes can be created against the CIs, vulnerable items can be filed, and any other workflow you have that uses CIs.
Data Model for Unstructured Data Sources
There are a couple of tables we have not touched on yet, which are “Unmatched API Endpoint” and the “API Endpoint Discovered”. These tables (1 is a CI class and 1 is meant to serve as more of a staging table) are used for use cases where the API data is very unstructured and needs to be processed beyond the normal means available in IntegrationHub-ETL.
The reason for creating this extra class for what is effectively the same as the API component class is that certain products in ServiceNow require a CI (like vulnerability management), but the data does not come into the system in a way where it can have standard identification rules applied. For the sake of data quality while still facilitating workflows like vulnerability management, it was necessary to introduce this unmatched class as a place where CIs could be created but have offline analytics run to further cleanse and possibly migrate the data to the API component class if possible.
Unstructured API Tables along with API and component
Let’s take a quick example of various types of data and how this may be handled.
Example 1: GET https://host/api/users/v1/{id}
Example 2: GET https://host/api/users/v2/{id}
Example 3: GET https://host/users/john/company/123/
Example 4: GET https://host/users/jane/company/345/
Example 5: GET https://host/users/{alpha}/company/{int}
Handling this data will be dependent upon how an integration is built, by either ServiceNow, a partner, or a customer. However, we can look more closely at the first 2 examples and possibly use logic that says to parse up to a “/v[numeric]” which would result in us know there is a “users” API with 2 versions and a path expecting an “{id}”. If this was the only data coming, then we could create 2 API records (cmdb_ci_api), one for each version, and a single corresponding API component for the GET method.
The last 3 examples are where the unstructured process and tables comes in. Examples 3 and 4 are effectively the same API component request, but they have different query parameters encoded into the URL. There is no way to programmatically know how to parse this URL and make it generic enough like other similarly structured records (or unstructured) that might have more or less query parameters in the URL string. These records would end up in the “Unmatched API Endpoint” class.
Sometime later the source system recognized the pattern and makes it a more general entry when seeing this traffic. This results in Example 5 coming into the system, which represents the same endpoint as Examples 3 and 4, but now is made more generic. Without going into the ways this could be evaluated for proper structure, we can assume that it would meet the quality checks necessary and store this record in the API Component class while also marking the records in “Unmatched API Endpoint” for Examples 3 and 4 as “Absent” or “Non-operational” or some other way that disqualifies them from operational use.
The staging table (it’s not an import set row table) is there to serve as a place to store the raw data and then perform processing as described above.
Flexibility in the Model
As you have hopefully gathered from this article, API related data comes in many forms. The goal of this data model in CMDB is to provide customers the flexibility to capture what’s important to facilitate their workflows and based on what data sources they have available with such information. The intent from ServiceNow is to continue to build out the data model as we create out of the box integrations to help customers populate the data model and keep it up to date automatically. This is just the beginning!
@vNick I am importing my data using Integration hub ETL and its working fine except for the relationship step. I cant get any relationships to populate so I'm confused on how to accomplish this. A lot of our APIs depend on each other and I'm capturing those relationships in my import, but the relationship step doesn't allow me to pick anything. Is there a setting I'm missing before this part? The APIs and the API components aren't "parent/child" relationships, but I would hope I can specify relationships between classes here. Ideally I want to be able to specify "provides" or "depends on" relationships.
Importing class to Component or API table based on an identifier.
And here's where I run into issues, the relationships button is grayed out and if I select the first dropdown no option is present for the second.
@IndianaJones - sorry for the late response. I think the relationship setup has to do with the conditional class usage. You should map to both API and API Component in the ETL (or have 1 for API and a different one for component, but in the same app, then reference the API source native key when mapping the component). There is a suggested relationship between API and API component so the relationship setup is possible.
I have modeled Microsoft Azure Resource API as a POC (populating API for each API path/endpoint and API Component for each API operation)
Now I want to map it against API Client. I intend to use API Client --|consumes|--> API Component.
In Microsoft we can create "Applications" or "Service Principals" and assign API permissions and Client ID/Secret to consume these APIs.
Whether I have an inhouse API Gateway deployment or SaaS based API Gateway (eg: Microsoft Azure Resource Manager), I believe the API Clients should be stored in cmdb_ci_api_consumer_subscription. Or should i use api_consumer?
What's the difference between api_consumer (non CMDB) and cmdb_ci_api_consumer_subscription (description: An API Consumer Subscription is created by a developer once they register on a developer portal and want to access one or more API products. This registers with one or more API product bundles in order to obtain a key to access the APIs within the bundle.)?
In short, where should I store these API Client definitions?
