Introduction
As a current BMC shop taking a serious look at a move to ServiceNow, we currently have a substantial investment in BMC's tool stack in the monitoring, event management, and discovery space.
One of the tools from BMC that we would need to integrate with ServiceNow is ADDM, which is BMC's discovery tool. Like most customers, we'd like to populate our CMDB with automatically discovered data wherever possible. ADDM exposes it's data through a "CSV API" which is documented here. So if I had my way, my preference would be to take data into ServiceNow via a MID server using that CSV API, which is more or less a very simple REST API
Unfortunately ServiceNow does not support MID server REST calls as a data source in it's off-the-shelf Data Sources, so using the native ServiceNow Data Source import method would not be possible. As an alternative, I wrote an integration that comes as close as possible to emulating that approach. An update set with the integration is attached below (built on Fuji). Since it makes use of the RESTMessageV2 API it will not work on prior SN versions at the moment. I'd appreciate any feedback since this is my first attempt at this sort of integration with ServiceNow.
Approach
The overall approach is divided into 6 parts:
- Determine what specific ADDM data you want to import into ServiceNow
- Generate an ADDM "raw query" for each set of CI data (i.e. servers, databases, etc.) you want to import
- Set up mappings in ServiceNow to consume data from ADDM using those raw queries
- Load the data into Import Set Tables
- Use ServiceNow's native Transform Map capabilities to map the data from the Import Set tables to the appropriate CMDB tables.
- Set up a scheduled job to periodically run the import
I'll go into the details of each step in the following sections.
Determine the Data to Import
ADDM, like many discovery tools finds a lot of stuff, much of which is not necessarily appropriate for loading into your CMDB. So from the outset it was a very conscious decision that we would explicitly choose the types of items to import, and the specific attributes from those items. We would not be looking at a "pull in everything" approach, especially since every CI type that is pulled in will ultimately need transform mappings and logic to get it into the right place in the CMDB and the effort to do that is not justifiable for certain types of ADDM data. We wanted the ability to focus on particular CI types and prioritize our efforts in on-boarding them into the CMDB.
In our case that meant our "first wave" of activity would focus on servers, databases, web servers, and app servers since those constitute most of the stack for most of our applications. Your mileage may vary on this front, but the integration is designed to pull "slices" of ADDM data along these lines. If anyone has tried more of a "load everything" approach and been successful, I'd love to hear from you and compare notes.
Generate ADDM Raw Queries
When integrating with RDBMS data source we need to define SQL queries that select the data to pull. In the case of ADDM, we use something called a "Raw Query" in an analogous way. The example below is a raw query I've used for getting Oracle listeners. For better readability I've added line breaks but in reality this would need to be a single unbroken line:
SEARCH
SoftwareInstance
WHERE
(type = 'Oracle Net Services (TNS) Listener')SHOW
product_version,listen_tcp_sockets AS 'Listen Tcp Sockets',
served_oracle_instances AS 'Served Oracle Instances',
version AS 'Full Version',
ora_home AS 'Ora Home',#RunningSoftware:HostedSoftware:Host:Host.key AS 'Runs On Host Key',
#InferredElement:Inference:Primary:DiscoveredProcess.cmd AS 'Primary processes',#RunningSoftware:HostedSoftware:Host:Host.name AS 'Host'
As you can see although the syntax is different from SQL, it is pretty intuitive to understand what these ADDM raw queries are doing. The only challenge is knowing what the attribute names are, and figuring out how to traverse ADDM relationships to pull in extra goodies like the "runs on host" information.Fortunately, ADDM can generate these queries for you. Just use their GUI to customize a report with all of the attributes of interest and whatever filtering you want. Then just click the Raw Query tab and ADDM shows you the text of the raw query and you can just copy/paste it from there. You would go through this process to obtain a different raw query for each set of CI data you'd like to import.
Set up Mappings in ServiceNow
This is where we start to actually use components of the integration solution in ServiceNow. The update set adds a new application menu to the nav panel called "BMC ADDM Integration" with modules for the common administrative tasks associated with the integration. Hopefully it is pretty self-explanatory.
Before we set up our mappings though, we need to tell ServiceNow how to connect to ADDM. The "Properties" module brings up a list of the available settings, where this can be specified as shown below:
Now that our ServiceNow instance knows where the ADDM server is, we can go ahead and set up mappings between the raw queries we obtained from ADDM in the previous step, and an Import Set Table. We define these mappings using the "ADDM Query Mappings" module, and then click on New to create a new one. This is what the form looks like once we have filled it in using the raw query from above:
So basically we are giving our mapping a name/description, specifying the raw query to use when making our request to ADDM, providing a label name for the import set table to be used, and specifying whether or not the mapping is currently active. Some points worth mentioning here:
- The Description field is not really used for anything. It is just to help you keep track of your mappings.
- The name of the import set table will be derived from the table label contained in the "Import Set Table" field by prefixing "u_", converting everything to lower case, and replacing any non-alphanumeric characters with underscores.
- If the import set table does not yet exist it will be created the first time the mapping is processed, along with an Import Set.
- The columns in the import set table will be automatically created using the column names of the data that ADDM returns.
- If you subsequently change the query for this mapping, any new columns needed in the import set table will be dynamically added to it
- Columns that are subsequently removed from your ADDM raw query will not be removed from your import set table (we only add columns)
- If you want to rebuild your import set table from scratch (thus losing all previous import data for this mapping) you can use the "Delete Import Set and Import Table" UI Action. Use with caution since you will lose your data!!
Load the Data
Now for the moment of truth. At this point we can manually initiate a load by clicking either the "Process This Mapping" or "Process All Mappings" UI Action on the ADDM Query Mapping form. If all goes well, the import will complete and redirect us to the Import Set it was placed into. It should look something like the screenshot below. Note that the "Import Set Runs" tab contains no rows yet since we have not defined a Transform Map.
Next Steps
At this point we are in well-known territory, which is simply defining Transform Maps for our newly imported data and then setting up a Scheduled Job to run the import automatically as needed. I'm not going to go into details on those since they are pretty basic stuff (as many of my college texts liked to say, it is "left as an exercise to the reader"). However I will call out a few items of interest.
Scheduled Job vs. Scheduled Import
First of all, I would prefer to have implemented this in such a way that instead of using the more generic "Scheduled Job" feature in ServiceNow, we could use "Scheduled Imports". We are importing data, after all. Unfortunately since Scheduled Imports relies on Data Sources, this would have involved modifying the out-of-the-box sys_data_source table and it's associated workflow and it is not clear to me that hacking into that would be a good idea when it comes to upgrades and such. So Scheduled Job it is (at least for now).
Script for the Scheduled Job
The Scheduled Job for these imports would need to run a script that looks like this:
var addm = new BMCADDMIntegration();
addm.processAllMappings();
This will import all active ADDM Query Mappings that you have defined.
Import Sets - Asynchronous vs. Synchronous
To work around the fact that we initially have no Transform Maps defined when we process the query mappings for the first time, the Import Sets that are generated will have their Mode set to Asynchronous. Before running your scheduled job for the first time you will need to change the Mode of each Import set to Synchronous.
Conclusion
So there it is, a solution for integrating BMC ADDM with ServiceNow. I'm sure there are lots of improvements that could be made, and I'm interested in any feedback that could help it evolve into something better. Also if you've integrated with ADDM using a completely different approach, I'd love to hear from you so we can compare notes.
Oh yeah, and here's the update set...
