CMDB Data Ingestion API to the REST Insert Multiple

[ram_mandadapu]

Beginning with the Quebec release, partners and customers are advised to switch from the CMDB Data Ingestion API to the REST Insert Multiple endpoint of Import Set API. Please refer to the Import Set API documentation here. 

Differences between the CMDB Data Ingestion API and Import Set API

1. Be sure to define the import set table prior to making any calls to the REST Insert Multiple API. The existing import set tables created with CMDB Data Ingestion API will work with new Import Set API. 
2. The REST Insert Multiple supports synchronous and asynchronous transformation. It also allows ordered transformation of data. 
3. REST Insert Multiple groups data via Multi Imports. This will transform data asynchronously without blocking semaphore threads. 

Migration steps

1. Assign the import_admin role to the integration user. 
2. Change the ServiceNow endpoint to ‘/api/now/import/{stagingTableName}/insertMultiple’ on the external system. 
3. The ingest API is driven based on the Data Source, whereas Import Set API by default won’t honor the Data Source settings. To change the default behavior of REST Insert Multiple, create an entry in the REST Insert Multiples [sys_rest_insert_multiple] table, then select the Use data source format check box.   

Example CURL command for Ingestion API

curl "http:// <instancename>.service-now.com/api/now/cmdb/ingest/<datasource_sys_id>" \ 

--request POST \ 

--header "Accept:application/json" \ 

--header "Content-Type:application/json" \ 

--data "{ 

    \"DeviceApps\": [ 

        { 

            \"ApplicationName\": \"Agent\", 

            \"Version\": \"5.7.0\", 

            \"BuildVersion\": \"26\", 

            \"Status\": 2, 

            \"Size\": \"35442688\", 

            \"ApplicationIdentifier\": \"com.air-watch.agent\", 

            \"Type\": \"Public\", 

            \"IsManaged\": false 

        }, 

        { 

            \"ApplicationName\": \"VMware Workspace ONE\", 

            \"Version\": \"\", 

            \"BuildVersion\": \"\", 

            \"Status\": 1, 

            \"Size\": \"0\", 

            \"ApplicationIdentifier\": \"com.air-watch.appcenter\", 

            \"Type\": \"Public\", 

            \"IsManaged\": true, 

            \"Id\": { 

                \"Value\": 167 

            } 

        } 

    ] 

}" \ 

--user 'username':'password' 

Example CURL command for REST Insert Multiples

curl "http://<instancename>.service-now.com/api/now/import/<import_table_name>/insertMultiple" \ 

--request POST \ 

--header "Accept:application/json" \ 

--header "Content-Type:application/json" \ 

--data "{ 

    \"DeviceApps\": [ 

        { 

            \"ApplicationName\": \"Agent\", 

            \"Version\": \"5.7.0\", 

            \"BuildVersion\": \"26\", 

            \"Status\": 2, 

            \"Size\": \"35442688\", 

            \"ApplicationIdentifier\": \"com.air-watch.agent\", 

            \"Type\": \"Public\", 

            \"IsManaged\": false 

        }, 

        { 

            \"ApplicationName\": \"VMware Workspace ONE\", 

            \"Version\": \"\", 

            \"BuildVersion\": \"\", 

            \"Status\": 1, 

            \"Size\": \"0\", 

            \"ApplicationIdentifier\": \"com.air-watch.appcenter\", 

            \"Type\": \"Public\", 

            \"IsManaged\": true, 

            \"Id\": { 

                \"Value\": 167 

            } 

        } 

    ] 

}" \ 

--user 'username':'password' 

Note: When you want to divide related large data into multiple parts and push it, first request to the Import Set API returns a multi import set ID. You need to pass this as query param starting from the second request so that all the import sets will be grouped under the same Multi-import set record. 

 Example: "multi_import_set_id": "082c3fb1ffb310105cf343d0653bf151" 

   

Nested Payload or Data in Single Column

When the customers/partners want to load data in single column and use the nested payload feature on IH-ETL, they must select the ‘Data in Single Column’ check box on the data source and create an entry in the REST Insert Multiples [sys_rest_insert_multiple] table with ‘Use data source format’ flag enabled. 

Orlando and Paris fix to process import sets that are not processed by Ingestion API

For partners/customers who are using CMDB Ingestion API on the Orlando or Paris release, they should not use admin user credentials to push the data. Please create an integration user with the role cmdb_import_api_admin for this purpose. Also, create a scheduled job per app scope and add the script below to process import sets that are loaded but not processed by the CMDB Ingestion API.  

Note: Set the appropriate table name for the variable importSetTableName.

Example script: 

var importSetTableName = "<import_set_table_name>"; 

var gdt = new GlideDateTime(); 

gdt.addSeconds(-300); 

var importSet = new GlideRecord('sys_import_set'); 

importSet.addQuery('table_name',importSetTableName); 

importSet.addQuery('sys_created_on','<',gdt); 

importSet.addQuery('mode', 'asynchronous'); 

importSet.addQuery('state', 'loaded'); 

importSet.query(); 

while(importSet.next()){ 

     var importSetToProcess = new GlideRecord('sys_import_set'); 

     importSetToProcess.get(importSet.getUniqueValue()); 

     var transformer = new GlideImportSetTransformer(); 

     transformer.transformAllMaps(importSetToProcess); 

} 

If you have multiple import set tables, move this code into a function and pass the import set table name as a parameter.

[sanjay_gupta]

Great conent.