Managing delta with IntegrationHub ETL

[midjoule]

Dear experts,

Context

I need to update a custom field called "Monitored By" in the CMDB thanks to an import from an external system.

This import is done through IntegrationHub ETL and a REST API call done in multiple batches.

This scheduled data import runs on a daily basis. Each day, ServiceNow retrieves the servers that are monitored by a system (here : PRTG). It provides the current state of monitoring. All the servers that are monitored by PRTG are present in the import set. Thanks to IntegrationHub, I can retrieve the server that needs to be updated.

Challenge

Now, my problem is how to handle the delta, where a server was yesterday monitored and today not any longer. I need to remove the value "prtg" from the field "Monitored by" of that particular server. As it is not present in the payload, IntegrationHub ETL doesn't remove it automatically.

I've tried to create a pre-script / a post script in either the scheduled data import or in the transform map of the data source  (ie. CMDB Integration Studio Application Data Source), but without being able to complete it.

• If the script is in the Scheduled Data Import, I don't have access to the payload of items that have been transformed to identify those that have been provided by PRTG.
• If the script is in the transform map of the data source, I don't have the full access to all items provided by PRTG, because it is managed by batch.

Has anybody gone through a similar challenge ? How ?

Thanks for your help. 

[Samuel von Hofs]

You could consider scheduling a separate job to run after the full PRTG input is processed, and check for CIs that has discovery source = PRTG but is not updated by the last input, and then update those CIs. A bit risky if there are delays or errors during the PRTG input processing but might be worth looking into.

You could also consider using the CMDB health staleness rules, use those to identify the relevant CIs and put this correction into the remediation activities.

[midjoule]

Thanks Samuel for your inputs. 

The problem with your options and my setup is that the processing doesn't update the CI in case there is no change (was monitored yesterday and today also). 

[Samuel von Hofs · Accepted solution]

Oh, thanks for clarifying. In a similar scenario, we set up the source to provide us with a full sync (always sending all of the current monitored hosts and producing at the very least a time stamp update if nothing else has changed) as well as the delta (only new inserts and updated values) at more frequent intervals. In this scenario, a server previously updated through this input and then suddenly not being updated would be a flag to us that it's removed, so you could run the clean-up job after the full sync has been processed. I thought "All the servers that are monitored by PRTG are present in the import set." would put you in this scenario. 

Otherwise I would again look at the staleness rules to estimate how often you'd expect the records to be seen through the delta. It's a less reliable and more manual approach, but could work if you have a sturdy process.