We're reclaiming inactive PDIs to keep them available for active builders. Learn what's changing, who's affected, and how to protect your work. Read More

Map ETL for Foundation Data sync

pratik_karande
Tera Contributor

We have a requirement to sync cmdb from one servicenow instance to another using FDS. We have configured up till FDS request creation and sending sample data from provider to consumer. 
Next step is to map the inbound data using ETL transform. In this we are facing issue while mapping the key value table to the cmdb tables. As key value is a standalone table it does not give an option to use ETL or transform map on the subscription item on consumer instance. however when we add an IRE to the cmdb tables for cmdb_key_value table we get an option to configure a lookup on that particular table (say cmdb_ci_group) using the key value table. how do I get the related key value data in the particular cmdb table through same offering item. As there is no direct relation between the cmdb table and key value table. Refer screenshots below for better understanding 

pratik_karande_0-1782984666206.png

below is cmdb_key_value table subscription item without any transforming option. How do we get the data recieved in this table and map it to the other cmdb table data as above.

pratik_karande_1-1782984723465.png

 

1 REPLY 1

Vikram Reddy
Tera Guru

Hi Pratik,

 

I would not treat cmdb_key_value as a normal CMDB class in the FDS consumer mapping.

cmdb_key_value is a dependent/supporting table. The actual relationship to the CI is through the configuration_item reference plus the key. It is not a CI class by itself, so Lens/ETL will not behave the same way as it does for cmdb_ci_group, cmdb_ci_network_service_instance, etc.

The important point is this:

You cannot reliably map cmdb_key_value into a CI table unless the key/value row has a stable way to identify the parent CI on the consumer instance.

The provider CI sys_id is not enough by itself, because the consumer instance will have a different CI sys_id after IRE creates or updates the CI.

 

Recommended approach:

1. Sync the parent CI first

For example:

cmdb_ci_group
cmdb_ci_network_service_instance
cmdb_ci_ni_site

Map those records through ETL and IRE first.

Make sure each CI has a stable correlation value from the provider, for example:

source_native_key
correlation_id
external_id
provider_sys_id stored in a custom field
discovery_source plus native key

This value is what you will use later to locate the same CI on the consumer side.

 

2. Treat cmdb_key_value as child data

The key/value record should be processed only after the parent CI exists on the consumer instance.

For each incoming key/value row, you need:

configuration_item from provider
key
value

Then resolve the provider configuration_item to the consumer CI using the stable identifier from step 1.

After you find the local CI sys_id, upsert the cmdb_key_value record using:

configuration_item = local consumer CI sys_id
key = incoming key
value = incoming value

Coalesce should be:

configuration_item + key

 

3. Do not expect the cmdb_key_value subscription item to map directly like a CMDB class

The subscription item for cmdb_key_value can receive the data, but it does not automatically know which target CI record in the consumer instance should be updated unless you build that lookup.

That is why you see mapping options when you are inside a CMDB class ETL/IRE configuration, but not the same transform behavior directly on the standalone cmdb_key_value subscription item.

 

4. If the key/value data belongs to a specific CI offering, include it with that CI payload if possible

The cleanest design is to include the key/value data as part of the same offering item/payload as the parent CI.

For example, for a network service instance payload, send:

network service instance fields
source native key
related key/value attributes

Then the ETL for that CI class can use the key/value data while it is processing that parent object.

If the key/value data is sent as a completely separate offering item, then you need a second-stage process to connect it back to the already-created CI.

 

5. If separate subscription item is required, use post-processing

If FDS requires cmdb_key_value to be a separate subscription item, I would create a post-processing step on the consumer side.

This can be:

Business Rule
Flow
Scheduled job
Scripted transform logic
Custom post-import processor

The logic should be:

Read inbound cmdb_key_value row
Get provider configuration_item identifier
Look up local CI by source_native_key or correlation field
If local CI is found, insert or update cmdb_key_value
If local CI is not found, leave the row pending and retry after parent CI sync

 

Pseudo logic:

var localCI = new GlideRecord('cmdb_ci');
localCI.addQuery('u_source_native_key', incoming.configuration_item);
localCI.query();

if (localCI.next()) {
var kv = new GlideRecord('cmdb_key_value');
kv.addQuery('configuration_item', localCI.sys_id);
kv.addQuery('key', incoming.key);
kv.query();

if (!kv.next()) {
kv.initialize();
kv.configuration_item = localCI.sys_id;
kv.key = incoming.key;
}

kv.value = incoming.value;
kv.update();
}

 

6. Do not use provider sys_id directly as configuration_item

This is the most common issue in cross-instance CMDB sync.

The configuration_item value coming from the provider instance points to a CI sys_id in the provider instance. That sys_id will not be the same in the consumer instance.

So the key/value transform must translate provider CI identity to consumer CI identity before writing cmdb_key_value.

 

7. Processing order matters

The parent CI ETL must run first.

Then the key/value processing should run.

If key/value data is processed before the parent CI exists, the lookup will fail.

For that reason, I would keep a retry/pending mechanism for key/value rows where the parent CI has not yet been created by IRE.

 

Recommended final design:

1. Parent CI subscription item maps to the CMDB class through ETL and IRE.
2. Parent CI stores provider/native key on the consumer CI.
3. Key/value subscription item receives key/value rows.
4. A post-processing step resolves provider configuration_item to the local consumer CI.
5. The process inserts or updates cmdb_key_value using local CI sys_id plus key.
6. Any unresolved key/value rows are retried after the parent CI sync completes.

 

So the short answer is:

The cmdb_key_value table should not be mapped as if it is the target CMDB class. It should be handled as dependent CI metadata. You need to resolve the parent CI first, then upsert the key/value records using the consumer CI sys_id and key as the unique combination.

If you want to use it in the same ETL map, the related key/value data must be available in the same source object/collection being processed for that parent CI. If it is coming as a standalone subscription item, use a second-stage transform or scripted post-processing to attach it to the correct CI.

 

Thank you,

Vikram Karety

Octigo Solutions INC