IH-ETL Guided Setup Step 6 - Test and Rollback Integration Results

[Siyou Li]

Previous Step

IH-ETL Guided Setup Step 5 - Relationship Mapping

After you configure the mapping, you are ready to see what the data looks like. Run an integration test and view a summary of the results for the sample data. The summary includes total numbers for relationships that were created, mapped classes, partial and incomplete payloads that IRE couldn’t process. You can also view detailed messages from RTE and IRE. After you view the details in the summary page, you can return to any step to make adjustments and then rerun the integration. Below is an example:

1. Click “Preview Mapping Results” 

2. Click “Run Integration”

Be aware that this executes your transform map and loads data into CMDB. You always have the options to rollback loaded data later.

3. Execution might take a little while. After it executes you should see your results

4. Click “Edit Mapping” or “Mark as Complete”

The system prompts you to roll back your changes or retain your data.

5. If you want to delete the CIs created by your preview, select “Perform Rollback” 

6. If you want to retain the CIs in your CMDB, select “Retain Data”

Tips and Caveats

A conditional class lets you map different sets of data records to different target classes according to specific column values, or the status of a specific plugin. 

Preview Result Size

Use property “sn_int_studio.preview.size” to control preview result size (max 10k). By default, IH-ETL will only process 100 input rows.

Partial Payload 

• IRE unique identification requires that the payload item has source_native_key values, or the full set of the identification criterion attributes specified for the CI class, or both 

• IRE isolates items for which data sources did not provide enough information to uniquely identify the CI and therefore processing cannot continue 

• Some of these items are identified as partial items, which get stored for potential later processing

• For example: SCCM has multiple feeds such as a disk feed and a computer feed. The disk feed might have complete information about the disk but insufficient information about the computer CI that it depends on 

Below are scenarios that will be identified as partial payload 

• MISSING_MATCHING_ATTRIBUTES –— Item does not have identification criterion attributes to use at least one identifier entry for matching 

• REQUIRED_ATTRIBUTE_EMPTY –— Unable to create a CI because a required attribute is missing

• MISSING_DEPENDENCY –— Dependent CI is missing a dependency relation which is specified in the payload 

• INSERT_NOT_ALLOWED_FOR_SOURCE –— An IRE data source rule prevents the specified data sources from creating CIs of the specified class 

Partial payload IRE documentation

Refer to FAQ section for common error scenarios and solutions

Incomplete Payload 

A payload item is determined to be an incomplete item if: 

• It does not contain all the data necessary for IRE unique identification, and

• It has an error that is not associated with a partial item 

IRE unique identification is not possible if neither source_native_key, nor the full set of identification criterion attributes specified for the CI class, is provided. 

Incomplete payload IRE documentation

Refer to FAQ section for common solutions

Additional Resources

Landing page

IH-ETL common error scenarios

Next Step

FAQs and debug process for common error scenarios

[Community Alums]

Hi,

Thanks for the detailed information. Its really helpful 🙂

I have few doubts on IntegrationHub ETL

1. sn_int_studio.preview.size - max 10K - which means through integrationHub ETL we can load only 10000 records at once

2. Relationship - IntegrationHub allow to add only CI-CI relationship (cmdb_rel_ci) 

can we add CI-Group relationship (cmdb_rel_group) through IntegrationHub ETL? any possibility?

Please suggest .

[Siyou Li]

Re: @Community Alums 

Hi,

1. Yes, only max 10k records can be loaded into IH-ETL. If user has large amount of data, we suggest only loading a partial amount of data which is enough for previewing data structure. Then construct the mapping logic using IH-ETL. Then create scheduled transform to transform the entire data set.

2. In version < 3.1 we do not support non-CI classes. New release version 3.1 we will support any class/table that has IRE identifier rules. However, I just checked that IRE does not allow identifier rule for cmdb_rel_group table. Please contact CMDB IRE team if you have specific use cases for cmdb_rel_group table.

Thanks

Siyou

[Community Alums]

Hi Siyou,

Thanks for your response.

I referred the system property 'sn_int_studio.preview.size' where it is set 100 as default value.

when i try to change the value to any value between 100 to 10000 it works but if i use the application scope as IntegrationHUB

for my case i have created a separate application and created a IH ETL and i can see only 100 records in preview stage even if i change the value of the property to more than 100 its not showing morethan 100.

any thoughts on this ? 

if the property (value 100) is for only IntegrationHUB application scope then how it is same for my application also? and it is not working if i change the value and trigger the integration from my aplication scope.

if the preview value is 100 , means only 100 records will be loaded ?

if yes, then is that possible to have preview records morethan 100 for my custom application. 

please suggest a way how can we achieve this ?

Thank you

[Siyou Li]

Re: @Community Alums 

Hi Tamilselvan,

If preview value is 100, then IH-ETL will only load 100 records from your import set.

I'm not able to reproduce the problem you mentioned. Could you please reproduce that scenario in an instance and reach out to support. If support cannot resolve the issue, please directly assign that case to app-cmdb-application team, so we could further investigate your issue.

Thanks,

Siyou

[Community Alums]

@Siyou Li 

Thank you and i will contact support as i am able to reproduce the issue in PDI also.

Did you mean the preview limit is only to preview the number of records in IH ETL? and not dependent on count of records to be imported?

Example: if preview is set to 100 and source has 1000 records then 1000 records will be considered as import even if preview is set 100? please correct me if my understanding is wrong.

I tried in my PDI and the records loaded from import set is based on the limit set in the preview (sys properties) 

if i set preview limit 500 then 500 records only imported but source has 3000 records.

i will check and contact support.

[Siyou Li]

Re: @Community Alums 

Hi Tamilselvan,

Preview size property controls how many records are pulled into IH-ETL. For example, your raw import set have 10k rows of data. If you set preview size to 500, then only 500 rows will be pulled into IH-ETL from your raw 10k rows of data.

Thanks,

Siyou

[Community Alums]

Hi @Siyou Li ,

Got it . Thank you for the response.

i am able to load more data using the preview property.

[Aravin_C]

Hello @Siyou Li 

Is there any possibility to import/update more than 10K records in a single run?

Since the preview size is limited to 10K. Can we make sure to import/update the remaining records in the next schedule if my import contains more than 10K records?

Thanks!

Aravind.

[Siyou Li]

Hi @Aravin_C,

Yes, the next schedule will import the remaining data. And generally you should not run >10k data as a preview in IH-ETL.

IH-ETL test&rollback function is meant to have a preview of what's your data would actually be like in your DB after transform and import. It's not designed to do the actual heavy transform/insert work.

For example, you have 1M rows of raw data. After set up transform map using IH-ETL and go to the final preview step, by default IH-ETL will only actually transform and write 100 records to your DB. Then you make sure the 100 preview records look good in your DB, and rollback these 100 preview records from your DB by clicking "Rollback" button. Then you can set up a schedule associated to this transform map, and when this schedule runs, it will bypass IH-ETL and directly use backend engine RTE, which is meant for the actual heavy work, to transform all your 1M raw data using the transform map you defined.

Thanks,

Siyou

[Aravin_C]

Hello @Siyou Li 

I tried by setting the Preview Size Override to 10,000 in the first step of ETL creation and I was trying to import more than 50K records, I was not able to see the results dashboard after running the integration.