Previous Step
How to set up an IH-ETL transform map
In this step, input the basic details for your ETL transform map. Below is an example configuration:
1. To start configuring your mapping, click “Import Source Data and Provide Basic Details”.
2. Specify basic information for your ETL transform map.
In IH-ETL V3.2, we have added below two functions
The "Previous Size Override" number will determine how many import set rows to pull into IH-ETL for preview for current mapping. If no value is provided, it will fall back to use the value in system property "sn_int_studio_preview.size", by default the value is 100.
For nested payload, in order to improve loading performance, we by default disabled loading complete schema fields. IH-ETL will only preview schema fields that are within the rows specified in "Preview Size Overrider", and if this value is not provided, it will use the value in "sn_int_studio.preview.size" property. Unless user finds that the required fields are missing in field mapping step, user should not enable this option for best loading performance. Note that for newly created mapping, IH-ETL will always load complete schema.
3. If you have already created a CMDB Application, select it from the drop-down. If you have not, leave it set at “Add new…” and specify the name.
This is the name of your application, not the name of this data source.
4. Select your discovery source
This should be unique to above CMDB application.
5. In the “Name” field, specify the name of this ETL transform map
6. In “Data Source”, select the data source prepared in previous steps
7. After you select the data source, select an import set if available. If not, select “Auto-pull a new import set” and IH-ETL will pull a new import set for you
8. Click “Save” and then “Mark as Complete”
Tips and caveats
- Discovery source is used by IRE reconciliation. This value will be stored in “discovery_source” field for all records inserted or updated by this ETL map
- If there is an existing Import Set, it is automatically selected in “Sample Import Set” dropdown. Alternately, if there is no existing import set, a new import set will be auto pulled after save
- “Test Load 20 Records” type import set is not supported by IH-ETL
- By default, IH-ETL will only pull 100 rows from data source. Use property “sn_int_studio.preview.size” to adjust the number (max number is 10k)
How to Change the Data Source for an Existing ETL Transform Map
Overview
In IntegrationHub ETL (IH-ETL), the Data Source field in the Basic Details step defines where raw source data is imported from for your ETL Transform Map. You may need to change this data source after a mapping has already been created — for example, if you've set up a new data source or need to point the mapping to a different import source.
This article explains how to change the data source and the conditions that must be met for the change to succeed.
Steps to Change the Data Source
-
Open the ETL Transform Map you want to modify.
-
Navigate to Step 1: Basic Details.
-
Locate the Data Source field. It is a reference picker that searches the
sys_data_sourcetable. -
Clear or type over the current data source value and select the new data source from the search results, or click the reference lookup icon to browse available data sources.
-
Once you select a new data source:
-
The Sample Import Set dropdown will be cleared and repopulated with import sets associated with the new data source.
-
If the new data source has a different import set table than the previous one, an informational notification will appear alerting you of the difference.
-
-
Click Save to persist the change.
Note: After changing the data source, you may need to re-pull a sample import set. If the "Auto-pull a new import set" toggle is enabled, a new import set will be pulled automatically when you mark the step as complete.
Restrictions and Conditions
The following conditions are enforced when changing the data source. If any of these checks fail, the save will be rejected and an error message will be displayed.
1. Application Scope Must Match
The new data source must belong to the same ServiceNow application scope as the one you are currently working in. An ETL Transform Map must be created and maintained within the same application scope as its data source.
Error if violated:
"The data source, '{name}', is in the {scope A} application, but {scope B} is the current application. An ETL Map needs to be created from the same application scope as its data source."
2. Import Set Table Must Not Be Used by Another Active ETL Map
The new data source's import set table (import_set_table_name) must not already be associated with another active ETL Transform Map (sys_robust_import_set_transformer). Each import set table can only be linked to one active ETL Map at a time.
This check excludes the current ETL Map itself, so re-selecting the same data source or another data source that shares the same import set table currently used by this map is allowed.
Error if violated:
"Data source table already associated with '{ETL Map name}' ETL Map"
3. Import Set Table Must Not Be Used by an Active Table Transform Map
The new data source's import set table must not be associated with any active Table Transform Map (sys_transform_map). ETL Maps and Table Transform Maps cannot share the same import set table.
Error if violated:
"Data source table already associated with '{Transform Map name}' Table Transform Map"
Summary Table
| Condition | What Is Checked | Outcome if Failed |
|---|---|---|
| Same application scope | Data source scope matches the current working scope | Error — save is rejected |
| No duplicate ETL Map | Import set table is not used by another active ETL Map | Error — save is rejected |
| No Table Transform Map conflict | Import set table is not used by any active Table Transform Map | Error — save is rejected |
Additional Notes
-
Import set table difference warning: If the new data source uses a different import set table than the previously saved data source, an informational notification will appear. This is a non-blocking warning to make you aware that the underlying table structure may differ, which could affect your existing field mappings.
-
Existing field mappings: Changing the data source does not automatically remove or update any entity or attribute mappings you have already configured. If the new data source has a different schema (different columns), you may need to revisit the Prepare Source Data and Map to CMDB steps to update your mappings accordingly.
-
Preview data reset: When the data source or import set changes, the preview data is reset and will be re-fetched the next time you open a preview.
Additional Resources
Next Step
IH-ETL Guided Setup Step 2 - Preview and Prepare Data
