--- sourceDocument: Yokohama ServiceNow AI Platform Capabilities sourceDocumentLink: https://www.servicenow.com/docs/r/yokohama/servicenow-platform Release : - yokohama ft:locale : - en-US ft:publication_title : - Yokohama ServiceNow AI Platform Capabilities ft:clusterId : - platcap bundleId : - platcap workflow : - Platform --- # IntegrationHub ETL # IntegrationHub ETL {#ariaid-title1} * Release version: Yokohama * * Updated January 30, 2025 * * ![](https://www.servicenow.com/docs/portal-asset/ico-clock) 7 minutes to read Summarize ![AI sparkle icon](https://servicenow.com/docs/portal-asset/ai-sparkle-icon) Summarized using AI This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete. ## Summary of IntegrationHub ETL IntegrationHub ETL enables ServiceNow customers to create and manage ETL transform maps that integrate third-party data into the CMDB or supported non-CMDB tables while maintaining data integrity. It offers a simplified user interface with guided setup and supports end-to-end integration processes, including testing sample data before scheduling recurring data imports. IntegrationHub ETL leverages the Robust Transform Engine (RTE) and Identification and Reconciliation Engine (IRE) to process, transform, and reconcile data effectively. Show full answer Show less ## Key Features * **Integration Management:** Create or customize integrations using the IntegrationHub ETL store app with the `snintstudio` plugin. * **Support for Non-CMDB Tables:** From the Yokohama release, integration extends to certain non-CMDB tables supported by IRE, with some differences in handling class associations and relationships. * **Guided Setup and Testing:** Guided setup enforces task order and dependencies, while integration tests on sample data allow for validation and adjustment before full deployment. * **Data Processing Engines:** RTE transforms raw data from staging tables, and IRE applies reconciliation to avoid duplicates and maintain data quality. * **Read-Only Mode:** Allows examination of integrations with no incoming data for troubleshooting or cloning purposes without modification rights. * **Nested Data Handling:** Supports processing nested JSON payloads when configured properly, with rules for field naming and consistent use of arrays or objects. * **Improved Over Import Sets:** Incorporates IRE automatically and offers a guided, user-friendly interface with testing capabilities, unlike traditional Import Sets and transform maps. ## How It Works for Customers Customers can import raw source data (e.g., from SCCM), apply transformations to meet target CMDB or non-CMDB table requirements, and map attributes using ETL transform maps. Multiple connections and transform maps can be configured per CMDB application to handle different data feeds. After configuration and testing, recurring integrations can be scheduled. Customers benefit from enhanced data integrity through IRE and streamlined integration setup and management. ## Practical Considerations * Users require the `cmdbinstadmin` role to create or customize integrations. * CMDB applications must have a configured discovery source before use. * Non-CMDB table support requires understanding of IRE support and configuration. * Nested data payloads must follow strict naming conventions and consistent data structure usage. * Use the CMDB Integrations Dashboard to monitor integration progress, results, and errors. Use the IntegrationHub ETL store app to create and manage ETL transform maps, which integrate third-party data into the CMDB or into non-CMDB tables without compromising the integrity of data. IntegrationHub ETL provides a simplified user interface that guides you through the integration process end-to-end, including a test integration run of sample data. The IntegrationHub ETL (sn_int_studio) plugin provides the IntegrationHub ETL functionality. * Use the CMDB Integrations Dashboard to track progress, results, and errors associated with using custom integrations created in IntegrationHub ETL. The CMDB Integrations Dashboard is included in the [Integration Commons for CMDB](https://www.servicenow.com/docs/D5WmDWQQwWe_cmLk0Xi2sw "The Integration Commons for CMDB (sn_cmdb_int_util) store app contains the CMDB Integrations Dashboard and a set of Robust Transform Engine (RTE) transforms and script includes.") store app. * Watch the [IntegrationHub ETL \| Importing resources into the CMDB](https://www.youtube.com/watch?v=YvsCY0M7JLw) video for an introduction and walk through of the IntegrationHub ETL tool. {#integrationhub-etl__ul_jx1_sss_zvb} ## Request apps on the Store {#integrationhub-etl__section_k5h_xhq_zhb} Visit the [ServiceNow Store](https://store.servicenow.com/sn_appstore_store.do#!/store/home) website to view all the available apps and for information about submitting requests to the store. For cumulative release notes information for all released apps, see the [ServiceNow Store version history release notes](https://www.servicenow.com/docs/r/store-release-notes/sn-store-release-notes.html).{#integrationhub-etl__inline-send-to-store} ## Roles required {#integrationhub-etl__section_ywm_c33_slb} Users with the cmdb_inst_admin role can use IntegrationHub ETL to create integrations, or customize a pre-existing integration provided by ServiceNow or a vendor at the ServiceNow Store. A vendor can create a new integration and provide it as an application for anyone to use. ## Support for non-CMDB tables {#integrationhub-etl__section_ukf_bqc_wvb} Starting with the Yokohama release, IntegrationHub ETL supports the integration of third-party data into some non-CMDB tables. IntegrationHub ETL supports those non-CMDB tables that are supported by Identification and Reconciliation (IRE). For details about which non-CMDB tables are supported and any needed configuration, see [IRE support for non-CMDB tables](https://www.servicenow.com/docs/Lo~g5OIosVL47FATI2ccwg "Apply Identification and Reconciliation Engine (IRE) processes to supported non-CMDB tables to ensure data integrity and health of those tables."). Supported non-CMDB tables are available in IntegrationHub ETL when specifying classes, conditional classes, class associations, and reference sources in mapping definitions. However, there are some differences between using CMDB classes and non-CMDB tables in IntegrationHub ETL: * Specifying class associations isn't mandatory for non-CMDB tables. * Adding relationships doesn't apply to non-CMDB tables. * Class associations for a non-CMDB table is based on a reference field instead of a CMDB relationship. {#integrationhub-etl__ul_ps2_fnj_wvb} Note: Although the IntegrationHub ETL user interface and accompanying documentation references CMDB and CMDB elements, most of those references also apply to supported non-CMDB tables. ## Process {#integrationhub-etl__section_wcj_xl1_mkb} The two key components that IntegrationHub ETL uses for processing are: * [Robust Transform Engine (RTE)](https://www.servicenow.com/docs/access?context=robust-import-set-transformers&version=yokohama&pubname=yokohama-integrate-applications&ft:locale=en-US): Used to transform raw source data that is stored in staging tables, into the data that is mapped and integrated into the CMDB. RTE uses ETL transform maps that were created for the integration during data transformation. * [Identification and Reconciliation engine (IRE)](https://www.servicenow.com/docs/jH3RMGCMFFoQPqCe_BWP0A "IRE is an underlying key component in Identification and Reconciliation, providing a centralized framework to perform identification and reconciliation processes across different data sources. IRE uses identification rules, reconciliation rules, and IRE data source rules when processing incoming data before inserting that data to the CMDB."): Used as a centralized framework for identification and reconciliation processes across different data sources. IRE processes help maintain data integrity in the CMDB and in supported non-CMDB tables. {#integrationhub-etl__ul_uj5_hnc_wqb} IntegrationHub ETL uses RTE and IRE which work together to process and integrate data. Data is first imported from a data source, and is then stored in temporary staging tables in Import Sets systems. Using the data in the staging tables and the ETL transform map created by IntegrationHub ETL, RTE creates IRE payloads which are then processed by IRE. IRE applies reconciliation processes to avoid potential problems such as duplicate CIs, ensuring that the CMDB or non-CMDB tables remain healthy, and then integrates the resulting data. When you create an integration, you import source data, transform data if needed, and select target CMDB classes (or non-CMDB tables) and attributes to map the data to. Eventually, you run an integration test of the sample data, using your settings in the IntegrationHub ETL. You can then preview the integration test results and adjust any settings before scheduling recurring integration runs for large data sets. If you develop and test the ETL transform map on a development instance, then you can test and adjust the configuration before implementation on a production instance. For example, you can integrate data from SCCM (Microsoft System Center Configuration Manager). Refer to the community page [IntegrationHub-Extract Tranform Load (IH-ETL) is GA in ServiceNow store](https://community.servicenow.com/community?id=community_blog&sys_id=3efdcd51dbf4d09013b5fb243996190a) for an overview of IntegrationHub ETL, including its components and workflow. ## Guided Setup {#integrationhub-etl__section_bb3_dd4_15b} A guided setup organizes all the tasks in the correct order, tracks the completion of tasks, and enforces any task dependencies. Tasks that depend on the completion of other tasks, are enabled or disabled as you step through the tool and complete tasks. ## Read-only mode {#integrationhub-etl__section_yzp_d14_15b} When opening a Service Graph Connector in which IntegrationHub ETL isn't detecting any incoming data from the data source, the integration is available in read-only mode. In read-only mode you can access all the guided setup tasks on the ETL Transform Map Assistant page. You can examine all the settings and definitions in the integration even though it isn't populated with actual data. However, you can't make any updates to a read-only connection. Read-only mode is useful for studying an existing connection for the purpose of creating a new connection that is similar to the read-only connection. The read-only mode can also assist in troubleshooting issues with the connection. ## IntegrationHub ETL and Import Sets {#integrationhub-etl__section_r3m_vh4_rlb} Using IntegrationHub ETL and ETL transform maps has the following advantages over using Import Sets and transform maps: * Identification and Reconciliation Engine (IRE) processes are incorporated into the IntegrationHub ETL so all data is automatically processed by IRE as part of the integration. Using Import Sets and transform maps does not provide a simple way to apply IRE processes. * IntegrationHub ETL uses guided setup which provides guidance and a simple user interface for the entire process of integrating third-party data. * IntegrationHub ETL includes an integration test for a small data set using the new ETL transform map. This test lets you review the results and adjust configuration settings before scheduling recurring integrations. {#integrationhub-etl__ul_e4x_51l_jkb} ## Terms {#integrationhub-etl__section_ont_nbl_jkb} The following terms are associated with the IntegrationHub ETL: CMDB application : Name of the third-party vendor such as SCCM 2019. A CMDB application has two associated attributes: Name and Discovery Source. When creating a new integration, ensure to configure a discovery source for the CMDB application that you plan to use, before using the IntegrationHub ETL. Data source : The source feed, such as SCCM 7.0 Computer Identity, where the raw source data is imported from. If you use various REST endpoints for different types of data, then each REST endpoint is associated with its own data source and an ETL transform map. ETL transform map : The output generated by IntegrationHub ETL. You can integrate third-party data into the CMDB or into non-CMDB tables using an ETL transform map which is configured for the respective integration. Source data : Original, raw data that have been imported into IntegrationHub ETL. Source data can be used in its original form, or you can transform the data before mapping and integration. Transform : An operation, that you can apply to a specific data column to transform the data values. For example, to transform the format of the data values. Use transforms to standardize data formats and meet other system requirements. Transformed data : Some of the source data might not be compliant with the requirements of its target CMDB attributes and classes or non-CMDB tables. In those cases, you can apply various types of transforms to the source data, before mapping the data to the target CMDB classes and attributes or non-CMDB tables. Transforms, can for example convert data format, replace values, and concatenate values from multiple data columns. Each CMDB application can have multiple connections for retrieving raw data. Each connection that is used to retrieve a certain type of data, has its own pair of data source and an ETL transform map. Therefore, one CMDB application can have multiple ETL transform maps, and each of those ETL transform maps is associated with a single Data Source. For example: {#integrationhub-etl__table_pyk_df5_hkb__entry__3} | CMDB Application | ETL Transform Map | Data Source | |-|-|-| | SCCM | SCCM Computer Identify | /sccm/2019/comp | | SCCM | SCCM Disk | /sccm/2019/disk | | SCCM | SCCM Application | /sccm/2019/appl | [ ] {#integrationhub-etl__table_pyk_df5_hkb} ## Nested data payloads {#integrationhub-etl__section_zfm_hth_34b} To process nested data payloads, you must first ensure that the data source that is used for the integration, is set with the [Data in single column](https://www.servicenow.com/docs/access?context=r_DataSourceFileTypeFields&version=yokohama&pubname=yokohama-integrate-applications&ft:locale=en-US) option. With that setting, you can correctly represent nested data in a JSON payload which IntegrationHub ETL then processes as nested data, rather than as flat data. Sample of nested data: { "u_computer_fqdn": "computer2-fqdn", "u_computer_id": 2, "u_computer_ip": "computer2-ip", "u_computer_location": "PDX", "u_computer_mac": "computer2-mac", "u_computer_name": "nested-payload-computer2", "u_computer_os": "computer2-os", "interfaces": [ { "u_interface_ip": "computer2-eth1-ip", "u_interface_mac": "computer2-eth1-mac", "u_interface_name": "computer2-eth1", "ip": "" }, { "u_interface_ip": "computer2-eth2-ip", "u_interface_mac": "computer2-eth2-mac", "u_interface_name": "computer2-eth2", "ip": { "u_ip_address": "computer2-eth2-ip", "u_mac_address": "computer2-eth2-mac" } } ], "software": [ { "u_software_name": "computer2-software2", "u_software_version": "computer2-software2-1.0", "instance": { "u_software_instance_name": "computer2-software1-instance" } }, { "u_software_name": "computer2-software2", "u_software_version": "computer2-software2-2.0", "instance": { "u_software_instance_name": "computer2-software2-instance" } } ] }, You can view the layers of nested data in a separate panel in IntegrationHub ETL, apply transforms, map, and integrate that data into the CMDB. When creating a nested data JSON payload, the following restrictions apply: * Field names must start with a letter (between A-Z or a-z) or with '_', and must only contain letters (between A-Z or a-z), digits (0-9), or the '_' character. For example, a field name can't contain special characters such as \*, \[,\], #, $, spaces, and dot. * Field names can't be "temp" or "object", which are reserved for internal use. * Consistently throughout the payload, you must use an array or an object to represent data in a specific level, regardless of the number of items in the level. If you use an array for multiple items in one object, you must also use an array to represent a single item in other objects. {#integrationhub-etl__ul_bwr_ywz_j4b} For a demo about working with nested payload data, watch the [Integration Hub - ETL nested payload feature demo](https://youtu.be/jonkMkFiNaQ) video on the ServiceNow YouTube channel. **Related reference** * [Teams related list](https://www.servicenow.com/docs/rI5fgz5F~DpI1NbUMqKAzw "Related lists in CI records display additional components contained by that CI, such as disk drives on a server and the rules that control the behavior of a network router.")