Robust Import Set Transformers
Summarize
Summary of Robust Import Set Transformers
Robust import set transformers provide an advanced method for extracting, transforming, and loading (ETL) data into one or more target tables within the ServiceNow AI Platform. Unlike traditional transform maps, which map data from a staging table to a single target table and combine both transformation and processing functions, robust import set transformers separate these functions for greater flexibility and efficiency.
Show less
They leverage the Robust Transform Engine (RTE) to process data in batches, improving performance especially when multiple target tables are involved. This approach requires fewer read operations compared to multiple transform maps, making it ideal for complex data integration scenarios.
Key Features
- Separation of Transform and Processing: Allows extraction into intermediary data structures, transformation, and then loading into multiple target tables independently.
- Batch Processing: Enhances performance by processing records in batches rather than individually.
- Efficient Multi-Target Handling: Supports loading data into multiple target tables with a single read operation, reducing processing overhead.
- Extract Transform Load (ETL) Definitions: Enables defining entities and fields (similar to tables and table fields), mapping between entities, and specifying entity operations to control data flow.
- Support for Nested Data Structures: Since the Paris release, ETL definitions can handle nested JSON/XML payloads within a single column, allowing complex hierarchical data to be processed effectively.
Practical Use for ServiceNow Customers
ServiceNow customers can use robust import set transformers to streamline and optimize data integration tasks, especially when dealing with complex data sources and multiple target tables. By adopting this approach, customers benefit from improved performance, reduced redundancy, and greater control over data transformation logic.
Creating robust import set transformers involves defining ETL configurations that abstract tables and fields into entities, mapping these entities, and specifying how data should be processed and loaded. This modular design facilitates maintainability and scalability of your data import processes.
Next Steps
- Create robust import set transformers to define your ETL workflows.
- Develop robust transform definitions to manage data transformation logic.
- Establish Extract Transform Load (ETL) definitions to map and process data between entities.
Use robust import set transformers instead of transform maps if you want to extract, transform, and load data to one or more target tables.
Robust import set transformers versus transform maps
Transform maps define the mapping from imported data stored in a staging table to a single target table in the ServiceNow AI Platform. Transform maps also insert data into target tables, performing both transform and processing functions. You can define multiple table mappings with multiple transform maps.
The Robust Transform Engine (RTE) and the robust import set transformer separate the transform and processing functions, providing a more flexible alternative to transform maps. The robust import set transformer allows you to extract data from a source table into an intermediary data structure. You can transform the data as desired and then load that data to one or more target tables. Records are processed as batches to enhance performance.
With transform maps, if you want data from the source table to go to three different target tables, you must create three separate transform maps. Each transform map parses the data separately, which results in three separate read operations. By contrast, the robust import set transformer requires only a single read operation to prepare the data for three target tables. The robust import set transformer is more efficient, especially when dealing with multiple target tables.
Extract Transform Load (ETL)
When you use the robust import set transformer, Extract Transform Load (ETL) functionality transfers imported data to target tables. You define how the data is extracted, transformed, and loaded to one or more target tables. You can use ETL definitions to do the following:
- Define entities (an abstraction similar to tables).
- Define entity fields (an abstraction similar to table fields).
- Define mapping between entities, and optionally designate whether a specific mapping should be ignored during data integration.
- Define entity field mappings.
- Define entity operations.
For an overview of ETL definitions, see Extract Transform Load (ETL) definition overview. For a step-by-step guide on how to create ETL definitions, see Create Extract Transform Load (ETL) definitions.
{
"records":[
{
"network":{
"location":"San Diego",
"computers":[
{
"id":"C100",
"os":"Mac",
"disks":[
{
"size":"200GB",
"type":"SSD"
},
{
"size":"1TB",
"type":"Magnetic"
},
{
"size":"1TB",
"type":"Magnetic"
}
]
},
{
"id":"C200",
"os":"Windows",
"disks":[
{
"size":"5TB",
"type":"Magnetic"
}
]
}
]
}
}
]
}