Robust Import Set Transformers
Summarize
Summary of Robust Import Set Transformers
Robust import set transformers provide a more flexible and efficient way to extract, transform, and load (ETL) data into one or more target tables in the ServiceNow AI Platform compared to traditional transform maps. They separate the transformation and processing functions, allowing batch processing and reducing the number of read operations when dealing with multiple target tables.
Show less
Key Features
- Separation of transform and processing: Unlike transform maps, robust import set transformers distinguish between data transformation and processing, enhancing flexibility.
- Batch processing: Records are processed in batches, improving performance especially when handling large datasets.
- Efficient multi-target loading: A single read operation can prepare data for multiple target tables, reducing redundant parsing.
- ETL functionality: Users can define ETL workflows by specifying entities (abstract tables), entity fields, mappings between entities, and operations to control data integration.
- Support for nested data structures: From the Paris release onward, ETL definitions support complex nested JSON/XML payloads within single columns, enabling integration of hierarchical data.
Practical Benefits for ServiceNow Customers
By using robust import set transformers, customers can streamline data import processes involving multiple target tables, improve performance with batch operations, and manage complex data structures more effectively. This approach reduces the overhead of creating multiple transform maps and optimizes data handling, making it ideal for advanced integrations and large-scale data imports.
Next Steps
- Create robust import set transformers to define how data is extracted, transformed, and loaded.
- Develop robust transform definitions to customize transformation logic.
- Build ETL definitions to manage entities, fields, mappings, and operations for your data integration workflows.
Use robust import set transformers instead of transform maps if you want to extract, transform, and load data to one or more target tables.
Robust import set transformers versus transform maps
Transform maps define the mapping from imported data stored in a staging table to a single target table in the ServiceNow AI Platform. Transform maps also insert data into target tables, performing both transform and processing functions. You can define multiple table mappings with multiple transform maps.
The Robust Transform Engine (RTE) and the robust import set transformer separate the transform and processing functions, providing a more flexible alternative to transform maps. The robust import set transformer allows you to extract data from a source table into an intermediary data structure. You can transform the data as desired and then load that data to one or more target tables. Records are processed as batches to enhance performance.
With transform maps, if you want data from the source table to go to three different target tables, you must create three separate transform maps. Each transform map parses the data separately, which results in three separate read operations. By contrast, the robust import set transformer requires only a single read operation to prepare the data for three target tables. The robust import set transformer is more efficient, especially when dealing with multiple target tables.
Extract Transform Load (ETL)
When you use the robust import set transformer, Extract Transform Load (ETL) functionality transfers imported data to target tables. You define how the data is extracted, transformed, and loaded to one or more target tables. You can use ETL definitions to do the following:
- Define entities (an abstraction similar to tables).
- Define entity fields (an abstraction similar to table fields).
- Define mapping between entities, and optionally designate whether a specific mapping should be ignored during data integration.
- Define entity field mappings.
- Define entity operations.
For an overview of ETL definitions, see Extract Transform Load (ETL) definition overview. For a step-by-step guide on how to create ETL definitions, see Create Extract Transform Load (ETL) definitions.
{
"records":[
{
"network":{
"location":"San Diego",
"computers":[
{
"id":"C100",
"os":"Mac",
"disks":[
{
"size":"200GB",
"type":"SSD"
},
{
"size":"1TB",
"type":"Magnetic"
},
{
"size":"1TB",
"type":"Magnetic"
}
]
},
{
"id":"C200",
"os":"Windows",
"disks":[
{
"size":"5TB",
"type":"Magnetic"
}
]
}
]
}
}
]
}