Overview
Robust Transform Engine (RTE) is an architectural enhancement to the traditional
transform map mechanism in ServiceNow. It decouples the transformation logic from
processing logic, offering better scalability, flexibility, and performance—especially for
asynchronous and batch processing of large volumes of data.
Why Robust Transform Engine (RTE)?
Traditional transform maps perform both transformation and processing in a single step, often leading to scalability issues in high-volume integrations. RTE separates these two stages and thereby,
- Reduces the application server thread exhaustion
- Improves node responsiveness
- Enables higher throughput of REST-based data imports
What is ETL in ServiceNow?
Extract-Transform-Load (ETL) is a new capability built on top of the RTE architecture, introduced during Orlando release and has evolved since then. It enables a more visual, flexible, and scalable way to define data integrations using entities and operations.
Components of ETL
1. Extract
Data is pulled from external sources via:
o Data sources (JDBC, REST, etc.)
o Staging tables
These represent the temporary holding area before transformation begins.
2. Transform
o Transformation logic is now defined in Transform Definitions
o Operations like concatenation, coalescing, formatting, value transformation are applied through a new UI
o Fields are transformed via entities and operations, replacing the need for scripting
o Data is processed in batches to improve performance and reliability
3. Load
o Transformed records are loaded into target tables
o ETL supports multi-target table transformations from a single source
o Reference fields are auto resolved; if missing, they can be configured to auto-create
RTE Architecture
Commonly Seen Customer Challenges
Heavy inbound REST POST request can overwhelm ServiceNow’s application nodes leading to exhausted thread pools and potentially impacting instance performance.
- Direct integrations to ServiceNow’s table API can lead to node exhaustion due to complex business logic is executed on insert/update actions
- App nodes are categorized into,
- UI Nodes: Handles user interactions
- Worker Nodes: Handles schedule jobs, event processing and API requests.
- High volume bursts of API traffic may impact User Experience by exhausting worker nodes.
This commonly seen challenge can be handled by having “Asynchronous Handling and Error Management” with Integration Hub ETL ( along with several data streaming options available for consideration).
- With Integration requests handled asynchronously, immediate response confirms the receipt and not the processing outcome
- Customers can verify the success or failure of the request at a later point by using a separate API call with the corresponding sys_id and reference ID provided
- Customers can define and implement custom error retry mechanisms based on their business needs
Best Practice Approach
- Implement a robust monitoring, alerting, and error logging mechanisms to ensure visibility into asynchronous processes.
- Clearly define and document error retry logic, thresholds for retries, and handling mechanisms for persistent failures.
- Consider using JDBC Step in DataStream Actions if the third-party tool is a Database which supports SQL commands.
- Address data consistency explicitly, ensuring that duplicate processing or data loss scenarios are mitigated, especially with Kafka-driven integrations.
- Provide flexible templates or patterns to help customers consistently and securely implement custom business logic within Script Actions or Script Includes.
Benefits Realized
- Decouples payload acceptance from intensive processing
- Reduces thread pool exhaustion risk
- Enhances ServiceNow instance’s resilience and User Experience during heavy loads
- Proven solution for large-scale integrations, accommodating high data volumes efficiently
Reference Links
- Robust Import Set Transformers
- Scheduled scripts and Events
- Custom Queue Event Handling
- Inbound REST API Rate Limiting
- Data Stream actions and pagination
