--- sourceDocument: Australia Workflow Data Fabric sourceDocumentLink: https://www.servicenow.com/docs/r/australia/integrate-applications Release : - australia ft:locale : - en-US ft:publication_title : - Australia Workflow Data Fabric ft:clusterId : - crint bundleId : - crint workflow : - Creator --- # Stream Producer # Stream Producer {#ariaid-title1} Release version: Australia Updated July 7, 2026 ![](https://www.servicenow.com/docs/portal-asset/ico-clock) 7 minutes to read Summarize ![AI sparkle icon](https://servicenow.com/docs/portal-asset/ai-sparkle-icon) Summarized using AI This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete. ## Summary of Stream Producer Stream Producer is a configuration-based ServiceNow feature that enables automatic streaming of table changes to Kafka topics using change data capture (CDC) technology. It eliminates the need for custom scripts or business rules by capturing inserts, updates, and deletes from specified tables and sending them as messages to Kafka in real time. This helps ServiceNow customers synchronize data efficiently with external systems while reducing manual effort and performance impact on their instances. Show full answer Show less ## Key Features * Automatic capture and streaming of table changes (insert, update, delete) using CDC without custom scripting. * Configuration interface to define monitored tables, fields included in payloads, filtering criteria, Kafka topics, message keys, and headers. * Supports payload serialization in JSON (human-readable) or Avro (schema-versioned and compressed). * Optional inclusion of metadata headers such as change action type, collection timestamp, and table name for enhanced downstream processing. * Domain separation compliance ensuring data isolation aligned with ServiceNow domain rules. * Role-based access control with distinct roles for creating, managing, and viewing Stream Producer configurations. * Automatic filtering of sensitive fields (passwords, encrypted fields) from payloads to protect sensitive data. * Supports filtering of records, multiple Stream Producers management, and tracking of runtime statistics and metrics. ## How It Works After creating and activating a Stream Producer configuration, the CDC framework detects database-level changes in the selected table. These changes are queued with assigned labels and polled by the Stream Producer job, which constructs and sends formatted messages to the configured Kafka topic continuously without manual intervention. ## Limitations * Does not support message chunking or interval-based capture; only real-time streaming. * Large messages exceeding Kafka's max size fail to send and are logged accordingly. * File attachments are not supported for streaming. ## Security and Permissions Stream Producer enforces multi-layered security controls including role-based access, table deny-listing to restrict monitored tables, and automatic sensitive field exclusion. Only explicitly allowed tables can be selected for monitoring to prevent unauthorized data access. ## Practical Usage for ServiceNow Customers * **Setup:** Define Stream Producer configurations via the UI to specify tables, events, fields, and Kafka topics. * **Activation:** Activate producers to begin real-time streaming; deactivate anytime to stop and discard queued messages. * **Monitoring:** Use provided statistics and metrics to monitor producer performance and CDC queue health to optimize throughput and scale. * **Integration:** Stream Producer enables seamless, reliable data integration workflows with external Kafka-based systems without custom development. Stream Producer enables you to automatically stream changes from ServiceNow tables to Kafka topics using change data capture (CDC), eliminating the need for custom scripts or business rules. Stream Producer provides a configuration-based alternative to writing scripts or business rules for exporting data. Instead of relying on error-prone manual solutions, you can define producers once and automatically stream table changes to external systems. Stream Producer uses change data capture (CDC) technology to capture inserts, updates, and deletes on the tables you select. The captured changes are formatted as messages and sent to a Kafka topic, enabling real-time data synchronization with external applications. ## Key benefits {#stream-producer__section_benefits} * Eliminate dependency on business rules and custom scripts for data export. * Reduce performance impact on the ServiceNow instance. * Set it up once and automatically keep external data in sync. * Streamline complex data integration workflows. ## How it works {#stream-producer__section_how_it_works} The following steps show the workflow for using Stream Producer. 1. Create a Stream Producer configuration and specify a table to monitor. 2. Define which change events to capture: insert, update, or delete. 3. Select which fields to include in the payload (or include all fields). 4. Configure an optional key and headers to accompany each message. 5. Specify the target Kafka topic. 6. Activate the producer. Once activated, the Stream Producer automatically captures matching table changes and sends them to your Kafka topic without requiring any additional configuration or manual intervention. ## Supported features {#stream-producer__section_supported_features} * Apply filters to capture only specific records. * Serialize payloads as plain text or in an Avro format. * Select individual fields or include all table fields. * Configure custom Kafka message keys with optional prefixes. * Add headers to messages with field values and metadata (including action type and collection timestamp). * Track producer statistics and activity metrics. * Manage multiple Stream Producers from a single console. ## Limitations {#stream-producer__section_limitations} The following features aren't supported. * Large messages. If the message is larger than the max message size for the target Kafka environment, the message will fail to send and will be logged as `RecordTooLargeException`. The change record is skipped, and processing continues with the next change. * Message chunking for payloads over a size limit. * Interval-based change capture (trigger only on scheduled intervals). * File attachments. ## Change data capture operation {#stream-producer__CDC} Stream Producer is built on a change data capture (CDC) framework, which provides a scalable, event-driven approach to detecting and exporting table changes. Rather than polling tables or relying on business rules, CDC captures changes at the database level and routes them through a message queue (CDC queue) for consumption by Stream Producer. Stream Producer configurations define what data to capture, how to filter it, and where to send it. Once activated, the Stream Producer job runs continuously, processing change events and sending them to Kafka topics in your configured serialization format (JSON or Avro). The following list provides an overview of how the CDC framework operates. 1. Change detection: When a record is inserted, updated, or deleted in a monitored table, the CDC framework captures the change at the database level. 2. CDC queue: The change is written to the Stream Connect CDC Change Queues \[`cdc_queue_stream_connect`\] table, with metadata including the table name, operation type (insert/update/delete), record sys_id, and timestamp. 3. Label assignment: Each Stream Producer configuration is assigned a label. Changes matching the producer's filter criteria are tagged with that label in the CDC queue. 4. Producer polling: The Stream Producer job polls the CDC queue periodically, retrieving changes for its label. 5. Payload construction: Stream Producer builds a message payload containing the specified fields and formats it according to your serialization choice. 6. Kafka dispatch: The formatted message is sent to the configured Kafka topic. ## Key components {#stream-producer__section_key_components} Stream Producer consists of the following components: Stream Producer configuration (UI form) : The user-facing interface where admins define producers. Contains sections for table information, filtering, and Kafka topic configuration. Managed by the Stream Producers \[`sys_sc_stream_producer`\] table. The Active field is read-only and automatically managed by the activation process. CDC Listener (background service) : Started when a Stream Producer is activated. Monitors the configured table for changes matching the filter criteria. Respects domain separation rules (if applicable). Stream Connect CDC Change Queues \[`cdc_queue_stream_connect`\] : Stores change events with labels identifying which Stream Producers should consume them. Index on labels field is not available; labeled lookups use full-table scans. Stream Producer Job : Scheduled batch job that polls the CDC queue, retrieves labeled changes, constructs payloads, and sends the payloads to Kafka. Runs every minute for a maximum of 2 minutes per execution. You can specify the maximum number of bytes written in a single Kafka transaction with the glide.ih.kafka.stream_producer.max_kafka_transaction_bytes property. The default value is 10MB. Stream Producer CDC Statistics \[`sys_sc_cdc_statistics`\] : Tracks metrics for each Stream Producer: messages and bytes produced, processing time, CDC depth (queue depth). Statistics are aggregated per job execution and updated periodically. Stream Producer \[`sys_sc_stream_producer`\] : Internal representation of an active producer. Created automatically when a Stream Producer is activated. Links the Stream Producer configuration to its runtime state and Kafka topic routing. ## Payload formats {#stream-producer__payload_formats} Stream Producer supports two serialization formats for message payloads. * JSON (human-readable, widely compatible) and * Avro (schema-versioned, compressed). You can select the serialization format when you [Create a Stream Producer](https://www.servicenow.com/docs/lk3Rgh~Vh2yTu2mnnBPhpw "Create a Stream Producer configuration to automatically stream table changes to a Kafka topic. You can specify which table to monitor, which change events to capture, which fields to include, and configure keys and headers for routing and tracking."). For more information about schema management for Avro, see [Schema management in Stream Connect](https://www.servicenow.com/docs/vXF5rE~GjaCLi6zZ1vveVA "Import and create schemas to send and receive messages in an Apache Avro format. Using an Avro format can reduce the size of the payload and simplify your integration to your local Kafka instance."). ## ISO format in JSON payloads {#stream-producer__section_usm_gww_gkc} Date and time fields in JSON payloads are serialized to ISO 8601 formats. * Date fields are serialized to JSON as 2026-04-15. * All fields containing a date and time are serialized to JSON as 2026-04-15T00:29:38.000Z when stored in UTC, or as 2026-04-15T00:29:38.000 when stored in wall-clock time. * All fields containing only a time are serialized to JSON as 00:29:38Z when stored in UTC, or as 00:29:38 when stored in wall-clock time. * All Boolean, integer, and float fields are serialized as Boolean, integer, and float in JSON. * Payload fields that aren't dates, times, Booleans, or numbers aren't changed and continue to use `toString()`. {#stream-producer__ul_ipb_sww_gkc} ## Headers and metadata {#stream-producer__section_headers_metadata} Each message sent to Kafka includes optional headers that carry metadata about the change event. Headers enhance message routing and processing in downstream systems. **Default headers:** When enabled, Stream Producer automatically adds the following headers to every message. * `action`: The type of change that triggered the message. Values are `insert`, `update`, or `delete`. * `collection_timestamp_ms`: The millisecond timestamp when the change was captured in the CDC queue. * `table_name`: The name of the source table being monitored. {#stream-producer__ul_qbg_f2b_gkc} **Custom headers:** You can configure additional headers using any fields from the source table. For reference or choice fields, headers include both the value and its display name, separated by a colon. For example, a reference field header might appear as `assigned_to: c39e3639876aba141444337e0ebb354f:Bob Person`. ## Domain separation support {#stream-producer__section_domain_separation} Stream Producer follows domain separation rules. When domain separation is enabled: * Stream Producer configurations are scoped to a domain and can only see records visible to that domain. * The CDC Advisor (change detection layer) filters changes by domain, ensuring domain isolation. * Domain inheritance follows the ServiceNow domain hierarchy (TOP → TOP/ACME, etc.). * A Stream Producer in a parent domain can stream changes from child domains; a Stream Producer in a child domain can't access parent domain data. {#stream-producer__ul_dgj_5p5_fkc} ## Security and permissions {#stream-producer__section_security_permissions} Stream Producer implements multi-layer security controls to protect table access and sensitive data, including: * Role-based access control (RBAC) at the configuration and runtime layers. * Table deny-listing to restrict which tables can be monitored. * Automatic sensitive field filtering from payloads and schemas. {#stream-producer__ul_dss_frb_gkc} Role-based access control: Stream Producer configurations require specific roles to access and manage producers. * `kafka_producer`: Required to create, activate, and deactivate Stream Producer configurations. * `stream_connect_admin`: Administrative role with full access to Stream Producer management and Stream Producer \[`sys_sc_stream_producer`\] table operations. * `stream_connect_viewer`: View-only role. Users with this role can't access the Stream Producer configuration UI. {#stream-producer__ul_wzg_j4b_gkc} Table deny-listing: Only tables explicitly allowed by the platform or administrator configuration are selectable when defining Stream Producer configurations. This prevents users from inadvertently configuring producers on restricted or unsupported tables. Some table categories are typically allowed by default: * Incident, Problem, Change Request, and other ITSM tables. * Custom application tables (with administrator approval). * Integration-related tables that support external data synchronization. {#stream-producer__ul_gqw_qrb_gkc} Note: Accessing the deny-list requires the `maint` role (only ServiceNow Support). Sensitive field filtering: Stream Producer automatically excludes sensitive field types from message payloads, Avro schemas, and UI displays to protect sensitive information, including: * `password`: User password fields. * `password2`: Secondary password fields or confirmation password fields. * `glide_encrypted`: Fields encrypted with the ServiceNow platform encryption mechanism. {#stream-producer__ul_xzg_j4b_gkc} ## Plugin {#stream-producer__section_amc_gp5_fkc} Stream Producer requires the ServiceNow Stream Producer \[`com.glide.hub.stream_connect.stream_producer`\] plugin. This plugin is activated when you activate the ServiceNow Stream Connect Installer \[com.glide.hub.stream_connect.installer\] plugin. * **[Create a Stream Producer](https://www.servicenow.com/docs/lk3Rgh~Vh2yTu2mnnBPhpw)** Create a Stream Producer configuration to automatically stream table changes to a Kafka topic. You can specify which table to monitor, which change events to capture, which fields to include, and configure keys and headers for routing and tracking. * **[Activate or deactivate a Stream Producer](https://www.servicenow.com/docs/jkrUK1cvlFrBRG1qFPx0RA)** Activate a Stream Producer configuration to begin capturing and streaming table changes to your Kafka topic. You can deactivate it at any time to stop streaming changes. When you deactivate a producer, any unprocessed messages in the CDC queue are discarded. * **[Monitor and optimize Stream Producer performance](https://www.servicenow.com/docs/P2OkgsKtHrBi0u2~VGyDzA)** Monitor Stream Producer performance metrics and change data capture (CDC) queue health to identify bottlenecks and optimize for your deployment scale and throughput requirements.