Viewing Kafka subscriptions and statistics

  • Release version: Zurich
  • Updated July 31, 2025
  • 4 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Viewing Kafka subscriptions and statistics

    This feature allows ServiceNow customers to monitor Kafka consumer subscriptions and their performance metrics in detail. It provides insights into the number of records added, processed, and remaining in Kafka topics, helping optimize message consumption and troubleshoot issues effectively.

    Show full answer Show less

    How to Access

    Navigate to IntegrationHub > Stream Connect > Subscriptions. Select the info icon next to the desired subscription and open the record to view detailed information.

    Key Sections in a Subscription Record

    • Kafka subscription: Displays configuration and state details such as consumer ID, topic, auto position policy, serialization format, concurrency settings, and error indicators.
    • Kafka subscription partition groups: Shows partition group ownership status, update timestamps, partition ranges, input rates, processing rates, and queue depths over multiple time intervals.
    • Stream Connect logs: Provides log entries related to the consumer including timestamps, log levels, messages, and links to alerts or subscriptions. Detailed logging can be enabled temporarily for troubleshooting.
    • Kafka consumer statistics: Presents interval-based statistics like produced and consumed messages, consumed bytes, offsets, lag messages, and partition group references. Logging interval and behavior can be configured via system properties.

    Performance Metrics

    • Metrics are reported over three time frames: last 5 minutes, last hour, and last 24 hours.
    • Includes topic input rate, topic queue depth, consumer processing rate, and estimated time to process the current queue.

    Why This Matters

    By reviewing Kafka subscriptions and their associated metrics, ServiceNow customers can:

    • Understand consumer performance and workload distribution.
    • Detect and resolve processing bottlenecks or errors.
    • Optimize consumer configurations such as concurrency and batch sizes.
    • Monitor partition ownership and message processing lifecycle.

    Configuration and Logs

    Adjustable system properties allow control over logging detail, interval durations for statistics, and behaviors when no messages are produced or consumed. This flexibility helps balance monitoring needs with system performance and storage.

    View detailed information about a Kafka consumer and its performance, including the number of records added to the topic, the number of records processed, and the number of records remaining to be processed.

    A subscription is a record associated with a consumer. It stores configuration and performance details about the consumer. A subscription is created when a Kafka stream is activated.

    Each subscription page includes the following sections.
    • Kafka subscription
    • Metrics
    • Kafka subscription partition groups
    • Stream Connect logs
    • Kafka consumer statistics

    To view a subscription record, navigate to IntegrationHub > Stream Connect > Subscriptions. Select the info icon (info icon) for the subscription you want to view, then select Open Record.

    Figure 1. Sample subscription page for a Kafka consumer
    The Kafka subscription page with subscription details, metrics, and information on partition groups, consumer logs, and consumer statistics.

    Kafka subscription

    The Kafka subscription section shows configuration details for the consumer and subscription.
    Field Description
    Number Number of the subscription record.
    State State of the consumer. Options are:
    • ACTIVE
    • REFRESHING
    • STOPPING
    Topic Topic the consumer is processing messages from.
    Consumer ID Unique identifier for the consumer.
    Kafka Auto Position Policy Specifies where the consumer starts reading messages from the topic. Options are:
    • earliest: The consumer starts reading from the earliest available message in the topic.
    • latest: The consumer reads only the messages delivered to the topic after the consumer started.
    Serialization format Serialization format for the message. Options are:
    • Plain Text
    • Encoded
    Has Error Indicates if the subscription has any errors.
    Max Concurrency Maximum number of parallel processors used to consume messages and the number of partition groups.
    Partition group timeout Maximum time, in milliseconds, allocated to each partition group in the subscription.
    Kafka partitions Number of partitions in the topic.
    Max Batch Size Maximum number of messages processed per run. Options are:
    • automatically optimized: The system chooses the number of messages.
    • a number: The user manually selected a number.

    Metrics

    Each metric is measured over three time intervals: the previous five minutes, the previous hour, and the previous 24 hours.
    • Topic Input Rate: The average number of records added to the topic per second.
    • Topic Queue Depth: The average number of records remaining to be processed in the topic.
    • Consumer Processing Rate: The average number of records processed per second. The Estimated time to process the current queue shows the time required to process the messages currently in the queue. It's calculated based on the queue depth and the consumer processing rate. The time estimate is formatted as HH:MM:SS.

    Related Links

    View the record for the consumer that is processing the messages by selecting Open consumer processing record.

    Kafka subscription partition groups

    The Kafka subscription partition groups tab shows the following details.
    Field Description
    Number Number for the partition group.
    Owner of this partition group Ownership state of the partition group. Options are:
    • NONE: The partition group isn't owned by any Kafka consumer jobs. It's available to pick up and process.
    • PROCESSING: The partition group is owned by a Kafka consumer job and is processing messages.
    • LOCKED: The partition group isn’t owned by a Kafka consumer job and isn’t available to pick up.

    When a Kafka consumer job is activated, it checks for the earliest updated record with an owner of NONE and picks one.

    Then the consumer job changes the owner to PROCESSING and processes it for 10 seconds.

    When it's done processing, the state is changed back to NONE, so it can be picked up and processed again later.

    If the consumer job processes all the messages, it sets the owner to LOCKED for 10 seconds, after which it's set to NONE again.
    Updated Date and time the partition group was last updated.
    When is the ownership lost Date and time that ownership is lost.
    First partition Number of the first partition for the partition group.
    Last partition (inclusive) Number of the last partition for the partition group.
    Input rate

    Given for the last five minutes, last hour, and last 24 hours.

    		 Average number of records added to the partition group per second.
    Consumer processing

    Given for the last five minutes, last hour, and last 24 hours.

    		 Average number of messages consumed from the partition group per second.
    Queue depth

    Given for the last five minutes, last hour, and last 24 hours.

    		 Average number of records remaining to be processed in the partition group.

    Stream Connect logs

    The Stream Connect Logs tab displays the log entries for the associated consumer. Each log entry shows when it was created; its level, message, and source; and links to any related Alert or Subscription records. Stream Connect logs are stored in the Stream Connect Logs [sys_consumer_log] table.

    You can enable more detailed logging in the Stream Connect logs by adding the system property glide.ih.kafka.stream_connect.debug and setting it to true. To avoid filling up the logs, this property is automatically disabled after 24 hours. If you need detailed logging for longer than 24 hours, you can re-enable the property manually.

    Kafka consumer statistics

    The Kafka Consumer Statistics tab shows the following details.
    Field Description
    Created Date and time the consumer statistics record was created.
    Interval Duration for the consumer statistics record, given in seconds.

    You can change the length of the interval with the glide.ih.kafka.consumer.log_interval property. The default value is 60 seconds.

    You can specify whether to log messages to the consumer log when there are no produced or consumed Kafka messages during the interval with the glide.ih.kafka.consumer.log_empty_stats property. The default value is true.
    Produced messages Number of messages produced to the topic during the interval time.
    Consumed messages Number of messages processed by the consumer during the interval time.
    Consumed bytes Number of bytes processed by the consumer during the interval time.
    Offset Sum of all the offsets for the partition group.
    Lag messages Difference between the end offset and the current offset.
    End offset Sum of all the end offsets for the partition group.
    Partition group Record for the partition group.