Viewing Kafka subscriptions and statistics

View detailed information about a Kafka consumer and its performance, including the number of records added to the topic, the number of records processed, and the number of records remaining to be processed.

A subscription is a record associated with a consumer. It stores configuration and performance details about the consumer. A subscription is created when a Kafka stream is activated.

Each subscription page includes the following sections.

• Kafka subscription
• Metrics
• Kafka subscription partition groups
• Consumer logs
• Kafka consumer statistics

To view a subscription record, navigate to IntegrationHub > Stream Connect > Subscriptions. Select the info icon () for the subscription you want to view, then select Open Record.

Figure 1. Sample subscription page for a Kafka consumer

Kafka subscription

The Kafka subscription section shows configuration details for the consumer and subscription.

Field	Description
Number	Number of the subscription record.
State	State of the consumer. Options are: ACTIVE REFRESHING STOPPING
Topic	Topic the consumer is processing messages from.
Consumer ID	Unique identifier for the consumer.
Kafka Auto Position Policy	Specifies where the consumer starts reading messages from the topic. Options are: earliest: The consumer starts reading from the earliest available message in the topic. latest: The consumer reads only the messages delivered to the topic after the consumer started.
Has Error	Indicates if the subscription has any errors.
Max Concurrency	Maximum number of parallel processors used to consume messages and the number of partition groups.
Partition group timeout	Maximum time, in milliseconds, allocated to each partition group in the subscription.
Kafka partitions	Number of partitions in the topic.
Max Batch Size	Maximum number of messages processed per run. Options are: automatically optimized: The system chooses the number of messages. a number: The user manually selected a number.

Metrics

Each metric is measured over three time intervals: the previous five minutes, the previous hour, and the previous 24 hours.

• Topic Input Rate: The average number of records added to the topic per second.
• Consumer Processing Rate: The average number of records processed per second.
• Topic Queue Depth: The average number of records remaining to be processed in the topic.

Kafka subscription partition groups

The Kafka subscription partition groups tab shows the following details.

Field	Description
Number	Number for the partition group.
Owner of this partition group	Ownership state of the partition group. Options are: NONE: The partition group isn't owned by any Kafka consumer jobs. It's available to pick up and process. PROCESSING: The partition group is owned by a Kafka consumer job and is processing messages. LOCKED: The partition group isn’t owned by a Kafka consumer job and isn’t available to pick up. When a Kafka consumer job is activated, it checks for the earliest updated record with an owner of NONE and picks one. Then the consumer job changes the owner to PROCESSING and processes it for 10 seconds. When it's done processing, the state is changed back to NONE, so it can be picked up and processed again later. If the consumer job processes all the messages, it sets the owner to LOCKED for 10 seconds, after which it's set to NONE again.
Updated	Date and time the partition group was last updated.
When is the ownership lost	Date and time that ownership is lost.
First partition	Number of the first partition for the partition group.
Last partition (inclusive)	Number of the last partition for the partition group.
Input rate Given for the last five minutes, last hour, and last 24 hours.	Average number of records added to the partition group per second.
Consumer processing Given for the last five minutes, last hour, and last 24 hours.	Average number of messages consumed from the partition group per second.
Queue depth Given for the last five minutes, last hour, and last 24 hours.	Average number of records remaining to be processed in the partition group.

Consumer logs

The Consumer Logs tab displays the log entries for the associated consumer. It shows when each entry was created, its log level, the message, and the source. Consumer logs are stored in the Consumer Logs [sys_consumer_log] table.

You can increase the number of log messages added to the consumer logs by enabling the glide.ih.kafka.debug.consume property.

Kafka consumer statistics

The Kafka Consumer Statistics tab shows the following details.

Field	Description
Created	Date and time the consumer statistics record was created.
Interval	Duration for the consumer statistics record, given in seconds. You can change the length of the interval with the glide.ih.kafka.consumer.log_interval property. The default value is 60 seconds. You can specify whether to log messages to the consumer log when there are no produced or consumed Kafka messages during the interval with the glide.ih.kafka.consumer.log_empty_stats property. The default value is true.
Produced messages	Number of messages produced to the topic during the interval time.
Consumed messages	Number of messages processed by the consumer during the interval time.
Consumed bytes	Number of bytes processed by the consumer during the interval time.
Offset	Sum of all the offsets for the partition group.
Lag messages	Difference between the end offset and the current offset.
End offset	Sum of all the end offsets for the partition group.
Partition group	Record for the partition group.