Stream Connect alerting
Summarize
Summary of Stream Connect alerting
Stream Connect alerting in ServiceNow provides monitoring and alert notifications for Stream Connect integrations by detecting events across multiple components. Alerts are generated through active and scheduled monitoring methods, helping you identify issues like producer or consumer failures and performance thresholds being exceeded. When alerts occur, they are logged and notifications are sent via email, SMS, or the ServiceNow mobile app.
Show less
Alert Types and Monitoring
- Event-type alerts: Real-time alerts triggered by errors in Stream Connect components, such as write or read failures. These use active monitoring to detect and respond immediately.
- Threshold alerts: Generated when metrics like message lag or processing time cross configured thresholds. These are detected via scheduled monitoring jobs running scripted checks.
Alert properties, including severity levels and notification timing, are configurable through the Stream Connect Alerting properties UI. Access requires the Kafka Admin role.
Alert Management and Data Retention
- Alerts are recorded in the Stream Connect Alerts [sysscalerts] table, with active alerts updated and inactive alerts deactivated after configurable time intervals.
- Alert logs are maintained in the Stream Connect Log [sysconsumerlog] and linked to alert records, with a 30-day retention period managed by table rotation schedules.
- Alerts and logs are accessible via the Integration Hub interface, enabling detailed review of alert number, severity, affected entities, and timestamps.
Notifications and Roles
- Notifications include detailed alert information and are controlled to avoid redundant alerts by using configurable time intervals and severity escalation rules.
- Three severity levels are supported for notifications: INFO, WARNING, and CRITICAL, with user preferences and group memberships determining notification recipients.
- Only members of the Stream Connect Alert Notifications group receive notifications; this group has the Stream Connect Alert Viewer role for read-only alert access.
- Roles required: Kafka Admin for full alert configuration and log access, and Stream Connect Alert Viewer for read-only alert viewing.
Security and Domain Separation
- Access controls (ACLs) restrict alert table visibility and editing, ensuring only authorized users can view or modify alerts.
- Domain separation allows alert properties to be overridden per domain, with alert records associated to domains based on the relevant consumer or topic.
- The alerting properties UI sets global configurations; domain-specific overrides require manual configuration as no UI is provided.
Plugin and Configuration
- The Stream Connect Alerting plugin is required and is automatically activated with the Stream Connect Installer plugin.
- Alert properties define how alerts are generated, managed, and notified, enabling administrators to tailor alerting behavior to organizational needs.
Receive alerts and alert notifications for Stream Connect integrations.
Stream Connect uses both active and scheduled monitoring to detect events across multiple Stream Connect components. If an issue is detected, the system creates an alert, logs a message to the Stream Connect Log, and sends out an alert notification.
Alerts
-
Event-type alerts (or real-time alerts) are created when there's an error in a Stream Connect component. For example, when a producer is unable to write to a topic, or a consumer is unable to read from a topic. These alerts are built into to the code and use the alerting properties to determine the severity level and when to send a notification. Event-type alerts are detected through active monitoring which uses various mechanisms to detect events across multiple Stream Connect components. The system caches the configuration and detects the events when they occur.
-
Threshold alerts are created when a value transitions from below the threshold to at or above the threshold. For example, when the lag on a topic goes beyond a certain number of messages, or the processing time of the messages in a topic is above a specified value. When a threshold alert decreases from at or above a threshold to below it, the active flag on the alert is set to false. Threshold alerts are detected through scheduled monitoring, where a scheduled job runs scripted checks on Stream Connect integrations. An alert is generated for thresholds that meet the alert configuration specified by the alerting properties.
For a list and description of the alerts, or to configure alerting properties, see Stream Connect Alerting properties. You must have the Kafka Admin role to view or modify alerting properties.
Alert table
When an alert is triggered for the first time, an Alert [sys_sc_alert] record is added to the Stream Connect Alerts [sys_sc_alerts] table. If the same alert is triggered again, and the original alert record is still active, the record’s Last Detections [last_detection] field is updated. If the original alert record is not active, a new record is added to the alerts table. An alert record is deactivated when the alert has not been triggered for a specified amount of time. This time interval varies depending on the type of alert and can be modified in the Stream Connect Alerting properties UI.
- An alert's Number, which references the alert record. Each alert record contains detailed information about the alert, including the name of the affected component, the severity level, and the timestamp.
- An alert's Affected Entity, which references the target for an alert, for example, a Kafka Stream or a Kafka Flow Trigger.
Alerts are available for 30 days. A table cleaner job handles the retention period for the Stream Connect alerts table. Active alerts are deactivated if they don’t reoccur within a specified duration. The duration is configured per alert type in the Stream Connect Alerting properties.
Alert logs
Log messages for Stream Connect alerts are written to the Stream Connect Log [sys_consumer_log]. This log has an Alert [sc_alert] field that references the associated Stream Connect Alert record.
Logs are available for 30 days. The log retention period is determined by the table rotation schedule for the Stream Connect Log.
Notifications
Receive alert notifications via email, SMS, or the ServiceNow® mobile app. Notifications contain detailed alert information, including the alert number, level, and a description.
When an alert is triggered, a notification is also triggered, unless a notification for that same alert was recently triggered. Each alert has a time interval property that specifies the amount of time, in seconds, required between alert detections to have the detections count as two separate alerts. A notification is only triggered if the time since the last notification is greater than that time interval or, in the case of threshold-based alerts, if the severity for the alert has increased since the last detection. This helps ensure that you don't receive multiple rapid notifications for the same alert. The time interval property for each alert can be modified from the Stream Connect Alerting properties UI.
- Stream Connect Alert - INFO
- Stream Connect Alert - WARNING
- Stream Connect Alert - CRITICAL
Notifications are sent to users who are members of the Stream Connect Alert Notifications group. The Stream Connect Alert Notifications group members inherit the Stream Connect Alert Viewer role so that everyone who receives notifications can use the link in the notification to see the alert record.
Roles
- The Kafka Admin [kafka_admin] role can view Stream Connect alert records and logs, configure Stream Connect Alerting properties, and configure alert notification settings.
- The Stream Connect Alert Viewer [stream_connect_alert_viewer] role has read-only access to Stream Connect alert records and logs.
ACLs and domain separation
ACLs on the Stream Connect Alerts table enable a user with the Stream Connect Alert Viewer role to view the alerts table and the reports on that table. Users without that role can’t read or report on that table. Only users with the Kafka Admin role can insert and update records on that table. ACLs make the Stream Connect Alerting properties page available only to Kafka Admins.
On a domain-separated instance, the properties that govern alerts can be overridden on a per-domain basis. See Domain separation application properties for details.
The domain used to look up properties is the domain of the topic or consumer. The Stream Connect Alerting properties UI for setting properties sets the global properties, not domain-specific overrides. There's no UI for setting domain-specific overrides. The sys_domain of the Stream Connect Alerts table should be the same as the domain of the consumer that triggered it, for consumer alerts, or the domain for the sys_kafka_topic record for producer alerts. For alerts where a consumer or topic isn't known, the sys_sc_alert domain is set to global.
Plugin
Stream Connect alerting requires the ServiceNow Stream Connect Alerting [com.glide.hub.stream_connect.alerting] plugin. This plugin is automatically activated when the ServiceNow Stream Connect Installer[com.glide.hub.stream_connect.installer] plugin is activated.