MID-less log streaming via ITOM Gateway in Health Log Analytics
Summarize
Summary of MID-less log streaming via ITOM Gateway in Health Log Analytics
Health Log Analytics (HLA) enables direct log data ingestion from external sources through the ITOM Gateway without requiring a MID Server. This architecture is optimized for cloud-native log sources like Amazon Data Firehose, Cribl, and OpenTelemetry, and is essential for supporting high-volume log deployments. By streaming logs via the ITOM Gateway, data is forwarded efficiently to HLA's processing backend, allowing for improved throughput compared to traditional MID Server-based streaming.
Show less
How It Works
External log sources send data to the ITOM Gateway using gRPC. The ITOM Gateway then forwards this data to the Hermes Messaging Service, which acts as a broker delivering logs to the AI Engine for processing and analysis within HLA. This separation between log streaming and ingestion enhances performance and scalability.
Deployment Scenarios
- Standard Deployment: Suitable for typical log volumes. Requires enabling ITOM Gateway and Hermes Messaging Service on the ServiceNow instance, configuring a JSON Web Token (JWT) provider and token, and setting up log sources via Integrations Launchpad.
- High-Volume Deployment: For handling 30,000+ log events per second, infrastructure scaling is necessary. This includes resizing AI Engine and Elasticsearch nodes and coordinating a migration. ServiceNow Support must be contacted to initiate this process. Note that scaling requires a 6-hour change window with 2–6 hours of expected downtime affecting only HLA functions.
Supported Log Sources
The following log sources are currently supported for streaming via ITOM Gateway:
- AWS Firehose
- Cribl Stream
- OpenTelemetry Collector
Each source has a dedicated configuration tile in the Integrations Launchpad for easy setup.
Authentication
Log sources authenticate to HLA using JSON Web Tokens (JWT). A JWT provider and token must be configured on the ServiceNow instance before activating ITOM Gateway integrations.
Key Components
- ITOM Gateway: Receives log data from external sources via gRPC and forwards it to Hermes Messaging Service.
- Hermes Messaging Service: Routes data from ITOM Gateway to HLA’s AI Engine.
- AI Engine: Processes and analyzes the ingested log data.
- Integrations Launchpad: ServiceNow interface for configuring supported log source integrations.
Health Log Analytics (HLA) can receive log data from external sources directly through the ITOM Gateway, without routing data through a MID Server. This architecture supports cloud-native log sources such as Amazon Data Firehose, Cribl, and OpenTelemetry, and is required for high-volume HLA deployments.
How log streaming via ITOM Gateway works
External log sources send data to the ITOM Gateway over gRPC. The ITOM Gateway forwards the data to the Hermes Messaging Service, which delivers it to the AI Engine, the HLA processing back-end. This design separates log streaming from ingestion, enabling higher throughput than direct MID Server streaming.
Deployment scenarios
| Deployment type | Setup |
|---|---|
| Standard | For typical log volumes, enable ITOM Gateway and the Hermes Messaging Service on your instance, configure a JSON Web Token (JWT) provider and token, and set up your log source from Integrations Launchpad. For more information, see Configure a JSON Web Token (JWT) provider and token for Health Log Analytics and Set up log streaming via ITOM Gateway for Health Log Analytics. |
| High-volume | For deployments requiring 30,000 or more log events per second, you must scale the HLA infrastructure before enabling ITOM Gateway. This process involves resizing the AI Engine and Elasticsearch nodes and coordinating a cross-team migration. Contact ServiceNow Support to request infrastructure scaling. For details and the required information to provide, see Set up log streaming via ITOM Gateway for Health Log Analytics Note: Infrastructure scaling requires a 6-hour change window and involves expected downtime of 2–6 hours for HLA functions only. |
Supported log sources
- AWS Firehose
- Cribl Stream
- OpenTelemetry Collector
Authentication
Log sources authenticate to HLA using a JWT token. You must configure a JWT provider and generate a token on the ServiceNow instance before activating an ITOM Gateway integration.
Key components
| Component | Description |
|---|---|
| ITOM Gateway | Receives incoming log data from external sources over gRPC and forwards it to the Hermes Messaging Service. |
| Hermes Messaging Service | Message broker that routes data from ITOM Gateway to the HLA back-end. |
| AI Engine | HLA back-end component that processes and analyzes ingested log data. |
| Integrations Launchpad | ServiceNow interface for configuring log source integrations. Each supported log source has a dedicated tile. |