Microsoft Azure Event Hubs data input configuration fields

  • Release version: Xanadu
  • Updated August 1, 2024
  • 4 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Microsoft Azure Event Hubs data input configuration fields

    This guide details the configuration fields for setting up Microsoft Azure Event Hubs data inputs in ServiceNow, specifically for streaming log data using the Health Log Analytics application (version 26.0.17 or later). It explains the required parameters and options to connect Azure Event Hubs with your ServiceNow instance via MID Servers or MID Server clusters, enabling efficient log ingestion and failover support.

    Show full answer Show less

    Basic configuration

    • Name: Required field to name the data input.
    • Description: Optional description of the data input.
    • Execute on: Choose between a specific MID Server or a MID Server cluster for pulling log data. This feature supports failover and load balancing.
    • MID Server: Select a MID Server that supports basic authentication (mTLS is not supported). The default max number of data inputs per MID Server is 10, adjustable in MID Server properties. Log ingestion is enabled automatically if needed.
    • MID Server Cluster: Select a failover MID Server cluster with MID Servers supporting basic authentication. Only one MID Server in the cluster runs the data input at a time, with automatic failover. Clusters must have at least one MID Server with capacity below 10 data inputs.
    • Service instance: Bind the log data to an existing or newly created operational service instance. This is required for proper data association.

    Read-only information fields

    • Status: Current status of the data input.
    • Transport: Protocol used to stream logs (Microsoft Azure Event Hubs).
    • Sources count: Number of log sources created by this input.
    • Disabled since: Timestamp when the data input stopped or failed.
    • Last log time: Timestamp of the last received log event.
    • Error message: Displays streaming errors automatically when they occur.

    Query settings

    • From: Required start date/time for reading data; setting a past date may cause system congestion due to large data loads.
    • Max batch size: Maximum events processed per call (default example: 100).

    Transport settings

    • Event Hubs namespace: Required container for event hubs.
    • Event Hub name: Required selection of the event hub for log data.
    • Event Hub credentials: Required shared access authorization policy for authentication.
    • Consumer Group name: Required consumer group to use for reading events.

    Advanced configuration

    Various optional fields allow tuning of data input performance and behavior:

    • Restore point rate: Number of events read before saving a restore point (default 100).
    • Processor threads count: Number of processing threads (default 2).
    • Operation timeout: Timeout in seconds for Event Hubs operations (default 120).
    • Receive idle timeout: Timeout for receive operations (default 60 seconds).
    • Prefetch count: Number of events fetched in advance (default 500).
    • Default timezone: Timezone applied if logs lack timezone info (default GMT).
    • Sub sample drop/receive ratio: Settings to reduce event fetch volume by discarding events in batches (default -1 = disabled).
    • Max length in bytes: Maximum event size (default 32766 bytes).
    • Character encoding: Encoding format of data input (default UTF-8).
    • Sleep interval: Seconds to wait before re-querying after no events returned (default 60 seconds).
    • Polling interval: Seconds to wait before polling new events (default 0).
    • Drop if queue is full: Option to discard logs if MID Server load is high (default false).

    Practical implications for ServiceNow customers

    By properly configuring Microsoft Azure Event Hubs data inputs using these fields, customers can reliably stream large volumes of log data into ServiceNow for analysis and monitoring. Understanding authentication requirements, failover cluster setup, and tuning advanced parameters helps ensure efficient, resilient log ingestion aligned with infrastructure capacity and operational needs.

    Description of the fields on the Microsoft Azure Event Hubs data input configuration form.

    Basic configuration

    Field Description
    Name Name of the new data input. This field is required.
    Description Description of the data input.
    Execute on Option to determine whether to use a specific MID Server or a MID Server cluster.

    This feature is supported in the Health Log Analytics application, Version 26.0.17 - February 2023 and later, available from the ServiceNow Store.
    MID

    (Only when the Execute on field is set to Specific MID Server)

    MID Server to which log data from Microsoft Azure Event Hubs is pulled.
    Note:
    • You can select only MID Servers that support basic authentication. MID Servers that support mTLS are not listed.
    • The default maximum number of data inputs streaming logs to a single MID Server is 10. You can modify this number in the MID Server properties.
    • If log ingestion is not enabled for the selected MID Server, Health Log Analytics enables it automatically.
    This field is required.
    MID Server Cluster

    (Only when the Execute on field is set to Specific MID Server Cluster)

    The MID Server cluster to which the log data is pulled.

    The data input runs on a single MID Server in the cluster until that MID Server fails. The system then moves all the data input tasks to the next available MID Server in the cluster according to the configured order.

    This feature is supported in the Health Log Analytics application, Version 26.0.17 - February 2023 and later, available from the ServiceNow Store.

    Note:
    • Health Log Analytics supports only failover MID Server clusters. In these clusters, multiple MID Servers are grouped together for failover protection. When selecting a cluster from the data input form, the MID Server Clusters list displays only failover clusters.
    • The MID Server cluster must include only MID Servers that support basic authentication. mTLS is not supported for log ingestion.
    • Log ingestion must be enabled for each MID Server in the cluster. If log ingestion is not enabled for the active MID Server, Health Log Analytics enables it automatically.
    • The default maximum number of data inputs streaming logs to a single MID Server is 10. A cluster passes capacity validation if it contains at least one MID Server with fewer than 10 data inputs running on it, even when that MID Server is down.
    For more information about MID Server clusters, see Configure a MID Server cluster.

    This field is required.
    Service instance The service instance to which to bind the log data.
    Note:
    If no relevant service instance exists, Create an service instance and add CIs to it. Set the status of the new service instance to Operational.
    This field is required.
    The following fields show read-only information:
    Field Description
    Status Status of the data input.
    Transport Protocol used to stream the log data.

    This data input uses Microsoft Azure Event Hubs to stream log data to your instance.
    Sources count The number of log sources this data input has created.
    Disabled since The time when the data input stopped or failed.
    Last log time The time when the last log streamed in the data input.
    Error message The streaming error.

    This field is populated automatically. It displays only when a streaming error has occurred.
    Table 1. Query settings tab
    Field Description Example
    From Starting date and time for reading the data. Data older than this date and time is not read.
    Note:
    Setting this value to a past date might require the system to read large amounts of data, causing congestion.

    This field is required.

    		 Now -1 week
    Max batch size The maximum number of events passed to a single process call. 100
    Table 2. Transport tab
    Field Description
    Event Hubs namespace The scoping container for the event hub or hubs. This field is required.
    Event Hub name The event hub from which to fetch log data. This field is required.
    Event Hub credentials Shared access authorization policy with which to authenticate to the event hub. This field is required.
    Consumer Group name The Consumer Group to use.

    This field is required.

    Advanced configuration

    Table 3. Advanced configuration form
    Field Description Default value
    Restore point rate The number of events the data input can read before a restore point is saved. 100
    Processor threads count The total number of processor threads. 2
    Operation timeout The number of seconds to wait before timing out event hubs operations. 120
    Receive idle timeout The number of seconds to wait before timing out receive operations. 60
    Prefetch count The number of events received in advance of event hubs operations. 500
    Default timezone The default timezone if the log doesn't include timezone information. GMT
    Sub sample drop ratio The number of events to batch together, out of which one will be discarded. This setting is used to reduce the number of fetched events. -1
    Sub sample receive ratio The number of events to batch together, out of which all but one will be discarded. This setting is used to decrease the number of received events. -1
    Max length in bytes The maximum length, in bytes, of events. 32766
    Character encoding The character encoding for this data input. UTF-8
    Sleep interval The interval, in seconds, to wait before querying again after a query has returned no events. 60
    Polling interval The interval, in seconds, to wait before polling for new events. 0
    Drop if queue is full Option for selecting to discard logs if there is a load on the MID Server. False