GCP PubSub data input configuration fields

  • Release version: Xanadu
  • Updated August 1, 2024
  • 4 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of GCP PubSub data input configuration fields

    This document describes the configuration fields for setting up a Google Cloud Platform (GCP) Pub/Sub data input within ServiceNow's Health Log Analytics. It guides customers on how to properly configure the data input form to stream log data efficiently and securely using MID Servers or MID Server clusters.

    Show full answer Show less

    Basic Configuration

    • Name: Required field to name the new data input.
    • Port: Required field to select a unique MID Server port within the allowed range. Ensure the port is opened by your security team.
    • Execute on: Choose to run on a specific MID Server or a MID Server cluster. This option is supported from Health Log Analytics version 26.0.17 onward.
    • MID Server: When running on a specific MID Server, select one with basic authentication and log ingestion enabled. MTLS-supported MID Servers are not listed. The default maximum is 10 concurrent data inputs per MID Server, adjustable in MID Server properties.
    • MID Server Cluster: For clusters, only failover clusters with MID Servers supporting basic authentication are available. Log ingestion must be enabled on all cluster MID Servers, and Health Log Analytics enables it automatically if necessary. The system ensures failover by running the input on one MID Server and switching upon failure. Capacity validation ensures cluster readiness.
    • Service instance: Required field to bind log data. If no service instance exists, create one with the status set to Operational and add configuration items (CIs) appropriately.

    Read-Only Status Information

    • Transport: Protocol used (PubSub).
    • Sources count: Number of log sources created.
    • Status: Current status of the data input.
    • Disabled since: Timestamp when data input stopped or failed.
    • Last log time: Timestamp of the last streamed log.

    Transport Configuration Fields

    • Project ID: Google Cloud project identifier.
    • Topic Name: GCP Pub/Sub topic to subscribe to.
    • GCP Credential Alias: Credential alias for authentication; select existing or create new. Default is snocc.HLACredentialAlias.
    • Subscription Name: Pub/Sub subscription name. Defaults to ServiceNow-Subscription if left blank.

    Advanced Configuration

    • Subscriber Thread Pool Size: Number of concurrent threads pulling messages. Default is 1.
    • Default timezone: Time zone used when logs lack timestamps. Default is GMT.
    • Max length in bytes: Maximum log message size, defaulting to 32,766 bytes.
    • Character encoding: Read-only set to UTF-8.
    • Sub sample drop ratio & receive ratio: Controls event sampling; default is -1 indicating no sampling.
    • Rate limit: Maximum events processed per second; -1 means no limit.
    • Drop if queue is full: Option to discard logs if MID Server queue is congested; default is unchecked.

    Practical Implications for ServiceNow Customers

    By correctly configuring these fields, customers can ensure secure and efficient ingestion of GCP Pub/Sub log data into ServiceNow Health Log Analytics. The options for specific MID Servers or failover clusters provide flexibility and reliability for log streaming. Proper port selection and credential management are critical for connectivity and security. Advanced settings allow fine-tuning performance and event handling to match organizational needs.

    Understanding these configurations helps customers maintain continuous log ingestion with failover protection, proper authentication, and compliance with organizational security policies.

    Description of the fields on the GCP PubSub data input configuration form.

    Basic configuration

    Field Description
    Name Name of the new data input. This field is required.
    Port The port for the MID Server.

    Select a unique port from the array. The placeholder shows the range of ports from which to choose. Make sure that your organization’s security team opens the selected port.

    This field is required.
    Execute on Option to determine whether to use a specific MID Server or a MID Server cluster.

    This feature is supported in the Health Log Analytics application, Version 26.0.17 - February 2023 and later, available from the ServiceNow Store.
    MID

    (Only when the Execute on field is set to Specific MID Server)

    The MID Server to which the logs are streamed.
    Note:
    • You can select only MID Servers with log ingestion capability that support basic authentication. MID Servers that support mTLS are not listed.
    • The default maximum number of data inputs streaming logs to a single MID Server is 10. You can modify this number in the MID Server properties.
    • If log ingestion is not enabled for the selected MID Server, Health Log Analytics enables it automatically.
    This field is required.
    MID Server Cluster

    (Only when the Execute on field is set to Specific MID Server Cluster)

    The MID Server cluster to which the log data is pulled.

    The data input runs on a single MID Server in the cluster until that MID Server fails. The system then moves all the data input tasks to the next available MID Server in the cluster according to the configured order.

    This feature is supported in the Health Log Analytics application, Version 26.0.17 - February 2023 and later, available from the ServiceNow Store.

    Note:
    • Health Log Analytics supports only failover MID Server clusters. In these clusters, multiple MID Servers are grouped together for failover protection. When selecting a cluster from the data input form, the MID Server Clusters list displays only failover clusters.
    • The MID Server cluster must include only MID Servers that support basic authentication. mTLS is not supported for log ingestion.
    • Log ingestion must be enabled for each MID Server in the cluster. If log ingestion is not enabled for the active MID Server, Health Log Analytics enables it automatically.
    • The default maximum number of data inputs streaming logs to a single MID Server is 10. A cluster passes capacity validation if it contains at least one MID Server with fewer than 10 data inputs running on it, even when that MID Server is down.
    For more information about MID Server clusters, see Configure a MID Server cluster.

    This field is required.
    Service instance The service instance to which to bind the log data. This field is required.
    Note:
    If no relevant service instance exists, Create an service instance and add CIs to it. Set the status of the new service instance to Operational.

    The fields in the following table show read-only information.

    Field Description
    Transport Protocol used to send the log data: PubSub.
    Sources count The number of log sources this data input has created.
    Status Status of the data input.
    Disabled since The time when the data input stopped or failed.
    Last log time The time when the last log streamed in the data input.
    Table 1. Transport tab
    Field Description
    Project ID The project ID of the Google Cloud project. For example, my-project-id.
    Topic Name The Google Cloud Pub/Sub topic to which to subscribe Health Log Analytics. For example, my_topic.
    GCP Credential Alias The credential alias to be used.

    Specify one GCP credential alias by selecting the magnifying glass icon and then either selecting an existing credential alias from the Connection & Credential Aliases list, or selecting New to create a new record. The default is sn_occ.HLA_Credential_Alias.

    For information about creating a credential alias, see Credential aliases for Discovery.
    Subscription Name The subscription Health Log Analytics uses to receive log data that is published on the Google Cloud Pub/Sub topic.

    If you leave this field blank, Health Log Analytics uses ServiceNow-Subscription.

    Advanced configuration

    Table 2. Advanced tab
    Field Description Default value
    Subscriber Thread Pool Size The number of concurrent threads that are downloading files from the Google Cloud Pub/Sub topic. 1
    Default timezone The default time zone of events. The system uses this default when the log does not specify a time zone. GMT
    Max length in bytes The maximum length of log messages, in bytes. 32,766
    Character encoding (Read-only) The character encoding for this data input. UTF-8
    Sub sample drop ratio The ratio of events to drop. -1
    Sub sample receive ratio The ratio of events to receive. -1
    Rate limit The maximum number of events per second that this data input processes. -1
    Drop if queue is full Option for selecting to discard logs if many processes are waiting in the queue to access the MID Server. Clear