Rsyslog, Filebeat, or Winlogbeat data input configuration fields

  • Release version: Xanadu
  • Updated August 1, 2024
  • 3 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Rsyslog, Filebeat, or Winlogbeat Data Input Configuration Fields

    This guide explains the configuration fields used when setting up Rsyslog, Filebeat, or Winlogbeat data inputs within ServiceNow’s Health Log Analytics. It helps ServiceNow customers configure log data streaming to the MID Server, enabling efficient log ingestion, processing, and anomaly detection.

    Show full answer Show less

    Basic Configuration

    • Data input name: Required field to specify the name of the new data input.
    • Description: Optional field to describe the data input.
    • MID Server: Required selection of a MID Server that supports log ingestion with basic authentication (mTLS servers are excluded). The default limit is 10 data inputs per MID Server, adjustable via MID Server properties.
    • Port: Required field to select an open port on the MID Server within suggested ranges. Coordination with security teams is necessary to ensure the port is accessible.
    • Content pack (Linux Filebeat only): Optional selection of content packs that provide default source types and mapping templates to accelerate setup and improve log parsing accuracy.

    Tagging and Binding

    • Path: Required full path (supports wildcards) for streaming logs.
    • Service instance: Required binding of log data to a service instance. If none exists, create one with an Operational status.
    • Component: Defines the device type or stack layer context for logs, aiding anomaly detection and correlation (e.g., Tomcat). Components usually correspond to CIs in the CMDB.
    • Source Type: Identifies how logs are parsed and handled by Health Log Analytics (e.g., Tomcat Catalina). Multiple source types can be assigned per data input.

    Advanced Configuration

    Rsyslog Data Inputs

    • Use SSL/TLS: Option to enable secure communication.
    • Look up hostnames: Enables DNS lookup to resolve IP addresses (default is false).
    • Boss and Worker thread counts: Controls connection management and data processing threads.
    • Read timeout: Sets the inactivity timeout before closing connections (default 30 seconds).
    • Default timezone: Specifies the timezone for events lacking timestamps (default GMT).
    • Sub sample drop/receive ratios: Manage event sampling rates (-1 means no sampling).
    • Max length in bytes: Maximum size of log messages (default 32766 bytes).
    • Character encoding: Encoding used for logs (default UTF-8).
    • Drop if queue is full: Option to discard logs if MID Server load is high.

    Beats (Filebeat/Winlogbeat) Data Inputs

    • Client inactivity timeout: Timeout to close inactive channels (default 15 seconds).
    • Worker thread count: Number of threads handling incoming data (default 4).
    • Default time zone: Timezone used if logs lack timestamps (default GMT).
    • Sub sample drop/receive ratios: Controls event sampling (-1 means no sampling).
    • Max length in bytes: Maximum log message size (default 32766 bytes).
    • Character encoding: Encoding of log data (default UTF-8).
    • Drop if queue is full: Option to drop logs when MID Server is under load (default false).

    Practical Application

    By accurately configuring these fields, ServiceNow customers can ensure reliable and secure log streaming from various sources through the MID Server into Health Log Analytics. This setup enhances log parsing, correlation, and anomaly detection capabilities, supporting better operational insights and incident management.

    Description of the fields on the Rsyslog, Filebeat, and Winlogbeat data input configuration forms.

    Basic configuration

    Table 1. Getting Started tab
    Field Description
    Data input name Name of the new data input. This field is required.
    Description Description of the data input.
    MID Server The MID Server to which the logs stream.
    Note:
    • You can select only MID Servers with log ingestion capability that support basic authentication. MID Servers that support mTLS are not listed.
    • The default maximum number of data inputs streaming logs to a single MID Server is 10. You can modify this number in the MID Server properties.
    This field is required.
    Port

    The port on the MID Server.

    Choose a port within the suggested range from the array. The port must not be occupied by another process. Make sure that your organization’s security team opens the selected port.

    This field is required.
    Content pack (Linux using Filebeat only) The content pack to use.

    Content packs contain default source types and mapping script templates. Health Log Analytics activates the selected pack automatically and uses its mapping script for mapping the data input sources. For more information, see Health Log Analytics content packs for quicker time to value.
    Table 2. Tagging and Binding tab
    Field Description
    Path The full path from which to stream logs. You can use a wildcard. This field is required.
    Service instance The service instance to which to bind the log data. This field is required.
    Note:
    If no relevant service instance exists, Create an service instance and add CIs to it. Set the status of the new service instance to Operational.
    Component The device type or stack layer as context for the logs that is used for anomaly detection and correlation. For example: Tomcat.

    Components typically represent CIs in the CMDB. Several components are often clustered together in a single service instance.
    Source Type The source type, which defines how Health Log Analytics handles a specific application and parses the log data. For example: Tomcat Catalina.

    Each data input can have multiple source types, based on the diversity of its log formats. Service instances and components can have any number of source types.

    Advanced configuration

    For Rsyslog data inputs:

    Table 3. Rsyslog advanced configuration form
    Field Description Default values
    Use SSL/TLS Option for selecting to use SSL/TLS.
    Look up hostnames Option for selecting to perform DNS lookup to resolve IPs to hostnames. false
    Boss thread count The number of threads that manage connections. 1
    Worker thread count The number of threads that handle incoming data. 4
    Read timeout seconds The timeout in seconds since the last read. When the timeout expires, the system closes the channel. 30
    Default timezone The default time zone of events. The system uses this default when the log does not specify a time zone. GMT
    Sub sample drop ratio The ratio of events to drop. -1
    Sub sample receive ratio The ratio of events to receive. -1
    Max length in bytes The maximum length of log messages in bytes. 32766
    Character encoding The character encoding for this data input. UTF-8
    Drop if queue is full Option for selecting to discard logs if there is a load on the MID Server.

    For data inputs that use Beats agents:

    Table 4. Beats advanced configuration form
    Field Description Default value
    Client inactivity timeout (sec) The timeout, in seconds, to close an inactive channel. 15
    Worker thread count The number of threads that handle incoming data. 4
    Default time zone The default time zone of events. The system uses this default when the log does not specify a time zone. GMT
    Sub sample drop ratio The ratio of events to drop. -1
    Sub sample receive ratio The ratio of events to receive. -1
    Max length in bytes The maximum length of log messages, in bytes. 32766
    Character encoding The character encoding for this data input. UTF-8
    Drop if queue is full Option for selecting to discard logs if there is a load on the MID Server. false