Rsyslog, Filebeat, or Winlogbeat data input configuration fields
Summarize
Summary of Rsyslog, Filebeat, or Winlogbeat Data Input Configuration Fields
This guide explains the configuration fields for setting up data inputs using Rsyslog, Filebeat, or Winlogbeat within ServiceNow's Health Log Analytics. These inputs enable streaming and ingestion of log data through a MID Server, facilitating log analysis, anomaly detection, and correlation within your environment.
Show less
Basic Configuration
- Data input name: Required field to name the new data input for easy identification.
- Description: Optional text to describe the data input.
- MID Server: Select a MID Server that supports log ingestion with basic authentication (mTLS-supported servers are excluded). By default, a single MID Server can handle up to 10 data inputs; this limit can be modified in MID Server properties.
- Port: Assign an available port within a suggested range on the MID Server; ensure your security team opens this port.
- Content pack (Linux/Filebeat only): Choose a content pack containing default source types and mapping scripts to accelerate setup and improve mapping accuracy.
Tagging and Binding Configuration
- Path: Specify the full log file path (wildcards allowed) to stream logs from; this is a required field.
- Service instance: Bind the log data to an existing or newly created service instance, which must be set to Operational.
- Component: Define the device type or stack layer context (e.g., Tomcat) to assist in anomaly detection and correlation. Components often map to Configuration Items (CIs) in the CMDB.
- Source Type: Define how logs are parsed and handled, supporting multiple source types per data input to accommodate diverse log formats.
Advanced Configuration for Rsyslog Data Inputs
- Use SSL/TLS: Option to secure log data transmission.
- Look up hostnames: Option to resolve IP addresses to hostnames via DNS lookup (default is false).
- Thread Counts: Configure boss threads (default 1) managing connections and worker threads (default 4) handling incoming data.
- Read timeout: Duration in seconds (default 30) after which inactive connections close.
- Default timezone: Time zone used when logs do not specify one (default GMT).
- Sub sample ratios: Control ratios of events to drop or receive (default -1 disables sampling).
- Max length in bytes: Maximum log message size (default 32766 bytes).
- Character encoding: Encoding standard for the input (default UTF-8).
- Drop if queue is full: Option to discard logs if MID Server load is high.
Advanced Configuration for Beats Data Inputs (Filebeat and Winlogbeat)
- Client inactivity timeout: Timeout in seconds (default 15) to close inactive channels.
- Worker thread count: Number of threads handling incoming data (default 4).
- Default time zone: Used when events lack time zone information (default GMT).
- Sub sample ratios: Ratios controlling event dropping or reception (default -1 disables sampling).
- Max length in bytes: Maximum size of log messages (default 32766 bytes).
- Character encoding: Encoding used for the data input (default UTF-8).
- Drop if queue is full: Option to discard logs under MID Server load (default false).
Practical Considerations
Ensure you select MID Servers with the appropriate log ingestion capabilities and configure ports that comply with your organization's security policies. Use content packs to expedite setup and improve log parsing effectiveness. Properly binding data inputs to service instances and components enhances log correlation and anomaly detection within your CMDB context. Adjust advanced settings to optimize performance and resource handling based on your environment's scale and requirements.
Description of the fields on the Rsyslog, Filebeat, and Winlogbeat data input configuration forms.
Basic configuration
| Field | Description |
|---|---|
| Data input name | Name of the new data input. This field is required. |
| Description | Description of the data input. |
| MID Server | The MID Server to which the logs stream. Note: This field is required.
|
| Port | The port on the MID Server. Choose a port within the suggested range from the array. The port must not be occupied by another process. Make sure that your organization’s security team opens the selected port. This field is required. |
| Content pack | (Linux using Filebeat only) The content pack to use. Content packs contain default source types and mapping script templates. Health Log Analytics activates the selected pack automatically and uses its mapping script for mapping the data input sources. For more information, see Health Log Analytics content packs for quicker time to value. |
| Field | Description |
|---|---|
| Path | The full path from which to stream logs. You can use a wildcard. This field is required. |
| Service instance | The service instance to which to bind the log data. This field is required. Note: If no relevant service instance exists, Create an service instance and add CIs to it. Set the status of the new service instance
to Operational. |
| Component | The device type or stack layer as context for the logs that is used for
anomaly detection and correlation. For example: Tomcat. Components typically represent CIs in the CMDB. Several components are often clustered together in a single service instance. |
| Source Type | The source type, which defines how Health Log Analytics
handles a specific application and parses the log data. For example: Tomcat
Catalina. Each data input can have multiple source types, based on the diversity of its log formats. Service instances and components can have any number of source types. |
Advanced configuration
For Rsyslog data inputs:
| Field | Description | Default values |
|---|---|---|
| Use SSL/TLS | Option for selecting to use SSL/TLS. | |
| Look up hostnames | Option for selecting to perform DNS lookup to resolve IPs to hostnames. | false |
| Boss thread count | The number of threads that manage connections. | 1 |
| Worker thread count | The number of threads that handle incoming data. | 4 |
| Read timeout seconds | The timeout in seconds since the last read. When the timeout expires, the system closes the channel. | 30 |
| Default timezone | The default time zone of events. The system uses this default when the log does not specify a time zone. | GMT |
| Sub sample drop ratio | The ratio of events to drop. | -1 |
| Sub sample receive ratio | The ratio of events to receive. | -1 |
| Max length in bytes | The maximum length of log messages in bytes. | 32766 |
| Character encoding | The character encoding for this data input. | UTF-8 |
| Drop if queue is full | Option for selecting to discard logs if there is a load on the MID Server. |
For data inputs that use Beats agents:
| Field | Description | Default value |
|---|---|---|
| Client inactivity timeout (sec) | The timeout, in seconds, to close an inactive channel. | 15 |
| Worker thread count | The number of threads that handle incoming data. | 4 |
| Default time zone | The default time zone of events. The system uses this default when the log does not specify a time zone. | GMT |
| Sub sample drop ratio | The ratio of events to drop. | -1 |
| Sub sample receive ratio | The ratio of events to receive. | -1 |
| Max length in bytes | The maximum length of log messages, in bytes. | 32766 |
| Character encoding | The character encoding for this data input. | UTF-8 |
| Drop if queue is full | Option for selecting to discard logs if there is a load on the MID Server. | false |