Data collection and discovery using Netflow

  • Release version: Yokohama
  • Updated January 30, 2025
  • 2 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Data collection and discovery using Netflow

    Service Mapping enables discovery of Configuration Items (CIs) and their connections using network traffic data collected via the Netflow protocol. Netflow is one of several traffic-based discovery methods Service Mapping supports, alongside netstat, lsof commands, and VPC Flow Logs. While out-of-the-box discovery uses TCP-related data from netstat, ss, and lsof, leveraging Netflow requires additional configuration but enriches discovery capabilities by providing detailed network flow information.

    Show full answer Show less

    Data Collection Setups

    • Testing Setup: The Netflow Collector is installed on a separate server within the organization's network, different from the MID Server host. This setup requires manual copying of Netflow data files to the MID Server and results in a semi-automated data collection flow.
    • Standard Operation Setup: The Netflow Collector is co-located with the MID Server on the same server inside the network, enabling fully automated data collection, processing, and analysis.

    Data Flow Process

    • The Netflow daemon captures flow data from network switches communicating with servers.
    • The Netflow Collector writes this data into a raw nfdump output file using the nfdump utility.
    • In test setups, this file may need compression (gzip) and manual transfer to the MID Server.
    • The MID Server processes the raw nfdump data and places processed information onto the ECC queue.
    • A sensor retrieves processed data from the ECC queue and writes it into the Flow Connection [saflowconnection] table.
    • Service Mapping uses data from both the cmdbtcp and saflowconnection tables to identify unique outbound connections not captured by discovery patterns, enriching the CI connection information and updating the service map accordingly.

    Benefits for ServiceNow Customers

    By configuring Service Mapping to use Netflow data, customers gain a more comprehensive and automated view of network traffic and CI relationships. This results in improved accuracy of service maps through enhanced detection of network connections, supporting better IT service management and impact analysis.

    Service Mapping can perform discovery based on data collected using the Netflow protocol. Netflow is a protocol that Service Mapping can use to collect data about CIs and their connections along with Netstat and lsof commands.

    Using the Netflow protocol for collecting data is one of the traffic-based discovery methods. Other methods deployed by Service Mapping are using netstat and lsof commands and the VPC Flow Logs. For more information, refer to Traffic-based discovery in Service Mapping.

    In base systems, which are the default or standard configurations, traffic-based discovery relies solely on TCP-related data collected using the netstat, ss, and lsof commands. Discovery based on Netflow and VPC logs requires additional configuration. You can enrich your traffic-based discovery by configuring Service Mapping to use the Netflow protocol.

    The component, which receives data in the Netflow format is the Netflow Collector. Its location depends on whether you configure data collection for testing purposes or standard operation:
    For the test purposes
    This setup results in half automated data collection flow, where Service Mapping imports data only if you manually copy it from the Netflow Collector. You place the Netflow Collector on a server inside your organization network. This must be a server different from the server hosting the MID Server. You configure and test this setup as described in Configure a Netflow data import for testing.
    For standard operation
    This setup results in fully automated data collection flow, where all involved components send, collect and analyze data automatically. You place the Netflow Collector on the same server as the MID Server inside your organization network. For instructions, see Configure data collection using Netflow.
    Netflow-based discovery has the following flow:
    1. The Netflow daemon runs and receives data from switches communicating with servers in the organization. The Netflow Collector writes received data from the Netflow daemon.
    2. The server, hosting the Netflow collector, uses the Netflow nfdump utility to write the data into the nfdump output file. This file summarizes the raw data on all switches used for server communication.
      Figure 1. Collecting data and writing it into the nfdump output file

      Standard Netflow configuration: collecting data and writing it into the nfdump output file
    3. In testing setups, where the Netflow Collector is located not on the same server as the MID Server, you may need to convert the nfdump into the gzip format. Then you must manually copy the raw data in the nfdump output file onto the MID Server.
      Figure 2. Copying the nfdump output file onto the MID Server

      Standard Netflow configuration: copying the nfdump output file onto the MID Server
    4. The MID Server processes the raw data in the nfdump output file and places the processed information onto the ECC queue.
      Figure 3. Analyzing the raw data and placing it at the ECC Queue

      Standard Netflow configuration: analyzing the raw data and placing it at the ECC Queue
    5. A sensor retrieves the processes data from the ECC queue and writes it into the Flow Connection [sa_flow_connection] table.

    6. Whenever Service Mapping checks the ECC queue and receives information on a discovered CI, it checks these tables for any data on outbound connections related to the CI: the cmdb_tcp and sa_flow_connection tables. If these two tables contain unique data that patterns did not discover, Service Mapping enriches the information about the CI connections and adds them to the map.

      Figure 4. Service Mapping retrieves data from the sa_flow_connection table

      Standard configuration: Service Mapping retrieves data from the sa_flow_connection table