Data collection and discovery using Netflow

  • Release version: Xanadu
  • Updated August 1, 2024
  • 2 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Data collection and discovery using Netflow

    Service Mapping supports discovery of Configuration Items (CIs) and their connections using data collected via the Netflow protocol, complementing other traffic-based discovery methods such as netstat, lsof commands, and VPC Flow Logs. While base system configurations rely on TCP-related data from netstat, ss, and lsof, Netflow-based discovery requires additional setup but enables richer traffic insights.

    Show full answer Show less

    Key Features

    • Netflow Collector: The central component that receives and processes Netflow data from network switches. Its deployment varies depending on the use case:
      • Testing purposes: Netflow Collector is placed on a separate server within the organization’s network, distinct from the MID Server. Data collection is semi-automated; the nfdump output file must be manually copied to the MID Server, possibly after gzip compression.
      • Standard operation: Netflow Collector resides on the same server as the MID Server for fully automated data collection, processing, and analysis.
    • Data processing flow:
      1. Netflow daemon collects traffic data from switches.
      2. Netflow Collector uses the nfdump utility to summarize raw data into an output file.
      3. MID Server processes the nfdump file and places processed data onto the ECC queue.
      4. A sensor retrieves this data from the ECC queue and writes it into the Flow Connection [saflowconnection] table.
    • Integration with Service Mapping: When Service Mapping discovers a CI, it queries the cmdbtcp and saflowconnection tables for outbound connection data. If new, unique connection data is found, Service Mapping enriches the CI’s connection map accordingly.

    Practical Benefits

    • Enables enhanced traffic-based discovery beyond standard TCP command data, leading to more comprehensive and accurate mapping of CIs and their network connections.
    • Flexible deployment options accommodate both evaluation/testing and production environments.
    • Automated processing in production setups reduces manual intervention and accelerates data availability for discovery.

    Service Mapping can perform discovery based on data collected using the Netflow protocol. Netflow is a protocol that Service Mapping can use to collect data about CIs and their connections along with Netstat and lsof commands.

    Using the Netflow protocol for collecting data is one of the traffic-based discovery methods. Other methods deployed by Service Mapping are using netstat and lsof commands and the VPC Flow Logs. For more information, refer to Traffic-based discovery in Service Mapping.

    In base systems, which are the default or standard configurations, traffic-based discovery relies solely on TCP-related data collected using the netstat, ss, and lsof commands. Discovery based on Netflow and VPC logs requires additional configuration. You can enrich your traffic-based discovery by configuring Service Mapping to use the Netflow protocol.

    The component, which receives data in the Netflow format is the Netflow Collector. Its location depends on whether you configure data collection for testing purposes or standard operation:
    For the test purposes
    This setup results in half automated data collection flow, where Service Mapping imports data only if you manually copy it from the Netflow Collector. You place the Netflow Collector on a server inside your organization network. This must be a server different from the server hosting the MID Server. You configure and test this setup as described in Configure onetime data import using Netflow for testing purposes.
    For standard operation
    This setup results in fully automated data collection flow, where all involved components send, collect and analyze data automatically. You place the Netflow Collector on the same server as the MID Server inside your organization network. For instructions, see Configure data collection using Netflow.
    Netflow-based discovery has the following flow:
    1. The Netflow daemon runs and receives data from switches communicating with servers in the organization. The Netflow Collector writes received data from the Netflow daemon.
    2. The server, hosting the Netflow collector, uses the Netflow nfdump utility to write the data into the nfdump output file. This file summarizes the raw data on all switches used for server communication.
      Figure 1. Collecting data and writing it into the nfdump output file

      Standard Netflow configuration: collecting data and writing it into the nfdump output file
    3. In testing setups, where the Netflow Collector is located not on the same server as the MID Server, you may need to convert the nfdump into the gzip format. Then you must manually copy the raw data in the nfdump output file onto the MID Server.
      Figure 2. Copying the nfdump output file onto the MID Server

      Standard Netflow configuration: copying the nfdump output file onto the MID Server
    4. The MID Server processes the raw data in the nfdump output file and places the processed information onto the ECC queue.
      Figure 3. Analyzing the raw data and placing it at the ECC Queue

      Standard Netflow configuration: analyzing the raw data and placing it at the ECC Queue
    5. A sensor retrieves the processes data from the ECC queue and writes it into the Flow Connection [sa_flow_connection] table.

    6. Whenever Service Mapping checks the ECC queue and receives information on a discovered CI, it checks these tables for any data on outbound connections related to the CI: the cmdb_tcp and sa_flow_connection tables. If these two tables contain unique data that patterns did not discover, Service Mapping enriches the information about the CI connections and adds them to the map.

      Figure 4. Service Mapping retrieves data from the sa_flow_connection table

      Standard configuration: Service Mapping retrieves data from the sa_flow_connection table