CN100576819C - Flow Analysis Method Based on Linux Kernel - Google Patents

Abstract

A flow analysis method based on the Linux kernel is realized by using a loadable kernel in Linux, and can realize packet-level and flow-level flow analysis. The monitoring and analysis indicators provided include traffic rate, link utilization rate, layer 2-7 protocol distribution, traffic rate of different protocols, packet size distribution, etc.; it can also monitor specific types of flows and provide related information according to various flow definition rules. statistical results. Due to the adoption of a loadable kernel module, overheads such as system calls and memory copies are reduced, and traffic analysis performance is improved. Experimental results show that for 100Mbps and 1000Mbps links, the method of sampling can realize real-time monitoring and analysis of traffic rates of 142000pps and 405000pps respectively. Comparing the user space traffic analysis method based on libpcap, this method can improve the processing power by about 50%.

Description

Flow Analysis Method Based on Linux Kernel

technical field

The invention relates to the technical field of computer network traffic analysis, in particular to design a Linux kernel-based traffic analysis method.

Background technique

Traffic indicators are one of the most important indicators for network operation and maintenance, and are the basis for network planning and design, protocol design, service deployment, traffic engineering implementation, attack detection, and fault diagnosis. Through the monitoring and analysis of network traffic, a large amount of network performance information, network service status, and attacks or weaknesses in the network can be obtained. Passive traffic monitoring through packet capture and protocol analysis is the most commonly used method for traffic monitoring to obtain traffic indicators at each protocol level.

In recent years, the network link rate has increased rapidly, and has begun to move from Mbit/s to Gbit/s, and may develop to 40Gbit/s or even Tbit/s in the near future. Gigabit network cards and gigabit switches are already quite cheap. Almost every new LAN user will adopt this technology. An ordinary home user can get more bandwidth than a company got two years ago. The popularity of network applications and the development of network technology have led to increasingly large network traffic. Passive traffic monitoring and analysis based on packet capture is limited by many technologies, such as PCI (Peripheral Component Interconnect, peripheral component interconnection) bus throughput and storage capacity. , memory access speed, and disk array speed ^[1] , so that the traditional passive flow monitoring and analysis technology that satisfies low-speed link flow monitoring and analysis cannot be applied to a higher-speed network environment, such as the flow monitoring and analysis tool based on libpcap. In order to overcome the performance limitations brought by the operating system, PCI bus bandwidth and system resources, many researchers began to use network processors for traffic monitoring and analysis. However, the use of network processors often requires dedicated equipment, and the development cycle is long, so it is not realistic to deploy a large number of them.

The packet capture method is the most common method for passive traffic monitoring analysis. In recent years, many researchers and hardware manufacturers have tried to achieve passive traffic monitoring and analysis through dedicated hardware, such as OC3MON and DAG. In addition, some new research projects on traffic monitoring and analysis are also underway. For example, SCAMP in Europe has designed a data packet capture architecture; MAGNet can monitor application flows; CoralReef is developed by the famous CAIDA organization. One of the important tools to collect and analyze Internet passive traffic monitoring information.

It is still the most commonly used method to implement traffic monitoring and analysis on a common platform, such as Tcpdump/Libpcap, Sniffer, etc. The basic principle is to capture data packets flowing through links or ports, analyze each data packet and obtain traffic statistics and analysis results. Traffic analysis methods are widely used in intrusion detection (snort, Bro), traffic analysis (ntop, ethereal, tcpflow) systems. Most of the current traffic analysis tools are implemented based on libpcap, which can provide basic network link status information, such as link utilization, bandwidth, and protocol distribution.

Libpcap is a packet capture library that can be applied to various operating systems (such as Linux, FreeBSD, Solaris, etc.), using this library can shield developers from the technology used in the underlying network link. Libpcap provides a powerful kernel packet filter - BPF (Berkeley Packet Filter). Before introducing BPF, first explain the implementation of network communication in the Linux operating system. Figure 1 is a flow diagram of packet reception in a simple linux system, including the location of BPF. The Linux network interface includes four important parts: network device interface, network core interface, network protocol stack and network socket interface. The first two parts are the basis for realizing network communication, and are also closely related to the traffic analysis method introduced in this article. The network device interface is one of the most critical parts to realize network communication. It can shield different physical media at the bottom layer and provide a unified data packet sending/receiving interface, such as the network device (Network Interface) shown in Figure 1. The network core interface is responsible for obtaining the corresponding data packets from the network device and buffering them in the kernel, and distributes them according to the processing functions registered by the user or the system by default. The BPF shown in Figure 1 belongs to the network core interface. BPF is a kernel-based packet filter and is one of the important components for packet capture. BPF is considered to be the best implementation of packet filtering components in Unix ^[10] . BPF consists of two main parts: packet duplicator and packet filter. The data packet duplicator is responsible for obtaining a copy of the data packets in the network link from the network device driver, and sending the copy to the data packet filter. The filter determines the data packet selection according to the filtering rules. It is very important that BPF only passes packets that meet the requirements of the rules to user space. At the same time, BPF also supports transferring only the required part of the data packet, such as the first 64 bytes of the data packet, to the user space to reduce the overhead of data packet copying. For example, for an Ethernet packet, the first 64 bytes are sufficient to analyze the Ethernet header and IP/TCP header.

Contents of the invention

The present invention is a kind of flow analysis method based on Linux kernel, and the key of this method is that it adopts the realization based on the loadable mode of Linux kernel, uses for user space program with the character device mode, and specific content is described as follows:

1) The traffic analysis method based on the Linux kernel is implemented in the Linux kernel space, and integrates the data packet filtering and network traffic analysis functions of BPF, which can support the analysis of more than 100 protocol data packets on the 2-7 layer. The implementation method can effectively reduce memory copy and system call time, reduce storage space requirements, and improve the efficiency of traffic monitoring and analysis. Accompanying drawing 1 is the system structural diagram of this method design.

2) The arrival and processing flow of the network data packet in the Linux system kernel can be described as follows: the network card sends a hard interrupt to the CPU when it receives the data packet, and the CPU will take the data packet out of the buffer memory of the network card and send it to the packet core of the kernel. Create a new node in the cache (socket buff, Skbuff) linked list, store the received data packet, and wait for the processing of the protocol stack or other registered function functions. Because the method completes all traffic analysis functions in the kernel, it only provides the analysis results to the user space program through the API. Moreover, the network data packet to be analyzed can be directly obtained from the Skbuff linked list in the kernel, which greatly reduces the memory copy and system call overhead from the kernel to user space.

This method directly obtains the content of the data packet in the Skbuff, and realizes all traffic analysis functions in the kernel space, including the data packet capture function, which is implemented in the kernel.

3) The traffic analysis method of the present invention supports two levels of traffic analysis: packet level and flow (Flow) level. Packet level, by default, it can complete the protocol analysis of all original data packets, and provide more than 200 protocol analysis results. These results can be obtained through the interface available in the user space provided by the method itself. The main analysis indicators include traffic rate (packet rate and byte rate), link utilization, protocol distribution at different levels, packet size distribution, packet interval distribution, and Different protocol traffic rates, etc. In addition, it is also possible to monitor specified flows and provide flow-based analysis indicators according to different flow rule definitions. The statistical indicators include statistical distribution of flow duration, flow rate, statistical distribution of flow size, and statistical distribution of concurrent flow numbers. etc.; other detailed indicators vary according to the definition of flow rules, such as distinction between inbound and outbound traffic, specific ports, specific AS domains, specific IP address segments, monitoring and analysis of specific services, top N IP hosts in traffic, and IP host pair And the top N ports and AS domains with traffic ranking.

4) Supporting the capture of original data packets is also an important function of traffic analysis tools, so this method provides functional support for capturing a certain number of original data packets. The configuration function provided in the method, the upload of the original data package and the statistical analysis data can be used by the user space program through the designed API interface. And since BPF (Berkeley PacketFilter) is a relatively mature and widely used filter, the method inherits the syntax support for BPF filtering rules.

Description of drawings

Figure 1 is a flow chart of the data packet receiving process under the Linux system.

Fig. 2 is a schematic diagram of the system structure of the kernel-based traffic analysis method.

Fig. 3 is a flowchart of a flow analysis method based on the Linux kernel.

Detailed ways

Figure 1: The trend diagram of the data packet receiving process in the Linux system kernel. First, the network card will send an interrupt request to the CPU when it receives the data packet. The CPU obtains the arriving data packet from the network card cache according to the interrupt request. Space, and create a node corresponding to the data packet in the Skbuff, and finally hand it over to the protocol stack or a BPF-like kernel-based processing module for processing. All processing and results communicate with user space programs through system calls according to user needs.

Figure 2: Architecture design diagram of kernel-based traffic analysis method. All analysis functions of this method are implemented in the kernel, and two levels are provided: packet-level and flow-level monitoring and analysis indicators, and support the capture of a certain number of original data packets. The analysis results and original data packets are provided to user space programs through API.

The flow analysis method based on the Linux kernel of Fig. 3, its steps are as follows:

First, the network card will send an interrupt request to the CPU when it receives the data packet, and the CPU will obtain the arriving data packet from the network card cache according to the interrupt request, and the data packet will be transmitted to the kernel space via the PCI bus, and a node corresponding to the data packet will be created in the Skbuff, and finally It is analyzed by the data packet analysis function registered in the kernel to obtain the above-mentioned flow analysis indicators.

The specific steps are:

Step S1, setting the network card to promiscuous mode to obtain all network packets;

Step S2, registering a kernel-level registration kernel-level network traffic analysis function in the operating system kernel to process the protocol type specified by the analysis function, and the protocol type specified by the traffic analysis function in this method is all Ethernet types;

Step S3, configuring traffic analysis parameters, so that the traffic analysis function can perform traffic analysis according to user-defined rules;

Step S4, if the packet kernel buffer linked list in the kernel space is not empty, then call the registered traffic analysis function for analysis, and put the result in the statistical result cache and the flow record cache;

Step S5, if the module receives the user space unloading module request, cancel the network card promiscuous mode and execute S6;

Step S6, canceling the registered kernel-level traffic analysis function;

Step S7, if the module receives a request from the user space to obtain the analysis result, process the request and return the corresponding result.

Specific examples:

1. Implement the traffic analysis function, configuration function, and result acquisition API mentioned in the method by means of character device drivers under the Linux kernel.

2. Load the character device driver that realizes the traffic analysis function, and configure the traffic analysis function in the kernel according to the provided character device operation API and obtain the analysis results.

Claims (5)

1, a kind of flow analysis method based on Linux kernel, at first network card receives data packet and can send interrupt request to CPU, and CPU obtains the data packet that arrives from network card cache according to interrupt request, and data packet is passed to kernel space via PCI bus, and Create a node corresponding to the data packet in the data packet kernel cache, and finally submit it to the data packet analysis function registered in the kernel for analysis to obtain the traffic analysis index analyzed by the data packet analysis function. The specific steps of the method are as follows: Step S1, setting the network card to promiscuous mode to obtain all network packets; Step S2, registering a kernel-level network traffic analysis function in the operating system kernel to process the protocol type specified by the analysis function, and the protocol type specified by the traffic analysis function in this method is all Ethernet types; Step S3, configuring traffic analysis parameters, so that the traffic analysis function can perform traffic analysis according to user-defined rules; Step S4, if the data packet kernel buffer list in the kernel space is not empty, call the registered traffic analysis function for analysis, and put the result in the statistical result cache and the flow record cache. 2. The traffic analysis method based on the Linux kernel according to claim 1, characterized in that the method uses a loadable kernel module and uses a character device for user space programs. 3. The Linux kernel-based traffic analysis method according to claim 1, characterized in that the method provides two different levels of traffic analysis indicators: packet level and flow level. 4. The flow analysis method based on the Linux kernel according to claim 3, wherein the packet-level indicators in the method provide more than 200 types of protocol analysis, and the analysis indicators at least include traffic rate, link utilization, and different levels of protocol distribution, packet size distribution, packet interval distribution, and traffic rates of different protocols. 5. The Linux kernel-based flow analysis method according to claim 3, wherein the flow-level indicators in the method include statistical distribution of flow duration, flow rate, statistical distribution of flow size, and statistical distribution of concurrent flow numbers.

Images