RU2841028C1 - Method of detecting and counteracting denial of service attacks - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to a method of detecting and counteracting denial of service attacks. Disclosed is a method during which, by processing network traffic, a training table is formed, wherein network characteristics and interaction characteristics are used to form, a neural network of the auto-encoder type is trained, creating tables of threshold values of network characteristics and interactions, then forming the test table, calculating the average reconstruction error for the test table, if threshold values are exceeded, all packets from an external digital data transmission network are blocked from passing to the local digital data transmission network, for which the set of two IP addresses, protocol and server port matches one of the abnormal interactions.

EFFECT: high probability of detecting and protecting against types of denial-of-service attacks using attack source IP address substitution, distributed attacks, deflected attacks and denial-of-service attacks with increasing activity.

1 cl, 13 tbl

Description

Field of technology to which the invention relates

The proposed invention relates to the field of information security of digital data transmission networks and can be used to protect individual computers in a network and corporate networks from computer attacks of the denial of service type.

State of the art

The purpose of a denial of service attack (DoS/DDoS attack) is to overload a server, network or application in order to slow down or completely stop service to legitimate users. Such attacks are usually aimed at exhausting network or server resources. To implement such an attack, an attacker can use a wide range of techniques. Denial of service can be caused by both a large flow of traffic and by exploiting vulnerabilities in some network protocols. To strengthen and complicate counteraction to this type of attack, an attacker can use various methods.

One way to amplify an attack is to use a distributed botnet. In the case of such a distributed attack, the number of attack sources is quite large, which makes it difficult to block this type of attack and increases the overall intensity of the attack traffic.

Generating a large traffic flow does not always require a large distributed botnet. Attack amplification can be achieved through attack reflection techniques. For example, an attacker can send a specific small query to a DNS server, and the server's response will be significantly larger. If the attacker replaces the source IP address in his query with the address of the target server, then the response from the DNS server will be sent to this target server.

Source address substitution can be used in other techniques to implement similar attacks. For example, an attacker can change the attack source addresses to addresses from the internal network or other addresses of legitimate users.

Also, the intensity of a denial of service attack is not always stationary. Attacks with increasing activity can be used. Thus, if the counteraction method is based on calculating threshold values, then the attack will not be blocked immediately, but only when the intensity of the attack increases so much that it exceeds the specified threshold values.

A method for detecting DDoS attacks is known (international patent application CN 107864155, priority dated 12.12.2017). The method involves collecting network traffic and aggregating traffic parameters taking into account IP addresses, packets and ports. In this way, a number of statistical characteristics are calculated in a single interval. If the set threshold value is exceeded, then an attack report is generated. The attack target and attacking IP addresses are determined using activity ranking. The addresses that received the most traffic are considered the attack target. The addresses from which the most activity was received are considered the attack source.

The known method has the following disadvantages:

• vulnerability to distributed attacks and IP address spoofing techniques, since the source of the attack in such cases may be determined incorrectly;

• vulnerability to attacks with increasing activity, since attack reports are generated when the established activity threshold is exceeded;

• vulnerability to reflected attacks, since the server used to reflect the attack can be actively used by the protected server, then blocking traffic from such a server can lead to incorrect operation or denial of service of the target server.

There is also a known method for detecting computer attacks (RU Patent No. 2683631, priority dated 08.12.2017). The method consists in the fact that

• establish and store threshold values of parameters calculated per unit of time,

• receive a sequence of message packets from the network,

• remember received message packets,

• extract from message packets the data that characterizes them,

• calculate the values of parameters that depend on the received message packets,

• compare the calculated parameter values with the threshold values,

• make a decision on the presence or absence of a computer attack by comparing the calculated values of the parameters with the threshold values,

at the same time

• pre-set the required number of attack types to be determined Z≥1, the number of training L and control C examples for each attack type,

• determine the types of protocols and the necessary parameters Fij, where i≤Z, required to recognize attacks of given types and use these parameters as those characterizing data packets,

• define the structure of the neural network:

neural network type R,

number of hidden layers U,

the number of neurons in the input layer, hidden layers and output layer

such that the number of neurons in the input layer N is determined by the number of protocol fields required to recognize attacks of given types, and the number of neurons in the output layer is equal to K,

• form the required number of training examples, which are vectors of dimension N+K, where the first N elements are equal to the values of certain protocol fields, and the last K are equal to zero, with the exception of the value corresponding to the type of attack to which this sample element belongs, which is equal to one,

• set the required recognition reliability D for the neural network,

• train a neural network using prepared examples,

• complete the training of the neural network after all iterations of sample generation have been completed or upon reaching the required recognition reliability,

• in the presence of a computer attack, the type of a single computer attack is determined by a combination of calculated values of the parameters Oj based on the following conditions:

if the value obtained on only one of the output neurons exceeds the threshold value, then the attack type Aj is determined,

If the value obtained on several output neurons exceeds the threshold value, then the attack type is considered combined and type Aj is added to the attack type.

As a neural network for implementing the method, a neural network of the multilayer perceptron type was formed and trained and its training was performed using the back propagation method.

It should be noted that the provided table of the results of the method contains the values of the accuracy of detecting computer attacks, and the term "accuracy of detection" is most likely not used entirely correctly; accordingly, it would probably be more understandable to use the term "proportion of detected attacks" or, taking this into account, the probability of detecting attacks.

The method uses a number of network characteristics calculated per unit of time. Thus, the method determines the presence or absence of an attack of one of the types used in training.

However, the known method has a number of disadvantages:

• relatively low probability of detecting distributed attacks, attacks using the technique of substituting the IP address of the attack source, reflected attacks and attacks with increasing activity,

• the need to pre-generate a significant number of training examples for each type of attack,

• inability to identify the flow related to the attack.

Disclosure of invention

The technical result is:

1) increasing the probability of detection and protection against types of denial of service attacks using IP spoofing of the attack source, distributed attacks, reflected attacks and denial of service attacks with increasing activity,

2) ensuring the ability to detect specific packets and flows belonging to the attack;

3) ensuring the ability to train a neural network only on legitimate traffic without the need to prepare labeled data for each type of attack.

For this purpose, a method is proposed for detecting and countering denial of service attacks using a computing system connected to an external digital data transmission network and a local digital data transmission network and including at least one computer that has an installed operating system and application programs that allow

• receive, process and store multidimensional network traffic data;

• ensure the formation, functioning and training of a neural network of the autoencoder type;

• block network packets passing from the external digital data transmission network to the local digital data transmission network;

a method consisting in the fact that:

• start the computer in controlled normal operation mode;

• create a table of network characteristics, initially filled with zero values;

• create a table of statistical characteristics of the network, initially filled with zero values;

• create a table of interaction characteristics, initially empty;

• create a table of statistical characteristics of interactions, initially empty;

• create a table for training, initially empty;

• set the period T1 for generating tables of flow parameters;

• set the time interval T2>T1 for forming the training table;

• receive and process data from network streams;

• supplement the table for training throughout the entire time interval T2, for this, with the period T1, the following actions are performed:

receive a table of flow parameters;

create a flow characteristics table, initially empty, with a number of rows equal to the number of rows in the flow parameters table and with columns for flow parameter values, network characteristics, and interaction characteristics;

fill in the flow parameter columns in the flow characteristics table with values from the columns of the flow parameter table;

calculate network characteristics for the flow parameter table;

smooth out the values of network characteristics;

write the smoothed network characteristics into the network characteristics table;

write down in the table of flow characteristics the values of the network characteristics from the table of network characteristics in columns for each of the network characteristics in the form of a set of identical values in each row;

calculate the values of statistical characteristics of the network;

record the values of the statistical characteristics of the network in the table of statistical characteristics of the network;

fill in the values in the interaction characteristic columns in the flow characteristics table by performing the following actions for each row of the flow characteristics table:

define the interaction of a stream using its two IP addresses, protocol and server port;

if the interaction characteristics table does not contain the interaction to which the flow belongs, then:

create a new interaction in the interaction characteristics table;

set the values for the number of streams, packets, and bytes per second to zero in the interaction characteristics table;

set the values of the number of packets and bytes equal to the number of packets and bytes in the stream in question;

otherwise, if the interaction is in the interaction characteristics table, then:

calculate the interaction characteristics for the flow under consideration;

smooth out the values of interaction characteristics;

set the time of the last interaction activity equal to the time of the flow appearance;

write the smoothed values of the interaction characteristics into the interaction characteristics table;

calculate statistical characteristics of the interaction;

write down the values of statistical characteristics of interaction in the table of statistical characteristics of interactions;

write down in the columns of the interaction characteristics of the flow characteristics table the values from the interaction characteristics table for the flow under consideration;

supplement the training table with rows from the flow characteristics table;

• at the end of the time interval T2, the symbolic values in the protocol and server port columns in the training table are digitized;

• digitize IP addresses by dividing the address into 4 byte values in a table for training;

• normalize the values in the byte and packet count columns, the eight digitized IP address columns, network characteristics, interaction characteristics, and the columns with the number of different TCP flags in the digitized training table, obtaining a normalized training table;

• form a neural network of the autoencoder type, which has

the dimension of the input layer and the output layer is equal to the number of columns in the normalized table for training,

one hidden layer with a dimension less than the number of columns in the normalized training table;

• train a neural network using a normalized table for training, and during training minimize the error in reconstructing the flow parameters R, which is calculated using the formula:

where n is the number of columns in the normalized table for training;

Y _i,j is the value of the i-th column of the j-th stream in the normalized table for training;

- the reconstructed value of the i-th column of the j-th stream in the normalized table for training at the output of the neural network;

• form a table of threshold values of network characteristics, initially empty;

• calculate threshold values for each network characteristic TN _i using the formula:

Where - assessment of the mathematical expectation of the i-th network characteristic from the table of statistical network characteristics;

- assessment of the standard deviation of the i-th network characteristic from the table of statistical network characteristics;

• record the threshold values of network characteristics in the table of threshold values of network characteristics;

• calculate threshold values for each interaction characteristic TC _i,j using the formula:

Where - estimate of the mathematical expectation of the i-th characteristic of the j-th interaction from the table of statistical characteristics of interactions;

- estimate of the standard deviation of the i-th characteristic of the j-th interaction from the table of statistical characteristics of interactions;

• record the threshold values of interaction characteristics in the table of threshold values of interaction characteristics;

• set the duration of the time interval for validation T3>T1;

• create a one-dimensional array of VR reconstruction mean error values, initially empty;

• throughout the entire time interval T3 with period T1, the following actions are performed:

receive a table of flow parameters;

create a flow characteristics table, initially empty, with a number of rows equal to the number of rows in the flow parameters table and with columns for flow parameter values, network characteristics, and interaction characteristics;

in the flow characteristics table, fill in the flow parameter columns with values from the flow parameter table columns;

calculate network characteristics for the flow parameter table;

smooth out the values of network characteristics;

write the smoothed network characteristics into the network characteristics table;

write down in the table of flow characteristics the values of the network characteristics from the table of network characteristics in columns for each of the network characteristics in the form of a set of identical values in each row;

calculate the values of statistical characteristics of the network;

record the values of the statistical characteristics of the network in the table of statistical characteristics of the network;

populate the values in the interaction characteristic columns of the flow characteristic table by performing the following steps for each row of the flow characteristic table:

define the interaction of a stream using its two IP addresses, protocol and server port;

if the interaction characteristics table does not contain the interaction to which the flow belongs, then:

create a new interaction in the interaction characteristics table;

set the values for the number of streams, packets, and bytes per second to zero in the interaction characteristics table;

set the values of the number of packets and bytes equal to the number of packets and bytes in the stream in question;

otherwise, if the interaction is in the interaction characteristics table, then:

calculate the interaction characteristics for the flow under consideration;

smooth out the values of interaction characteristics;

set the time of the last interaction activity equal to the time of the flow appearance;

write the smoothed values of the interaction characteristics into the interaction characteristics table;

calculate statistical characteristics of the interaction; record the values of statistical characteristics of the interaction in the table of statistical characteristics of the interactions;

write down in the columns of the interaction characteristics of the flow characteristics table the values from the interaction characteristics table for the flow under consideration;

digitize the symbolic values in the protocol and server port columns in the flow characteristics table;

digitize IP addresses by dividing the address into 4 byte values in the flow characteristic table;

normalize in the digitized flow characteristics table the values in the columns of the number of bytes and packets, in the eight columns of the digitized columns of IP addresses, network characteristics, interaction characteristics, and also in the columns with the number of different TCP flags;

obtain a validation table after normalizing the digitized flow characteristic table;

feed the validation table to the autoencoder input;

calculate the average reconstruction error MR using the formula

where n is the number of columns in the validation table;

m - the number of threads in the validation table;

Y _i,j is the value of the i-th parameter of the j-th stream in the validation table;

- the value of the i-th parameter of the j-th stream in the validation table reconstructed at the output of the neural network;

complement the VR array with the MR value;

• calculate the threshold value NT for the VR array;

• set the time period for the relevance of threshold values T4>T1;

• set the time interval of inactivity of interaction T5>T1;

• set the time interval for blocking interaction T6>T1;

• set the time interval for detecting and countering attacks T7>T1;

• throughout the entire period of time, to detect and counteract T7 attacks with a period of T1, the following actions are performed:

Stage A: obtain a table of flow parameters;

create a flow characteristics table, initially empty, with a number of rows equal to the number of rows in the flow parameters table and with columns for flow parameter values, network characteristics, and interaction characteristics;

fill in the flow parameter columns in the flow characteristics table with values from the columns of the flow parameter table;

calculate network characteristics for the flow parameter table;

smooth out the values of network characteristics;

write the smoothed network characteristics into the network characteristics table;

write down in the table of flow characteristics the values of the network characteristics from the table of network characteristics in columns for each of the network characteristics in the form of a set of identical values in each row;

calculate the values of statistical characteristics of the network;

record the values of the statistical characteristics of the network in the table of statistical characteristics of the network;

populate the values in the interaction characteristic columns of the flow characteristic table by performing the following steps for each row of the flow characteristic table:

define the interaction of a stream using its two IP addresses, protocol and server port;

if the interaction characteristics table does not contain the interaction to which the flow belongs, then:

create a new interaction in the interaction characteristics table;

set the values for the number of streams, packets, and bytes per second to zero in the interaction characteristics table;

set the values of the number of packets and bytes equal to the number of packets and bytes in the stream in question;

otherwise, if the interaction is in the interaction characteristics table, then:

calculate the interaction characteristics for the flow under consideration;

smooth out the values of interaction characteristics;

set the time of the last interaction activity equal to the time of the flow appearance;

write the smoothed values of the interaction characteristics into the interaction characteristics table;

calculate statistical characteristics of the interaction;

write down the values of statistical characteristics of interaction in the table of statistical characteristics of interactions;

write down in the columns of the interaction characteristics of the flow characteristics table the values from the interaction characteristics table for the flow under consideration;

digitize the symbolic values in the protocol and server port columns in the flow characteristics table;

digitize IP addresses by dividing the address into 4 byte values in the flow characteristics table;

normalize in the digitized flow characteristics table the values in the columns of the number of bytes and packets, in the eight columns of the digitized columns of IP addresses, network characteristics, interaction characteristics, and columns with the number of different TCP flags;

obtain a test table after normalizing the flow characteristics table;

a test table is fed to the autoencoder input;

calculate the average reconstruction error TRE for the test table

where n is the number of columns in the test table;

m - the number of threads in the test table;

Y _i,j is the value of the i-th parameter of the j-th stream in the test table;

- the value of the i-th parameter of the j-th flow in the test table of flow parameters reconstructed at the output of the neural network;

mark anomalous interactions as normal if they were not marked as anomalous during the T6 time interval;

if the TRE value is greater than NT, or at least one of the network characteristics exceeds a threshold from the network characteristic threshold table, then:

mark as abnormal all interactions in the interaction characteristics table whose characteristics exceed the threshold values from the interaction characteristics threshold values table;

block all packets from the external digital data transmission network from passing into the local digital data transmission network, for which the set of two IP addresses, protocol and server port matches one of the anomalous interactions;

remove interactions whose inactivity time has exceeded the T5 time interval from the interaction characteristics, interaction statistics, and interaction thresholds tables;

calculate and update the threshold values for the network characteristics and the characteristics of each interaction if the threshold values have not been updated during the time interval T4 and no abnormal interactions have been observed during the time interval T4;

if necessary, proceed to stage A. The proposed method uses a system for detecting denial of service attacks (DoS/DDoS attacks) based on the analysis of network interaction activity and overall network activity in real time.

One interaction is defined as the exchange of data that occurs when using a specific set of two IP addresses, a protocol, and a server port.

The method uses network flows to analyze traffic. The proposed implementation uses the NetFlow flow generation format, but other flow generation methods are also allowed. Multiple different flows may be related to a single interaction, provided that two IP addresses, the interaction protocol, and the server port number match. At least the following parameters are considered for each flow:

• Source IP address,

• Destination IP address,

• server port number,

• protocol,

• the number of TCP flags of each type,

• number of packets in the stream,

• number of bytes in the stream,

• time of flow appearance.

The table of flow parameters has the following form (Table 1)

It is assumed that during the operation of the method with a given period T1, tables of flow parameters will be formed. The parameters of all flows that were active during a given period of time will be recorded in such a table. In this case, the table of flow parameters is used only until the moment of updating the period; the table for the previous period is deleted.

The method can be implemented using a computing system that passes network traffic between an external network (Internet) and a protected local network (e.g., a corporate network), and includes at least one computer with an installed operating system and application programs. Thus, the method will analyze all incoming traffic from the external network and block individual packets based on a number of parameters.

The method can also be implemented using a computing system connected to a digital data transmission network and including at least one computer with an installed operating system and application programs. In this case, the computer will receive network traffic from a network device installed between the external network and the protected local network. In this case, the computer will be able to give commands to the network device to block individual packets based on a number of parameters.

Blocking a packet means that the blocked packet will not pass into the protected local network, however, such a packet will be used in the analysis method.

The method is based on calculating a number of network characteristics and interaction characteristics. A characteristic is an observed value that characterizes the current state of network activity or interaction.

Network characteristics are used to track activity across the entire network. Such characteristics are stored in a network characteristics table in which the columns are - network characteristics and in such a table there is only one row.

Network characteristics include:

• number of threads per second;

• number of packets per second;

• number of bytes per second.

The table of network characteristics looks like this (Table 2)

Interaction characteristics are used to track the activity of network interactions. Thus, each interaction will have its own set of characteristics. Such characteristics are stored in the interaction characteristics table. The rows in such a table are interactions (a unique interaction identifier is formed from a protocol, a server port, and two IP addresses, regardless of the order of the addresses), and the columns are interaction characteristics. In this case, for each network traffic flow or packet, it is possible to determine the interaction to which it belongs, since it is assumed that any flow or packet will have a set of two IP addresses, a protocol, and a server port. Based on this set of values, an interaction is determined for a flow or packet. Interaction characteristics include:

• number of interaction threads per second,

• number of interaction packets per second,

• number of interaction bytes per second,

• number of packets in the stream,

• number of bytes in the stream,

• time of last interaction activity.

The table of interaction characteristics is as follows (Table 3)

Network characteristics and interaction characteristics are used to detect and prevent denial of service attacks. Thus, exceeding the threshold values of network characteristics confirms the presence of an attack. Whereas exceeding the threshold values of certain interaction characteristics allows blocking only abnormal interactions, when the presence of an attack is confirmed.

Calculating thresholds may require some statistics that can be updated at each period when new flows are reported. The network statistics table and the interaction statistics table are used to store the network and interaction statistics.

For each interaction characteristic and network characteristic, several statistical characteristics can be stored. The proposed method assumes the use of estimates of the mathematical expectation and standard deviation. Thus, when updating a certain characteristic, the corresponding estimates of the mathematical expectation and standard deviation are also recalculated and updated.

In addition to tables, the method also uses a neural network of the autoencoder type with one hidden layer.

An autoencoder is a neural network architecture that learns to reconstruct input data from a compressed representation with minimal reconstruction error.

This neural network architecture assumes that the size of the input layer is the same as the size of the output layer, and the hidden layers are smaller than the input layer. The proposed method assumes that the hidden layer of the autoencoder is at least two times smaller than the input layer.

To train a neural network, a training table is created. To do this, a time interval T2 is set for forming a training table. It is assumed that the time interval T2 is longer than the period T1. The following actions are performed throughout the entire time interval T2 with the period T1:

• obtain a table of flow parameters;

• create a flow characteristics table with columns for flow parameters, network characteristics, and interaction characteristics:

Source IP address (flow parameter);

Destination IP address (flow parameter);

server port number (stream parameter);

protocol(stream parameter);

the number of TCP flags of each type (flow parameters);

number of flows per second over the network (network characteristic);

number of packets per second over the network (network characteristic);

number of bytes per second over the network (network characteristic);

number of interaction flows per second (interaction characteristic);

number of interaction packets per second (interaction characteristic);

number of bytes of interaction per second (characteristic

interactions);

number of packets in the flow (interaction characteristic);

number of bytes of the stream (interaction characteristic);

• fill the flow parameter columns in the flow characteristics table with values from the flow parameter table columns;

• calculate network characteristics and network statistics for the entire flow parameter table;

• calculate interaction characteristics and statistical interaction characteristics for each flow from the flow characteristics table;

• fill in the network characteristics and interaction characteristics columns in the flow characteristics table;

• supplement the training table with rows from the flow characteristics table.

In this case, the current values in the tables of network characteristics and interaction characteristics are updated when the table of flow characteristics is formed in each period. Thus, after the table of flow parameters is formed, the network characteristics are calculated. Moreover, for the entire table of flow parameters, one value of each of the network characteristics will be calculated.

The values of the network characteristics are smoothed taking into account the network characteristics for the previous period. The proposed implementation uses exponential smoothing, since this approach does not require storing a large number of previous values.

where s _t is the smoothed value at time t,

α - smoothing coefficient (0<α<1),

x _t is the observed value at time t.

Network characteristics are needed to track the overall activity in the entire network. In order to be able to conveniently transfer them to the neural network together with the parameters of individual flows and interaction characteristics, the network characteristics calculated for the entire flow parameter table are written to the columns of the network characteristics in the flow characteristic table. In this case, the values in such columns for each flow in one flow characteristic table will be the same for each row. However, when forming the flow characteristic table in the next period, the network characteristics will be updated and the values in the new flow characteristic table will differ. It is assumed that the period T1 of forming the flow parameter tables will be short enough, so the network characteristics can be considered relevant for each flow in one period.

In addition to the network characteristics, interaction characteristics for each flow will be calculated when forming the flow characteristics table. Interaction characteristics can be calculated for each flow in the flow parameters table as follows:

• define the interaction of a stream using its two protocol IP addresses and server port;

• if the interaction characteristics table does not contain the interaction to which the flow belongs, then:

create a new interaction in the interaction characteristics table;

set the values for the number of streams, packets, and bytes per second to zero in the interaction characteristics table;

set the values of the number of packets and bytes equal to the number of packets and bytes in the stream in question;

otherwise, if the interaction is in the interaction characteristics table, then:

calculate the interaction characteristics for the flow under consideration;

smooth out the values of interaction characteristics;

• set the time of the last interaction activity equal to the time of the flow appearance;

• calculate statistical characteristics of the interaction;

• record the values of statistical characteristics of interaction in the table of statistical characteristics of interactions;

• write down in the columns of the interaction characteristics of the flow characteristics table the values from the interaction characteristics table for the flow under consideration.

At the end of the training period, some columns in the training table need to be digitized or normalized. Digitization of symbolic parameters can be done using various methods, however, the proposed implementation assumes the following method of digitizing such parameters.

The protocol and server port are digitized using the One-Hot-Encoding method [https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html].

When digitizing symbolic parameters using the One-Hot-Encoding method, a set of unique values that a given symbolic parameter can take is first identified. This set of unique values is used both to digitize the training table and to digitize the validation and test tables. A separate column is created for each unique value. A separate column is also allocated for values that are not in the set of unique values. Such unknown values may appear when digitizing the validation and test tables, since the set of unique values for digitization remains unchanged. In this case, in each row of the resulting table, all values are initialized to zeros, except for the value in the column of the symbolic value of this row.

After the columns with digitization have been formed, the original column of symbolic values is deleted. This approach is often used to digitize symbolic values when using neural networks, since the symbolic values are converted to numerical values equal to 0 or 1.

For example, when encoding a protocol parameter, the set of unique symbolic values may be as follows: UDP, TCP, ICMP. Then the table of initial values (Table 4)

is digitized and converted into a table of digitized values (Table 5).

To digitize IP addresses, we use a split into 4 values by bytes (for example, the IP address 10.0.0.1 will be digitized using four values: 10, 0, 0, 1). We obtain a digitized table for training by replacing the columns of symbolic parameters in the table for training with the columns of these parameters after digitization.

The values of the number of bytes and packets, 8 columns, digitalized IP addresses and network characteristics, interaction characteristics and columns with the number of different flags in the flow in the digitalized table for training are normalized. Thus, we obtain a normalized table for training. The proposed implementation uses the Yeo-Johnson method, which uses logarithmic and exponential functions in order to bring the original distribution of the parameter closer to normal.

[https://www.researchgate.net/publication/31373954_A_new_family_of_power_transformations_to_improve_normality_or_symmetry].

The neural network is trained using a normalized training table, and the reconstruction error (RE) is optimized using the gradient descent method. The mean square error is used as the reconstruction error, which is calculated using the formula:

where n is the number of flow parameters;

Y _i is the value of the i-th parameter of the j-th stream in the training table;

- the value of the i-th parameter of the j-th stream reconstructed at the output of the neural network.

When training a neural network on legitimate traffic flows, it is assumed that if the reconstruction error of a particular flow is anomalous, then that flow is also anomalous.

However, in a denial of service attack, the network characteristics may be abnormal even for normal flows, then such flows can also be marked as abnormal by the neural network. Therefore, the neural network is used only as an additional way to calculate another network characteristic to confirm the presence of a denial of service attack.

Also, after training the neural network, threshold values are calculated for all network characteristics and interaction characteristics for each of the interactions.

Network characteristics are used to determine the presence of a denial of service attack. If one of the network characteristics exceeds a given threshold, then we consider that a denial of service attack occurs. Or if the average reconstruction error obtained by the neural network exceeds a given threshold, then we consider that a denial of service attack occurs.

Interaction characteristics are used to block specific interactions. If an attack is detected using the average reconstruction error or network characteristics, all interactions in which at least one of the interaction characteristics exceeds a given threshold are blocked.

Also, any new interactions (interactions for which information is not in the tables) are also blocked.

The threshold values of interaction characteristics are calculated for all interaction characteristics except the time of the last interaction activity. The threshold values of network characteristics and interaction characteristics are stored in the threshold values of network characteristics and interaction characteristics tables, respectively.

The columns in the interaction characteristic thresholds table are the thresholds for each interaction characteristic, excluding the time of the last interaction activity. The rows in such a table are the interactions.

The columns in the network characteristic threshold values table are the threshold values for each network characteristic. However, such a table contains only one row.

The table of threshold values of interaction characteristics has the following form (Table 6)

The table of threshold values of network characteristics is as follows (Table 7)

Calculating thresholds for each characteristic of each interaction can be quite a resource-intensive task in different implementations. Therefore, thresholds are calculated once during training and periodically after training if no abnormal interactions were noted during a given threshold update period.

During long-term training and subsequent long-term use of the method for detecting and countering attacks, the table of interaction characteristics can become quite large, since rows will be created in it for each new interaction. Therefore, to avoid overflowing the table of interaction characteristics, a maximum period of interaction inactivity is specified. If the duration of interaction inactivity exceeds the maximum period of inactivity, then such interaction is deleted from the table of interaction characteristics. When deleting an inactive interaction from the table of interaction characteristics, such interaction is also deleted from the table of statistical interaction characteristics and the table of threshold values of interaction characteristics.

In a real network, during normal operation, sudden peaks of activity between certain IP addresses for various interactions may occur quite often, due to certain behavioral characteristics of a particular network. Such interactions may not be related to a network attack. Also, new interactions may constantly appear in the network and existing ones may cease. Such activity is also often not a sign of a network attack. Thus, in case of random bursts of activity or the appearance of new interactions without sufficient impact on the network as a whole, legitimate interactions will not be blocked, since the threshold values of network characteristics will not be exceeded.

Thus, in distributed denial of service attacks, as well as in attacks using IP spoofing attacks, the probability of blocking legitimate interactions is reduced, and the probability of blocking interactions related to the attack is increased, since any interactions with anomalous activity will be blocked, as well as new interactions, including interactions generated by flows using IP spoofing.

The probability of detecting and blocking reflected attacks also increases, while the probability of blocking legitimate interactions decreases. In this case, only those flows whose interactions exceed the specified thresholds will be blocked. At the same time, interactions of legitimate users with such a legitimate server will not be blocked, since the characteristics of interactions of such flows will not exceed the specified threshold values.

After training the neural network, a certain period of validation time T3 is required to calculate the threshold value of the average reconstruction error. It is assumed that T3 is greater than T1. An empty array of average reconstruction errors VR is created. Such an array is needed for the subsequent calculation of the threshold value of the average reconstruction error for the flow characteristic table. Throughout the entire validation time T3 with a period of T1, the following actions are performed:

• obtain a table of flow parameters;

• create a flow characteristics table with columns for flow parameters, network characteristics, and interaction characteristics:

Source IP address (flow parameter);

Destination IP address (flow parameter);

server port number (stream parameter);

protocol(stream parameter);

the number of TCP flags of each type (flow parameters);

number of flows per second over the network (network characteristic);

number of packets per second over the network (network characteristic);

number of bytes per second over the network (network characteristic);

number of interaction flows per second (interaction characteristic);

number of interaction packets per second (interaction characteristic);

number of interaction bytes per second (interaction characteristic);

number of packets in the flow (interaction characteristic);

number of bytes of the stream (interaction characteristic);

• fill the flow parameter columns in the flow characteristics table with values from the flow parameter table columns;

• calculate network characteristics for the entire table of flow parameters;

• calculate the interaction characteristics for each flow from the flow characteristics table;

• fill in the network characteristics and interaction characteristics columns in the flow characteristics table;

• digitize the protocol and server port parameters in the flow characteristics table with the same set of unique values as during training;

• digitize two columns of IP addresses by splitting them into 4 bytes;

• a digitized table of flow characteristics is obtained by replacing the columns of symbolic parameters from the table of flow characteristics with digitized columns of symbolic parameters;

• normalize the values in the byte and packet count columns, the eight columns of digitized IP addresses, network characteristics, interaction characteristics, and the columns with the number of different TCP flags in the digitized flow characteristics table, obtaining a table for validation;

• calculate the average reconstruction error MR using the formula

where n is the number of columns in the validation table;

m - the number of threads in the validation table;

Y _i,j - the value of the i-th parameter of the j-th flow in the validation table;

- the value of the i-th parameter of the j-th stream in the validation table reconstructed at the output of the neural network;

• supplement the VR array with the MR value;

The table of flow characteristics is as follows (Table 8)

After the validation time, the threshold value of the average reconstruction error is calculated based on the VR array. The threshold value can be calculated in various ways, but in this implementation the following sequence of actions is used:

• set the relative number of permissible exceedances of the threshold value α during validation,

• select the threshold of the average reconstruction error NT as a quantile of the array of VR values of level 1-α.

This approach allows the threshold to be calculated based on the expected acceptable relative number of false positives when detecting attacks during periods of normal network operation. However, other implementations may use other approaches to calculate the threshold.

The average neural network reconstruction error for the flow parameter table is used as an additional network characteristic in the framework of countering the attack. Accordingly, abnormal and new interactions will be blocked only if the average neural network reconstruction error for the flow parameter table exceeds NT or one of the network characteristics exceeds the threshold value.

Thus, when sudden increased activity occurs, both network characteristics and the average neural network reconstruction error can signal the presence of an attack. However, if the attacker uses a denial of service attack technique with increasing activity, then the network characteristics may not exceed the specified thresholds. In this case, the neural network can signal the detection of an attack in the event of the appearance of atypical sets of flow parameters, network characteristics, and interactions for the network. As a result, the probability of detecting various types of attacks, including attacks with increasing activity, increases.

After validation, the following is asked:

• the time period of relevance of the threshold values T4,

• time interval of maximum inactivity of T5 interaction,

• T6 interaction blocking time interval,

• time interval for detecting and blocking T7 attacks. The attack presence flag AT=0 is set.

Counteraction to attacks occurs during the entire time interval T7, while the formation of tables for training, detection of the presence of an attack and detection of anomalous interactions is performed with a period of T1 throughout the entire time interval T7. Blocking of network traffic packets of anomalous interactions occurs on the network device all the time while the flag AT=1. Note that blocking of packets is performed in such a way that such packets do not enter the protected local network, however, they are used in the method for forming flows and a table of flow parameters.

During the entire T7 time interval, the T4 time interval is taken into account for the threshold values. If no abnormal interactions were detected during the T4 time interval, then the threshold values are calculated and updated. However, if anomalies were detected, then the T4 time interval counting starts over again. Then, if any changes occur in the normal operation of the network over time, the threshold values will be adjusted and the number of false positives during normal operation will not increase. This approach to updating the threshold values is used so that, in the presence of an attack, the threshold values are not updated and do not become overstated.

During the entire time interval T7, the maximum interaction inactivity time interval T5 is taken into account for each interaction. If the duration of interaction inactivity exceeds the time interval T5, then such interaction is removed from the tables of interaction characteristics, statistical interaction characteristics, and threshold values of interaction characteristics. Removing inactive interactions allows avoiding overflow of the tables.

During the entire T7 time interval, the time of the last time it was marked as an anomaly is taken into account for each anomalous interaction. Thus, if an anomalous interaction was not marked as anomalous during the T6 time interval, then it is marked as normal. This approach allows you to get rid of unnecessary rules for blocking network traffic when an attack is completed or when using an attack with the substitution of IP addresses of the attack source. Also, if a legitimate interaction was accidentally blocked during the attack period, then such interaction will be unblocked at the end of T6, if the anomalous activity of such interaction does not repeat during this time.

During the entire time interval T7 with the period T1, the following actions are performed to detect and prevent denial of service attacks:

• obtain a table of flow parameters;

• create a flow characteristics table with columns for flow parameters, network characteristics, and interaction characteristics:

Source IP address (flow parameter);

Destination IP address (flow parameter);

server port number (stream parameter);

protocol(stream parameter);

the number of TCP flags of each type (flow parameters);

number of flows per second over the network (network characteristic);

number of packets per second over the network (network characteristic);

number of bytes per second over the network (network characteristic);

number of interaction flows per second (interaction characteristic);

number of interaction packets per second (interaction characteristic);

number of interaction bytes per second (interaction characteristic);

number of packets in the flow (interaction characteristic);

number of bytes of the stream (interaction characteristic);

• fill the flow parameter columns in the flow characteristics table with values from the flow parameter table columns;

• calculate network characteristics for the entire table of flow parameters;

• calculate the interaction characteristics for each flow from the flow characteristics table;

• fill in the network characteristics and interaction characteristics columns in the flow characteristics table;

• digitize the protocol and server port parameters in the flow characteristics table using the One-hot encoding method with the same set of unique values as during training;

• digitize the two columns of the IP address by splitting them into 4 bytes;

• a digitized table of flow characteristics is obtained by replacing the columns of symbolic parameters from the table of flow characteristics with digitized columns of symbolic parameters;

• normalize the values in the byte and packet count columns, the eight columns of digitized IP addresses, network characteristics, interaction characteristics, and the columns with the number of different TCP flags in the digitized flow characteristics table, obtaining a test table;

• calculate the average reconstruction error TRE using the formula:

where n is the number of columns in the test table;

m - the number of threads in the test table;

Y _i,j is the value of the i-th parameter of the j-th stream in the test table;

- the value of the i-th parameter of the j-th flow in the test table of flow parameters reconstructed at the output of the neural network;

• mark abnormal interactions as normal if they have not been marked as abnormal in the recent past T6;

• if the TRE value is greater than NT, or at least one of the network characteristics exceeds the threshold value from the network characteristic threshold value table, then:

mark as abnormal all interactions in the interaction characteristics table whose characteristics exceed the threshold values from the interaction characteristics threshold values table;

block all packets coming from the Internet from passing into the protected local network, for which the set of two IP addresses, protocol and server port matches one of the anomalous interactions;

• remove interactions whose inactivity time exceeded the maximum inactivity time T5 from the interaction characteristics, interaction statistics, and interaction thresholds tables;

• calculate and update the thresholds for the network characteristics and the characteristics of each interaction if the thresholds have not been updated in the last T4 time and no abnormal interactions have been observed in the last T4 time.

The process of generating test tables and detecting abnormal interactions can be performed either during the set time interval T7 or as long as protection against denial of service attacks is required.

Traffic related to anomalous interactions will be blocked on the network device as long as the AT flag is set to 1.

Note that when generating flow parameter tables, only the interaction characteristics table and the interaction statistical characteristics table are updated, while the interaction characteristic threshold values table is not updated in each T1 period. Then, no threshold values will be defined for new interactions. Thus, attack interactions will be correctly blocked even if the attack start is detected with a delay.

The technical result is achieved by using a neural network of the autoencoder type and network characteristics to detect an attack and interaction characteristics to block individual interactions. This approach increases the probability of detecting specific interactions belonging to an attack, without requiring training using a pre-prepared sample for each type of denial of service attack.

The probability of detecting various types of denial of service attacks, including attacks with increasing activity, is produced by using network characteristics and the average error of neural network reconstruction. When a denial of service attack is detected, interactions generated by various distributed attack techniques, as well as techniques using source IP spoofing, will be blocked as interactions with abnormal activity or as new interactions. This increases the probability of detecting and blocking attacks using various distributed attack techniques, as well as techniques using source IP spoofing, while reducing the probability of blocking legitimate interactions.

Using interaction characteristics to block individual interactions also increases the likelihood of detecting attacks and reduces the likelihood of blocking legitimate interactions when using the technique of reflecting an attack from a legitimate server, since data exchange with the legitimate server will be blocked only for anomalous interactions.

Implementation of the invention

The proposed method can, in general, be implemented using one computer separating the external digital data transmission network (Internet) and the local digital data transmission network (corporate network).

More preferable, from the point of view of overall operating speed, may be the implementation of the proposed method, for example, in the following software and hardware environment.

A computer with the following hardware characteristics is selected as a computing system:

processor Intel Core i7-4770 3.40 GHz;

RAM 16 GB;

SSD 240 GB hard drive,

and installed software:

Microsoft Windows 10 operating system;

Python 3.10.11;

Python modules: numpy 1.26.0, pandas 2.1.1, torch 2.2.1.

Additionally, to implement the proposed method, software (SW) is formed and installed on the computer of the computing system, ensuring the execution of calculations and actions of the method. The development and formation of such SW can be provided by a programming specialist (programmer), based on knowledge of the actions in the proposed method.

Also, to implement the method, a network device installed between the corporate network and the external data transmission network is used. The network device, which is a firewall of the ViPNet xFirewall 5 type (https://infotecs.ru/products/vipnet-xfirewall-5) is capable of passing and blocking network traffic. The computer of the computing system is connected to the network device using a local digital data transmission network. The network device has the ability to block network traffic based on the blocking rule table. At the same time, to ensure correct data exchange with the computing system and blocking of traffic according to the instructions received from the computing system, the software of the network device was modified.

Network traffic from the corporate network is fed to the input of the network device, to which network attack traffic can be mixed. To generate network attack traffic, a separate computer (test stand) is used, connected via the network and equipped with specialized software that ensures the generation of attack traffic with multiple source IP addresses, with reflection from the DNS server and with increasing activity.

Network flows in the Netflow format are formed in the network device and sent via the UDP protocol to the computing system. Within the framework of the proposed method, the network device is used exclusively for forming flows and blocking packets. At the same time, the computing system can give commands to the network device to block packets with a certain set of parameters. Network flows in the Netflow format are sent to the computing system throughout the entire operation of the method.

After preparing the software and hardware, the computer of the computing system is started in the controlled normal operation mode.

The computing system creates a network characteristics table, initially filled with zero values with one row and columns:

• number of threads per second;

• number of packets per second;

• number of bytes per second.

A table of network statistics is created, initially filled with zero values. The table of network statistics contains one row and columns:

• assessment of the mathematical expectation of the number of flows per second;

• assessment of the mathematical expectation of the number of packets per second;

• estimation of the mathematical expectation of the number of bytes per second;

• estimation of the standard deviation of the number of flows per second;

• estimation of the standard deviation of the number of packets per second;

• estimate of the standard deviation of the number of bytes per second.

An interaction characteristics table is created, initially empty. The interaction characteristics table contains the following columns:

• number of interaction threads per second;

• number of interaction packets per second;

• number of interaction bytes per second;

• number of packets in the stream;

• number of bytes in the stream;

• time of last interaction activity.

A table of statistical characteristics of interactions is created, initially empty. The table of statistical characteristics of interactions contains the following columns:

• assessment of the mathematical expectation of interaction flows per second;

• assessment of the mathematical expectation of interaction packets per second;

• estimation of the mathematical expectation of interaction bytes per second;

• assessment of the mathematical expectation of packets in a flow;

• estimation of the mathematical expectation of bytes in a stream;

• estimation of the standard deviation of interaction flows per second;

• estimation of the standard deviation of interaction packets per second;

• estimation of the standard deviation of interaction bytes per second;

• estimation of the standard deviation of packets in the flow;

• estimation of the standard deviation of bytes in the stream. A training table is created, initially empty.

The period for generating network flow parameter tables is set to T1=10 s. The time for generating the training sample T2 is set to be one day, or T2=86400 s.

On the test stand, normal traffic from the corporate network is reproduced during the time T2. At the same time, network flow parameters are accumulated in the computing system with the period T1. Thus, 8640 periods of 10 seconds will be produced. During each period T1, a table of flow parameters with columns is formed from the accumulated flow parameters:

• Source IP address,

• Destination IP address,

• server port number,

• protocol,

• the number of TCP flags of each type,

• number of packets in the stream,

• number of bytes in the stream,

• time of flow appearance.

During different periods T1, the number of rows in the flow parameter table may vary, since the intensity of network traffic may fluctuate during different periods of time. After the flow parameter table is generated, a flow characteristics table is created with the number of rows equal to the number of rows in the flow parameter table and with columns for flow parameters, network characteristics, and interaction characteristics:

• Source IP address (flow parameter);

• Destination IP address (flow parameter);

• server port number (stream parameter);

• protocol (stream parameter);

• the number of TCP flags of each type (flow parameters);

• number of flows per second over the network (network characteristic);

• number of packets per second over the network (network characteristic);

• number of bytes per second over the network (network characteristic);

• number of interaction flows per second (interaction characteristic);

• number of interaction packets per second (interaction characteristic);

• number of interaction bytes per second (interaction characteristic);

• number of packets in the flow (interaction characteristic);

• number of bytes of the stream (interaction characteristic).

In the flow characteristics table, the flow parameter columns are filled with values from the flow parameter table columns.

The network characteristics for the flow parameter table are calculated:

where FPS is the number of threads per second,

PPS - packets per second,

BPS - bytes per second,

N - the number of threads in the thread parameters table,

D - the difference between the largest and smallest time of appearance of a stream from the stream parameters table in seconds,

PN - the sum of the number of packets of all flows in the flow parameter table,

BN is the sum of the number of bytes of all streams in the stream parameter table.

The calculated values of network characteristics, FPS, PPS, BPS, are smoothed using the exponential smoothing average formula:

Where - smoothed value of the i-th network characteristic,

PS _i - calculated value of the i-th network characteristic,

PT _i is the value of the i-th network characteristic from the network characteristics table, which was recorded in the previous period.

The smoothed network characteristics are written to the network characteristics table. The values from the network characteristics table are written to the network characteristics columns in the flow characteristics table. Thus, in each row of the flow characteristics table, the same set of three network characteristics values will be written in the three columns of the network characteristics in each row.

The statistical characteristics of the network are calculated according to the formulas:

Where - estimate of the mathematical expectation for the i-th network characteristic,

EPS _i - the estimate of the mathematical expectation from the table of statistical characteristics of the network, calculated for the previous period,

- the value of the i-th network characteristic from the network characteristics table,

m - number of previous periods,

- estimate of the standard deviation for the i-th network characteristic,

DPS _i is the estimate of the standard deviation from the table of statistical characteristics of the network, calculated for the previous period.

Network characteristics and statistical network characteristics are calculated for the entire flow characteristics table in each period T1.

After that, for each row from the flow characteristics table, the interaction characteristics are calculated. To do this, the flow interaction in the row is first determined. The interaction is determined using a set of two IP addresses, a server port, and a protocol. If the interaction is not in the interaction characteristics table, then:

• a new interaction is created in the interaction characteristics table;

• the values of the number of streams, packets and bytes per second are set to zero;

• the values of the number of packets and bytes of interaction are set equal to the values of the number of packets and bytes in the flow under consideration.

Otherwise, if the interaction is in the interaction characteristics table, then:

• the values of the number of packets and bytes are set equal to the values of the number of packets and bytes in the flow under consideration;

• other interaction characteristics for the flow under consideration are calculated using the formulas:

where IFPS is the number of interaction flows per second;

IPPS - number of interaction packets per second;

IBPS - Interaction Bytes Per Second;

IPN - number of interaction packets;

IBN - number of interaction bytes;

t - the time of the last interaction activity from the table of interaction characteristics for the previous period;

- the time of appearance of the flow in question.

After this, the interaction characteristics of the flow under consideration are smoothed using the formula:

Where - smoothed value of the i-th characteristic of flow interaction,

IPS _i - the calculated value of the i-th interaction characteristic of the flow,

IPT _i is the value of the i-th interaction characteristic of a flow from the interaction characteristics table, which was recorded during the previous appearance of the flow.

The moment of time of the last activity of the flow is set equal to the moment of time of the appearance of the flow in question. After that, the values in the table of interaction characteristics are updated with the calculated and smoothed values. The columns of interaction characteristics in the table of flow characteristics in the row in question are also filled with the smoothed values of the interaction characteristics.

Statistical characteristics of interaction are calculated according to the formulas:

Where - estimate of the mathematical expectation for the i-th interaction characteristic,

IEPS _i - the estimate of the mathematical expectation from the table of statistical characteristics of interaction, calculated for the previous interaction flow,

- the value of the i-th interaction characteristic from the table of interaction characteristics,

n is the total number of all previous interaction flows,

- estimate of the standard deviation for the i-th interaction characteristic,

IDPS _i is the estimate of the standard deviation from the table of statistical characteristics of interaction, calculated for the previous flow.

The calculated statistical characteristics of the interaction are written to the table of statistical characteristics of interactions in the row of the interaction under consideration. If such an interaction is not in the table of statistical characteristics, then it is created in a new row of the table and filled with the calculated values.

The interaction characteristics and statistical interaction characteristics are sequentially calculated for each row in the flow characteristics table in each period T1.

Thus, the interaction characteristics are calculated for each row in the network characteristics table. After filling the flow characteristics table, the training table is supplemented with rows of the flow characteristics table. Accordingly, in each period T1, the training table is supplemented with the flow characteristics table. In this case, in each new period, the flow characteristics table is created and filled with values anew.

At the end of the training time, the training table contains 30236224 rows with flow parameters. At the same time, the interaction characteristics table contains 1451 interactions (Table 9).

An example of several rows of the interaction characteristics table after completion of training with anonymized IP addresses looks like this (Table 10).

After the T2 training time is completed, the protocol and server port columns are digitized in the training table using the One-Hot-Encoding method. In this case, the sets of unique values for the protocol and server port that were used during digitization must be saved for further use of such sets when digitizing the validation and test tables. The IP address columns are digitized by splitting the IP address into 4 bytes.

In the digitized training table, the byte and packet count columns, the digitized IP address columns, network characteristics, interaction characteristics, and the columns with the number of different TCP flags are normalized using the Yeo-Johnson transform.

Using the pytorch module, a neural network of the autoencoder type is created, which has

• the dimension of the input layer and the output layer is equal to the number of columns in the normalized table for training 1728,

• one hidden layer with dimension 48.

The neural network is trained using a normalized training table, and during training, the error in reconstructing the flow parameters R is minimized, which is calculated using the formula:

where n is the number of columns in the normalized table for training;

Y _i,j is the value of the i-th column of the j-th stream in the normalized table for training;

- the reconstructed value of the i-th column of the j-th stream in the normalized table for training at the output of the neural network.

After training the neural network, threshold values are calculated for each network characteristic using the formula:

Where - assessment of the mathematical expectation of the i-th network characteristic from the table of statistical network characteristics;

- assessment of the standard deviation of the i-th network characteristic from the table of statistical network characteristics.

The obtained threshold values of network characteristics are recorded in the table of threshold values of network characteristics.

Threshold values for each characteristic of each interaction are also calculated using the formula

Where - estimate of the mathematical expectation of the i-th characteristic of the j-th interaction from the table of statistical characteristics of interactions;

- an estimate of the standard deviation of the i-th characteristic of the j-th interaction from the table of statistical characteristics of interactions.

The obtained threshold values of interaction characteristics are recorded in the table of threshold values of interaction characteristics.

The table of threshold values of network characteristics after completion of training looks like this (Table 11)

Several lines of the table of threshold values of interaction characteristics after completion of training look like this (Table 12, with anonymized IP addresses).

The duration of the time interval for validation T3 is set to be one day, or T3=86400 s. During the time T3, network traffic from the corporate network is reproduced on the test bench. An array of VR values is created, initially empty. During the entire time interval T3 with a period of T1, network flow parameters are accumulated. As soon as the period T1 passes, a table of flow parameters with columns is formed from the accumulated flow parameters:

• Source IP address,

• Destination IP address,

• server port number,

• protocol,

• the number of TCP flags of each type,

• number of packets in the stream,

• number of bytes in the stream,

• time of flow appearance.

The number of rows in the flow parameter table may vary over different periods of time because the intensity of network traffic may fluctuate during different periods of time.

After the flow parameter table is generated, a flow characteristics table is created with a number of rows equal to the number of rows in the flow parameter table and with columns for flow parameters, network characteristics, and interaction characteristics:

• Source IP address (flow parameter);

• Destination IP address (flow parameter);

• server port number (stream parameter);

• protocol (stream parameter);

• the number of TCP flags of each type (flow parameters);

• number of flows per second over the network (network characteristic);

• number of packets per second over the network (network characteristic);

• number of bytes per second over the network (network characteristic);

• number of interaction flows per second (interaction characteristic);

• number of interaction packets per second (interaction characteristic);

• number of interaction bytes per second (interaction characteristic);

• number of packets in the flow (interaction characteristic);

• number of bytes of the stream (interaction characteristic).

In the flow characteristics table, the flow parameter columns are filled with values from the flow parameter table columns.

The network characteristics are calculated for the flow parameter table using formulas (5)-(7). The calculated values of the network characteristics, FPS, PPS, BPS, are smoothed using formula (8). The smoothed network characteristics are written to the network characteristics table.

The table of flow characteristics records the values from the table of network characteristics in the columns of the network characteristics. Thus, in each row of the table of flow characteristics, the same set of three values of network characteristics will be recorded in the three columns of the network characteristics in each row. The statistical characteristics of the network are calculated according to formulas (9)-(10).

Network characteristics and statistical network characteristics are calculated for the entire flow characteristics table in each period T1.

After that, for each row in the flow characteristics table, the interaction characteristics are calculated. To do this, the flow interaction in the row is first determined. The interaction is determined using a set of two IP addresses, a server port, and a protocol. If the interaction is not in the interaction characteristics table, then:

• a new interaction is created in the interaction characteristics table;

• the values of the number of streams, packets and bytes per second are set to zero;

• the values of the number of packets and bytes of interaction are set equal to the values of the number of packets and bytes in the flow under consideration.

Otherwise, if the interaction is in the interaction characteristics table, then:

• the values of the number of packets and bytes are set equal to the values of the number of packets and bytes in the flow under consideration;

• other interaction characteristics for the flow under consideration are calculated using formulas (11)–(13).

After which the interaction characteristics of the flow under consideration are smoothed using formula (14).

The moment of time of the last activity of the flow is set equal to the moment of time of the appearance of the flow in question. After that, the values in the table of interaction characteristics are updated with the calculated and smoothed values. The columns of interaction characteristics in the table of flow characteristics in the row in question are also filled with the smoothed values of the interaction characteristics.

Statistical characteristics of interaction are calculated according to formulas (15)-(16). The calculated statistical characteristics of interaction are recorded in the table of statistical characteristics of interactions in the row of the interaction under consideration. If such interaction is not in the table of statistical characteristics, then it is created in a new row of the table and filled with the calculated values.

The interaction characteristics and statistical interaction characteristics are sequentially calculated for each row in the flow characteristics table at each period T1.

After the flow characteristic table has been filled with values, in each T1 period during the validation time, such a table is digitized. The protocol and server port columns are digitized using the One-Hot-Encoding method. In this case, the unique values for the protocol and server port are exactly the same as when digitizing the table for training. The IP address columns are digitized by splitting the IP address into 4 bytes.

In the digitized flow characteristics table, the columns of bytes and packets, the digitized columns of IP addresses, network characteristics, interaction characteristics, and the columns with the number of different TCP flags are normalized using the Yeo-Johnson transform. Thus, in each T1 period, we obtain a validation table.

We calculate the average reconstruction error MR using the formula

where n is the number of columns in the validation table;

m - the number of threads in the validation table;

Y _i,j is the value of the i-th parameter of the j-th stream in the validation table;

- the value of the i-th parameter of the j-th stream in the validation table reconstructed at the output of the neural network.

We supplement the VR array with the MR value in each T1 period.

Once the validation period is completed, the threshold value NT is chosen to be the quantile of the 0.995 level of values from the VR array. Thus, the threshold value NT is equal to 0.0022.

Next, the following questions are asked:

• the period of updating the threshold values T4, equal to one day, or T4=86400 s,

• the time interval of inactivity of interaction T5, equal to one day, or T5=86400 s,

• the time interval for blocking interaction T6, equal to five minutes, or T6=300 s,

• the time interval for detecting and countering attacks T7, equal to three hours, or T4=10800 s.

The method was tested using legitimate network traffic from a corporate network.

In the first test, only legitimate traffic was used as input, so that the number of type II errors could be estimated in the absence of an attack.

The total duration of long-term testing on legitimate traffic did not exceed 120 minutes,

In the second test, the network traffic for one hour of legitimate network activity was fed to the input, after which a denial of service attack using many different IP addresses was mixed into this legitimate traffic. This test allows us to estimate the probability of detecting and blocking attack flows, as well as the probability of blocking legitimate flows during distributed denial of service attacks and attacks with source IP address substitution. Such attacks are characterized by the use of many different IP addresses. In the case of distributed attacks, this is due to the presence of a distributed botnet, and when using the source IP address substitution technique, it is due to the generation of random IP addresses.

In the third test, the network traffic for one hour of legitimate network activity was fed to the input, after which a denial-of-service attack reflected from a legitimate DNS server was mixed into this legitimate traffic. This test allows us to estimate the number of type I and type II errors in countering reflected denial-of-service attacks using the technique of reflection from a legitimate server.

In the fourth test, one hour of network traffic with legitimate network activity was fed to the input, after which a denial of service attack with increasing activity was mixed into this legitimate traffic. In this case, the activity increased linearly until it reached the intensity of the attack from the first test. In this test, the IP addresses of the attack source were generated randomly, similar to the attack from the first test.

The testing of the method on different attacks was carried out separately, but according to the same order of actions. At the beginning of the testing to counter each of the three types of attacks, only legitimate traffic was fed to the input of the computing system, after which, with a delay of approximately 90 minutes, the attack traffic was mixed into the normal traffic. The duration of each type of attack did not exceed 25 minutes. At the end of the attack, legitimate traffic was fed to the input for some time.

The total duration of each test, taking into account legitimate activity before and after the network attack, did not exceed 200 minutes.

During the entire time interval T7 with the period T1, the parameters of network flows are accumulated. As soon as the period T1 passes, a table of flow parameters with columns is formed from the accumulated flow parameters:

• Source IP address,

• Destination IP address,

• server port number,

• protocol,

• the number of TCP flags of each type,

• number of packets in the stream,

• number of bytes in the stream,

• time of flow appearance.

After the flow parameter table is formed, a flow characteristics table is created with the number of rows equal to the number of rows in the flow parameter table and with columns of flow parameters, network characteristics, and interaction characteristics:

• Source IP address (flow parameter);

• Destination IP address (flow parameter);

• server port number (stream parameter);

• protocol (stream parameter);

• the number of TCP flags of each type (flow parameters);

• number of flows per second over the network (network characteristic);

• number of packets per second over the network (network characteristic);

• number of bytes per second over the network (network characteristic);

• number of interaction flows per second (interaction characteristic);

• number of interaction packets per second (interaction characteristic);

• number of interaction bytes per second (interaction characteristic);

• number of packets in the flow (interaction characteristic);

• number of bytes of the stream (interaction characteristic).

In the flow characteristics table, the flow parameter columns are filled with values from the flow parameter table columns.

The network characteristics for the flow parameter table are calculated using formulas (5)-(7). The calculated values of the network characteristics, FPS, PPS, BPS, are smoothed using formula (8).

Smoothed network characteristics are written to the network characteristics table. The values from the network characteristics table are written to the flow characteristics table in the network characteristics columns. Thus, in each row of the flow characteristics table, the same set of three network characteristics values will be written in the three columns of the network characteristics in each row. The statistical network characteristics are calculated according to formulas (9)-(10).

Network characteristics and statistical network characteristics are calculated for the entire flow characteristics table in each period T1.

After that, for each row from the flow characteristics table, the interaction characteristics are calculated. To do this, the flow interaction in the row is first determined. The interaction is determined using a set of two IP addresses, a server port, and a protocol. If the interaction is not in the interaction characteristics table, then:

• a new interaction is created in the interaction characteristics table;

• the values of the number of streams, packets and bytes per second are set to zero;

• the values of the number of packets and bytes of interaction are set equal to the values of the number of packets and bytes in the flow under consideration.

Otherwise, if the interaction is in the interaction characteristics table, then:

• the values of the number of packets and bytes are set equal to the values of the number of packets and bytes in the flow under consideration;

• other interaction characteristics for the flow under consideration are calculated using formulas (11)–(13).

After which the interaction characteristics of the flow under consideration are smoothed using formula (14).

The moment of time of the last activity of the flow is set equal to the moment of time of the appearance of the flow in question. After that, the values in the table of interaction characteristics are updated with the calculated and smoothed values. The columns of interaction characteristics in the table of flow characteristics in the row in question are also filled with the smoothed values of the interaction characteristics.

Statistical characteristics of interaction are calculated according to formulas (15)-(16). The calculated statistical characteristics of interaction are recorded in the table of statistical characteristics of interactions in the row of the interaction under consideration. If such interaction is not in the table of statistical characteristics, then it is created in a new row of the table and filled with the calculated values.

The interaction characteristics and statistical interaction characteristics are sequentially calculated for each row in the flow characteristics table at each period T1.

After the flow characteristics table has been filled with values, in each T1 period, during the T7 time interval, such a table is digitized. The protocol and server port columns are digitized using the One-Hot-Encoding method. In this case, the unique values for the protocol and server port are exactly the same as when digitizing the table for training. The IP address columns are digitized by splitting the IP address into 4 bytes.

In the digitized flow characteristics table, we normalize the columns of bytes and packets, digitized columns of IP addresses, network characteristics, interaction characteristics, and columns with the number of different TCP flags using the Yeo-Johnson transform. Thus, at each T1 period, we obtain a test table.

We calculate the average reconstruction error TRE using the formula

where n is the number of columns in the test table;

m - the number of threads in the test table;

Y _i,j is the value of the i-th parameter of the j-th stream in the test table;

- the value of the i-th parameter of the j-th stream in the test table reconstructed at the output of the neural network.

After calculating the average reconstruction error for the test table in each period T1, TRE is compared with the threshold value NT. Also, each of the network characteristics is compared with the current threshold value from the network characteristic threshold value table.

If TRE is greater than the threshold or one of the network characteristics is greater than the threshold, then:

• mark as abnormal all interactions in the interaction characteristics table whose characteristics exceed the threshold values from the interaction characteristics threshold values table;

• send a command from the computing system to the network device to create rules for blocking network packets related to anomalous interactions with a timeout equal to T6; thus, if during the T6 time interval the interaction is no longer marked as anomalous, then the rule for blocking packets will be deleted.

Also, in each period T1, the inactivity time of each interaction is checked. If the interaction was inactive during the time interval T5, then such interaction is removed from the tables of interaction characteristics, statistical characteristics of interactions, and threshold values of interactions.

The time since the last update of the threshold values is checked; if the threshold values have not been updated for more than the time interval T4 (day), and no abnormal interactions have been detected over the last day, then the threshold values of the network characteristics are updated according to formula (18), and the threshold values of the interaction characteristics are updated according to formula (19).

After which, if necessary, or if time T7 has not expired, a transition to a new period occurs.

After testing the method on three different denial of service attacks, the following results were obtained (Table 13)

When testing the method on traffic without network attacks, not a single legitimate flow was blocked out of 2,527,575 legitimate flows that were processed during testing.

When testing the method on traffic with a denial of service attack using multiple source IP addresses, 8165 flows were incorrectly blocked out of 4147391 legitimate flows (4139226 unblocked + 8165 blocked in total). At the same time, 13949172 attack flows were blocked out of 13957904 attack flows. Thus, when using a denial of service attack using multiple source IP addresses, only 0.19% of legitimate flows were incorrectly blocked and 99.94% of attack flows were correctly blocked.

The 99.94% rate can be interpreted as the probability of detecting an attack and, taking into account the blocking of flows, protecting against the attack.

When testing the method on traffic with a denial of service attack using the reflection technique from a legitimate DNS server, 9042 flows out of 4161614 legitimate flows were incorrectly blocked. At the same time, 12749374 flows out of 12758004 attack flows were blocked at the same time. Thus, when using a denial of service attack using the reflection technique from a legitimate DNS server, only 0.22% of legitimate flows were incorrectly blocked and 99.93% of attack flows were correctly blocked.

When testing the method on traffic with an attack type of denial of service with increasing activity, 4477 out of 4142914 legitimate flows were incorrectly blocked. At the same time, 684918 flows out of 697459 attack flows were blocked at the same time. Thus, when using the technique of attack type of denial of service with increasing activity, only 0.11% of legitimate flows were incorrectly blocked and 98.20% of attack flows were correctly blocked.

According to the testing results, the probability of detection and protection for three types of denial of service attacks is higher than that achieved in the prototype (85-96%, depending on the number of prepared examples). At the same time, special preparation of examples is not required to implement the proposed method.

Claims (146)

A method for detecting and countering denial of service attacks using a computing system connected to an external digital data transmission network and a local digital data transmission network and including at least one computer that has an installed operating system and application programs that allow: receive, process and store multidimensional network traffic data; ensure the formation, functioning and training of a neural network of the autoencoder type; block network packets passing from the external digital data transmission network to the local digital data transmission network; a method which consists in the fact that: start the computer in controlled normal operation mode; create a network characteristics table, initially filled with zero values; create a table of statistical characteristics of the network, initially filled with zero values; create a table of interaction characteristics, initially empty; create a table of statistical characteristics of interactions, initially empty; create a table for training, initially empty; set the period T1 for generating tables of flow parameters; set the time interval T2>T1 for forming the training table; receive and process data from network streams; supplement the table for training throughout the entire time interval T2, for this, the following actions are performed with the period T1: receive a table of flow parameters; create a flow characteristics table, initially empty, with a number of rows equal to the number of rows in the flow parameters table, and with columns for flow parameter values, network characteristics, and interaction characteristics; fill in the flow parameter columns in the flow characteristics table with values from the columns of the flow parameter table; calculate network characteristics for the flow parameter table; smooth out the values of network characteristics; write the smoothed network characteristics into the network characteristics table; write down in the table of flow characteristics the values of the network characteristics from the table of network characteristics in columns for each of the network characteristics in the form of a set of identical values in each row; calculate the values of statistical characteristics of the network; record the values of the statistical characteristics of the network in the table of statistical characteristics of the network; fill in the values in the interaction characteristic columns in the flow characteristics table by performing the following actions for each row of the flow characteristics table: define the interaction of a stream using its two IP addresses, protocol and server port; if the interaction to which the flow belongs is missing from the interaction characteristics table, then: create a new interaction in the interaction characteristics table; set the values for the number of streams, packets, and bytes per second to zero in the interaction characteristics table; set the values of the number of packets and bytes equal to the number of packets and bytes in the stream in question; otherwise, if the interaction is in the interaction characteristics table, then: calculate the interaction characteristics for the flow under consideration; smooth out the values of interaction characteristics; set the time of the last interaction activity equal to the time of the flow appearance; write the smoothed values of the interaction characteristics into the interaction characteristics table; calculate statistical characteristics of the interaction; write down the values of statistical characteristics of interaction in the table of statistical characteristics of interactions; write down in the columns of the interaction characteristics of the flow characteristics table the values from the interaction characteristics table for the flow under consideration; supplement the training table with rows from the flow characteristics table; at the end of the time interval T2, the symbolic values in the protocol and server port columns in the training table are digitized; digitize IP addresses by dividing the address into 4 byte values in a training table; normalize the values in the byte and packet count columns, the eight digitized IP address columns, network characteristics, interaction characteristics, and the columns with the number of different TCP flags in the digitized training table to obtain a normalized training table; form a neural network of the autoencoder type, which has the dimension of the input layer and the output layer is equal to the number of columns in the normalized table for training, one hidden layer with a dimension less than the number of columns in the normalized training table; train a neural network using a normalized table for training, and during training, minimize the error in reconstructing the flow parameters R, which is calculated using the formula where n is the number of columns in the normalized table for training; Y _i,j is the value of the i-th column of the j-th stream in the normalized table for training; - the reconstructed value of the i-th column of the j-th stream in the normalized table for training at the output of the neural network; form a table of threshold values of network characteristics, initially empty; calculate threshold values for each network characteristic TN _i using the formula Where - assessment of the mathematical expectation of the i-th network characteristic from the table of statistical network characteristics; - assessment of the standard deviation of the i-th network characteristic from the table of statistical network characteristics; record the network characteristic threshold values in the network characteristic threshold value table; calculate the threshold values for each interaction characteristic TC _i,j using the formula Where - estimate of the mathematical expectation of the i-th characteristic of the j-th interaction from the table of statistical characteristics of interactions; - estimate of the standard deviation of the i-th characteristic of the j-th interaction from the table of statistical characteristics of interactions; record the threshold values of interaction characteristics in the table of threshold values of interaction characteristics; set the duration of the time interval for validation T3>T1; create a one-dimensional array of VR reconstruction mean error values, initially empty; throughout the entire time interval T3 with period T1, the following actions are performed: receive a table of flow parameters; create a flow characteristics table, initially empty, with a number of rows equal to the number of rows in the flow parameters table, and with columns for flow parameter values, network characteristics, and interaction characteristics; in the flow characteristics table, fill in the flow parameter columns with values from the columns of the flow parameter table; calculate network characteristics for the flow parameter table; smooth out the values of network characteristics; write the smoothed network characteristics into the network characteristics table; write down in the table of flow characteristics the values of the network characteristics from the table of network characteristics in columns for each of the network characteristics in the form of a set of identical values in each row; calculate the values of statistical characteristics of the network; record the values of the statistical characteristics of the network in the table of statistical characteristics of the network; populate the values in the interaction characteristic columns of the flow characteristic table by performing the following steps for each row of the flow characteristic table: define the interaction of a stream using its two IP addresses, protocol and server port; if the interaction to which the flow belongs is missing from the interaction characteristics table, then: create a new interaction in the interaction characteristics table; set the values for the number of streams, packets, and bytes per second to zero in the interaction characteristics table; set the values of the number of packets and bytes equal to the number of packets and bytes in the stream in question; otherwise, if the interaction is in the interaction characteristics table, then: calculate the interaction characteristics for the flow under consideration; smooth out the values of interaction characteristics; set the time of the last interaction activity equal to the time of the flow appearance; write the smoothed values of the interaction characteristics into the interaction characteristics table; calculate statistical characteristics of the interaction; write down the values of statistical characteristics of interaction in the table of statistical characteristics of interactions; write down in the columns of the interaction characteristics of the flow characteristics table the values from the interaction characteristics table for the flow under consideration; digitize the symbolic values in the protocol and server port columns in the flow characteristics table; digitize IP addresses by dividing the address into 4 byte values in the flow characteristic table; normalize in the digitized flow characteristics table the values in the columns of the number of bytes and packets, in the eight columns of the digitized columns of IP addresses, network characteristics, interaction characteristics, and also in the columns with the number of different TCP flags; obtain a validation table after normalizing the digitized flow characteristic table; feed the validation table to the autoencoder input; calculate the average reconstruction error MR using the formula where n is the number of columns in the validation table; m - the number of threads in the validation table; Y _i,j is the value of the i-th parameter of the j-th stream in the validation table; - the value of the i-th parameter of the j-th stream in the validation table reconstructed at the output of the neural network; complement the VR array with the MR value; calculate the threshold value NT for the VR array; set the time interval for the relevance of threshold values T4>T1; set the time interval of inactivity of interaction T5>T1; set the time interval for blocking interaction T6>T1; set the time interval for detecting and countering attacks T7>T1; During the entire period of time, to detect and counteract T7 attacks with a period of T1, the following actions are performed: Stage A: obtain a table of flow parameters; create a flow characteristics table, initially empty, with a number of rows equal to the number of rows in the flow parameters table, and with columns for flow parameter values, network characteristics, and interaction characteristics; fill in the flow parameter columns in the flow characteristics table with values from the columns of the flow parameter table; calculate network characteristics for the flow parameter table; smooth out the values of network characteristics; write the smoothed network characteristics into the network characteristics table; write down in the table of flow characteristics the values of the network characteristics from the table of network characteristics in columns for each of the network characteristics in the form of a set of identical values in each row; calculate the values of statistical characteristics of the network; record the values of the statistical characteristics of the network in the table of statistical characteristics of the network; populate the values in the interaction characteristic columns of the flow characteristic table by performing the following steps for each row of the flow characteristic table: define the interaction of a stream using its two IP addresses, protocol and server port; if the interaction to which the flow belongs is missing from the interaction characteristics table, then: create a new interaction in the interaction characteristics table; set the values for the number of streams, packets, and bytes per second to zero in the interaction characteristics table; set the values of the number of packets and bytes equal to the number of packets and bytes in the stream in question; otherwise, if the interaction is in the interaction characteristics table, then: calculate the interaction characteristics for the flow under consideration; smooth out the values of interaction characteristics; set the time of the last interaction activity equal to the time of the flow appearance; write the smoothed values of the interaction characteristics into the interaction characteristics table; calculate statistical characteristics of the interaction; write down the values of statistical characteristics of interaction in the table of statistical characteristics of interactions; write down in the columns of the interaction characteristics of the flow characteristics table the values from the interaction characteristics table for the flow under consideration; digitize the symbolic values in the protocol and server port columns in the flow characteristics table; digitize IP addresses by dividing the address into 4 byte values in the flow characteristic table; normalize in the digitized flow characteristics table the values in the columns of the number of bytes and packets, in the eight columns of the digitized columns of IP addresses, network characteristics, interaction characteristics, and columns with the number of different TCP flags; obtain a test table after normalizing the flow characteristics table; a test table is fed to the autoencoder input; calculate the average reconstruction error TRE for the test table using the formula where n is the number of columns in the test table; m - the number of threads in the test table; Y _i,j is the value of the i-th parameter of the j-th stream in the test table; - the value of the i-th parameter of the j-th flow in the test table of flow parameters reconstructed at the output of the neural network; mark anomalous interactions as normal if they were not marked as anomalous during the T6 time interval; if the TRE value is greater than NT, or at least one of the network characteristics exceeds a threshold from the network characteristic threshold table, then: mark as abnormal all interactions in the interaction characteristics table whose characteristics exceed the threshold values from the interaction characteristics threshold values table; block all packets from the external digital data transmission network from passing into the local digital data transmission network, for which the set of two IP addresses, protocol and server port matches one of the anomalous interactions; remove interactions whose inactivity time has exceeded the T5 time interval from the interaction characteristics, interaction statistics, and interaction thresholds tables; calculate and update the threshold values for the network characteristics and the characteristics of each interaction if the threshold values have not been updated during the time interval T4 and no abnormal interactions have been observed during the time interval T4; if necessary, proceed to stage A.