WO2020168756A1 - Cluster log feature extraction method, and apparatus, device and storage medium - Google Patents

Abstract

A cluster log feature extraction method, and an apparatus, a device and a storage medium. The method comprises: a flume client collecting a log of a server cluster, and sending same to a database (S10); performing data cleaning on log data to screen out original data (S30); extracting feature values, comprising the mean value, effective value, peak value, root amplitude, waveform index, impulse index and kurtosis index, of the original data (S50); and respectively performing operation of a Pearson correlation coefficient on the extracted feature values and the original data, comparing calculated correlation coefficients with a correlation threshold, regarding correlation coefficients higher than the correlation threshold as being valid data, and regarding correlation coefficients lower than the correlation threshold as being invalid data and removing same (S70). Valid information of production data of each host in a server cluster can be effectively screened out, and feature values of the production data are extracted from the valid information, thereby facilitating failure prediction and failure classification of a production system and reducing the occurrence of production accidents.

Description

Cluster log feature extraction method, device, equipment and storage medium

This application claims the priority rights of Chinese Patent Application No. 201910123928.1 filed on February 19, 2019. The entire contents of the above cases are incorporated herein by reference.

Technical field

This application relates to base frame operation and maintenance, and specifically to a method, device, equipment, and storage medium for extracting cluster log features.

Background technique

In the era of explosive growth of information, it has become a reality for file size and data scale to reach terabytes or even petabytes. The number of cluster storage system nodes has reached the number of clusters with 64 nodes. Managing such a large cluster system has become a severe problem for data centers. challenge. It is especially important to track the running status of cluster nodes in time and accurately locate node error information. In the actual operation of the cluster storage system, there is currently a commonly used method of cluster storage system log management, which can send system logs regularly or in real time to realize the centralized transmission of logs, but the logs are not analyzed and managed, and the entire cluster cannot be globally understood. The operating status of the storage system cannot quickly locate the error message. However, as the number of cluster nodes increases, the management of the cluster system becomes more and more complicated. From the massive server data, it is particularly important to extract the characteristics that can reflect the server performance, accurately locate the potential failures of the cluster nodes, and do the corresponding performance detection in advance.

Summary of the invention

In order to solve the above problems, this application provides a cluster log feature extraction method, which is applied to electronic equipment, including the following steps: collect the log of the server cluster through the flume client and send it to the Hbase database, where the flume client processes multiple Agent processes Corresponding to the log of each server in the collection server cluster, the Agent regularly collects the log data on the corresponding server and sends it to the Hbase database through the API interface; uses Hadoop to clean the log data and filter out the original data. The original data is at least Including server disk occupancy rate, memory usage rate, cpu occupancy rate, and business interface call volume; extract the feature value of the original data including mean value, effective value, peak value, square root amplitude, waveform index, impulse index, and kurtosis index; Use the Pearson correlation coefficient to filter out the effective features, and perform the calculation of the Pearson correlation coefficient between the extracted feature values and the original data, and compare the calculated correlation coefficient with the correlation threshold. If the correlation threshold is higher than the correlation threshold, it is considered valid data , Below the correlation threshold, it is considered invalid data and removed.

This application also provides a cluster log feature extraction device, including: a log collection module, a data cleaning module, a feature extraction module, and an effective feature screening module. The log collection module is used to collect logs of the server cluster through the flume client and send it to Hbase database, where the flume client collects the logs of each server in the server cluster through multiple agent processes. The agent regularly collects the log data on the corresponding server and sends it to the Hbase database through the API interface; the data cleaning module is used for Use Hadoop to clean the log data to filter out the original data. The original data includes at least the server disk occupancy rate, memory usage rate, cpu occupancy rate, and the amount of business interface calls; the feature extraction module is used to perform average and effective Extraction of feature values of value, peak value, square root amplitude, waveform index, impulse index, and kurtosis index; effective feature selection module uses Pearson correlation coefficient to filter out effective features: Pearson correlation between the extracted feature values and the original data The calculation of the coefficient is compared with the correlation threshold based on the calculated correlation coefficient. If the correlation coefficient is higher than the correlation threshold, it is valid data, and if the correlation threshold is lower, the data is invalid, and it will be eliminated.

The application also provides an electronic device, the electronic device comprising: a memory and a processor, the memory stores a cluster log feature extraction program, and the cluster log feature extraction program is executed by the processor to implement the following steps: The flume client collects the logs of the server cluster and sends them to the Hbase database. The flume client collects the logs of each server in the server cluster through multiple agent processes. The agent regularly collects the log data on the corresponding server and passes the API The interface is sent to the Hbase database; the log data is cleaned using Hadoop, and the original data is filtered out. The original data includes at least the server disk occupancy rate, memory utilization rate, cpu occupancy rate, and business interface call volume; the original data includes average, Extraction of eigenvalues of effective value, peak value, square root amplitude, waveform index, impulse index, kurtosis index; use Pearson correlation coefficient to filter out effective features, and perform Pearson correlation coefficient calculation on the extracted eigenvalues and original data , According to the calculated correlation coefficient and the correlation threshold value, it is considered as valid data if it is higher than the correlation degree threshold value, and it is considered invalid data if it is lower than the correlation degree threshold value and removed.

The present application also provides a computer non-volatile readable storage medium, the computer non-volatile readable storage medium stores a computer program, the computer program includes program instructions, and when the program instructions are executed by a processor, Realize the above-mentioned cluster log feature extraction method.

This application can effectively filter out the effective information of the production data of each host in the server cluster, and extract the characteristic values of the production data from the effective information, which facilitates the failure prediction and classification of the production system and reduces the occurrence of production accidents.

Description of the drawings

By describing its embodiments in conjunction with the following drawings, the above-mentioned features and technical advantages of the present application will become clearer and easier to understand.

FIG. 1 is a schematic flowchart of a cluster log feature extraction method according to an embodiment of the present application;

2 is a schematic diagram of the hardware architecture of an electronic device according to an embodiment of the present application;

Fig. 3 is a block diagram of a cluster log feature extraction program according to an embodiment of the present application;

FIG. 4 is a unit structure diagram of a log collection module of an embodiment of the present application;

FIG. 5 is a unit structure diagram of a feature extraction module of an embodiment of the present application;

FIG. 6 is a unit structure diagram of a data cleaning module according to an embodiment of the present application;

Figure 7 is a schematic diagram of Flume's Agent process reading data.

detailed description

Hereinafter, embodiments of the cluster log feature extraction method, device and storage medium described in this application will be described with reference to the accompanying drawings. A person of ordinary skill in the art may realize that the described embodiments can be modified in various different ways or combinations thereof without departing from the spirit and scope of the present application. Therefore, the drawings and descriptions are illustrative in nature and are not used to limit the scope of protection of the claims. In addition, in this specification, the drawings are not drawn to scale, and the same reference numerals denote the same parts.

As shown in Figure 1, the cluster log feature extraction method of this embodiment includes the following steps:

In step S10, the logs of the server cluster are collected by the flume (distributed mass log collection, aggregation and transmission system) client, and sent to the Hbase database server. Flume takes the Agent process as the smallest independent operation unit, and an Agent process is a complete data collection tool. As shown in Figure 7, the Agent includes components Source (data collection component), Channel (transit temporary storage), and Sink. The three form an Agent. The source collects data from the server and passes it to the Channel. The Channel saves the data passed by the Source component. Event (data unit), Sink reads and removes the Event from the Channel, and passes the Event to the background. Flume collects log data corresponding to each server through multiple agents. Set up an Agent for each server, collect log data on the corresponding server regularly and send it to the background through the API interface.

Step S30: Use Hadoop (distributed system infrastructure) to perform data cleaning on the log data to filter out the original data, where the original data includes at least the server disk occupancy rate, the memory usage rate, the cpu occupancy rate, and the amount of service interface calls.

In step S50, the original data is extracted with feature values including average value, effective value, peak value, root square amplitude, waveform index, impulse index, and kurtosis index.

Step S70: Use the Pearson correlation coefficient to filter out the effective features: perform Pearson correlation coefficient calculations on the extracted feature values and the original data, and compare the calculated correlation coefficient with the correlation threshold. If it is higher than the correlation threshold, it is considered Data is valid, and data below the correlation threshold is considered invalid data and removed.

Further, the Laida criterion is used to remove data with gross errors in data cleaning, including the following steps:

及剩余误差

其中，x _i为单次Agent采集的日志数据； For log data x ₁ ,x ₂ ...,x _n , calculate the arithmetic average

And residual error

Among them, x _i is the log data collected by a single agent;

Calculate the standard deviation S _x ,

If the residual error v _b of x _b in the log data (1≤b≤n), it satisfies the formula

It is considered that x _b is a singular value with a gross error value, and the singular value is eliminated.

Furthermore, using Laida's law can effectively identify the singular values of the production data, but will produce null values for the deleted data. Therefore, the singular value of the identified log data is replaced with the median value to realize the preprocessing of the production data information. The median value means that the variable values x ₁ , x ₂ ..., x _n are arranged in order of magnitude to form a sequence, and the variable value in the middle of the variable sequence is called the median value.

In an optional embodiment, feature value extraction including mean value, effective value, peak value, root square amplitude, waveform index, impulse index, and kurtosis index is performed on the original data, wherein,

The effective value is calculated using the following formula:

The peak value is calculated using the following formula: X _p =max(x _i )

The square root amplitude is calculated using the following formula:

The waveform index is calculated using the following formula:

The impulse index is calculated using the following formula:

The kurtosis index is calculated using the following formula:

为采集的日志数据的算术平均值；X _rms为采集的日志数据的有效值；X _p为采集的日志数据的峰值；X _r为采集的日志数据的方根幅值；X _ws为采集的日志数据的波形指标；X _if为采集的日志数据的脉冲指标；X _kv为采集的日志数据的峭度指标。 Among them, x _i is the log data collected by a single agent; N is the number of log data collection;

Is the arithmetic mean of the collected log data; X _rms is the effective value of the collected log data; X _p is the peak value of the collected log data; X _r is the square root amplitude of the collected log data; X _ws is the collected log The waveform index of the data; X _if is the impulse index of the collected log data; X _kv is the kurtosis index of the collected log data.

Use the Pearson correlation coefficient to filter out the effective features. Specifically, the above feature values are calculated with the original data for the Pearson correlation coefficient, and the calculated correlation coefficient is compared with the correlation threshold. If it is higher than the correlation threshold, it is considered It is valid data. If it is lower than the correlation threshold, it is considered invalid data and needs to be eliminated so that valid data can be filtered out. For example, if the correlation threshold is 0.7, the correlation coefficient between the square root amplitude and the original data is 0.2, it indicates that the square root amplitude is invalid data, and the correlation coefficient between the kurtosis index and the original data is 0.85, and the kurtosis index is deemed valid data. Among them, the formula of Pearson's correlation coefficient is as follows:

是日志数据x ₁,x ₂...,x _n的算数平均值；

是y ₁,y ₂...,y _n的算数平均值；N为日志数据采集的次数。 Among them, x _i is the value of data collected by a single agent; y _i is a characteristic value extracted from the data collected by a single agent;

Is the arithmetic average of log data x ₁ , x ₂ ..., x _n ;

Is the arithmetic mean of y ₁ , y ₂ ..., y _n ; N is the number of log data collections.

In an optional embodiment, Flume includes multiple first-level agents and one second-level agent. Each first-level agent collects log data from a server, and the log data collected by multiple first-level agents are collected. To the second-level agent, and the second-level agent transmits to HDFS (distributed file system).

Refer to FIG. 2, which is a schematic diagram of the hardware architecture of an embodiment of the electronic device of the present application. In this embodiment, the electronic device 2 is a device that can automatically perform numerical calculation and/or information processing according to pre-set or stored instructions. For example, it can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server, or a cabinet server (including an independent server or a server cluster composed of multiple servers). As shown in FIG. 2, the electronic device 2 at least includes, but is not limited to, a memory 21, a processor 22, and a network interface 23 that can be communicatively connected to each other through a system bus. Wherein: the memory 21 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM) ), static random access memory (SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disks, optical disks, etc. In some embodiments, the memory 21 may be an internal storage unit of the electronic device 2, for example, a hard disk or a memory of the electronic device 2. In other embodiments, the memory 21 may also be an external storage device of the electronic device 2, for example, a plug-in hard disk equipped on the electronic device 2, a smart media card (SMC), a secure digital (Secure Digital, SD) card, flash card (Flash Card), etc. Of course, the memory 21 may also include both the internal storage unit of the electronic device 2 and its external storage device. In this embodiment, the memory 21 is generally used to store the operating system and various application software installed in the electronic device 2, such as the cluster log feature extraction program code. In addition, the memory 21 can also be used to temporarily store various types of data that have been output or will be output.

The processor 22 may be a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 22 is generally used to control the overall operation of the electronic device 2, for example, perform data interaction or communication-related control and processing with the electronic device 2. In this embodiment, the processor 22 is configured to run the program code or process data stored in the memory 21, for example, run the cluster log feature extraction program.

The network interface 23 may include a wireless network interface or a wired network interface. The network interface 23 is usually used to establish a communication connection between the electronic device 2 and other electronic devices. For example, the network interface 23 is used to connect the electronic device 2 with a push platform through a network, and establish a data transmission channel and a communication connection between the electronic device 2 and the push platform. The network may be Intranet, Internet, Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network , Bluetooth (Bluetooth), Wi-Fi and other wireless or wired networks.

Optionally, the electronic device 2 may also include a display, and the display may also be called a display screen or a display unit. In some embodiments, it may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an organic light-emitting diode (OLED) display, etc. The display is used to display the information processed in the electronic device 2 and to display a visualized user interface.

It should be pointed out that FIG. 2 only shows the electronic device 2 with components 21-23, but it should be understood that it is not required to implement all of the illustrated components, and more or fewer components may be implemented instead.

The memory 21 containing a readable storage medium may include an operating system, a cluster log feature extraction program 50, and the like. The processor 22 implements the following steps when executing the cluster log feature extraction program 50 in the memory 21:

In step S10, the logs of the server cluster are collected by the flume (distributed mass log collection, aggregation and transmission system) client, and sent to the Hbase database server. Flume takes the Agent component as the smallest independent operating unit, and an Agent component is a complete data collection tool. Flume collects log data corresponding to each server through multiple agents. Set up an Agent for each server, collect log data on the corresponding server regularly and send it to the background through the API interface.

Step S30: Use Hadoop (distributed system infrastructure) to perform data cleaning on the log data to filter out the original data, where the original data includes at least the server disk occupancy rate, the memory usage rate, the cpu occupancy rate, and the amount of service interface calls.

In step S50, the original data is extracted with feature values including average value, effective value, peak value, root square amplitude, waveform index, impulse index, and kurtosis index.

Step S70: Use the Pearson correlation coefficient to filter out the effective features, and calculate the Pearson correlation coefficient between the extracted feature values and the original data, and compare the calculated correlation coefficient with the correlation threshold. If it is higher than the correlation threshold, it is considered Data is valid, and data below the correlation threshold is considered invalid data and removed.

It should be noted that the specific implementation of the electronic device of the present application is substantially the same as the specific implementation of the cluster log extraction method described above, and will not be repeated here.

In this embodiment, the cluster log feature extraction program stored in the memory 21 can be divided into one or more program modules, and the one or more program modules are stored in the memory 21 and can be divided into one or more program modules. It is executed by two processors (in this embodiment, the processor 22) to complete the application. For example, FIG. 3 shows a schematic diagram of program modules of the cluster log feature extraction program. In this embodiment, the cluster log feature extraction program 50 can be divided into a log collection module 501, a data cleaning module 502, and a feature extraction module 503. , Effective feature screening module 504. Wherein, the program module referred to in this application refers to a series of computer program instruction segments that can complete specific functions, and is more suitable than a program to describe the execution process of the cluster log feature extraction program in the electronic device 2. The cluster log feature extraction method is realized through the specific functions of the program module.

This application also provides a cluster log feature extraction device, including: a log collection module 501, a data cleaning module 502, a feature extraction module 503, and an effective feature screening module 504.

Among them, the log collection module 501 is configured to collect logs of the server cluster through a flume (distributed mass log collection, aggregation and transmission system) client, and send them to the Hbase database server. Flume takes the Agent component as the smallest independent operating unit, and an Agent component is a complete data collection tool. Flume collects log data corresponding to each server through multiple agents. Set up an Agent for each server, collect log data on the corresponding server regularly and send it to the background through the API interface.

The data cleaning module 502 is configured to use Hadoop (distributed system infrastructure) to perform data cleaning on log data to filter out the original data, where the original data includes at least the server disk occupancy rate, memory usage rate, cpu occupancy rate, and business interface call volume.

The feature extraction module 503 is used for extracting feature values including average value, effective value, peak value, square root amplitude, waveform index, impulse index, and kurtosis index from the original data.

The effective feature selection module 504 uses the Pearson correlation coefficient to filter out the effective features, and performs the calculation of the Pearson correlation coefficient on the extracted feature values and the original data respectively, and compares the calculated correlation coefficient with the correlation threshold, which is higher than the correlation threshold Data is considered valid, and data below the correlation threshold is considered invalid data and removed.

In an optional embodiment, as shown in FIG. 6, the data cleaning module 502 includes a Laida criterion determining unit 5021, and the Laida criterion determining unit 5021 uses the Laida criterion to remove data with gross errors, including the following steps:

及剩余误差

其中，x _i为单次Agent采集数据值； For log data x ₁ , x ₂ ..., x _n , calculate its arithmetic average

And residual error

Among them, x _i is the data value collected by a single agent;

Calculate the standard deviation S _x ,

If the residual error v _b (1≤b≤n) of the data x _b satisfies the following formula

It is considered that x _b is a singular value with a gross error value, and the singular value is eliminated.

Further, the data cleaning module 502 further includes a singular value replacement unit 5022. Using Laida's law can effectively identify the singular value of production data, but will produce null values for the deleted data. The singular value replacement unit 5022 replaces the identified singular value of the log data with the median value to realize the preprocessing of the production data information. The median value means that the variable values x ₁ , x ₂ ..., x _n are arranged in order of magnitude to form a sequence, and the variable value in the middle of the variable sequence is called the median value.

In an optional embodiment, as shown in FIG. 5, the feature extraction module 503 includes a mean value extraction unit 5031, an effective value extraction unit 5032, a peak value extraction unit 5033, a square root amplitude extraction unit 5034, a waveform index extraction unit 5035, a pulse Index extraction unit 5036 and kurtosis index extraction unit 5037. Extract the eigenvalues of the original data, including the mean value, effective value, peak value, root square amplitude, waveform index, impulse index, and kurtosis index. Among them,

The effective value is calculated using the following formula:

The peak value is calculated using the following formula: X _p =max(x _i )

The square root amplitude is calculated using the following formula:

The waveform index is calculated using the following formula:

The impulse index is calculated using the following formula:

The kurtosis index is calculated using the following formula:

为采集的日志数据的算术平均值；X _rms为采集的日志数据的有效值；X _p为采集的日志数据的峰值；X _r为采集的日志数据的方根幅值；X _ws为采集的日志数据 的波形指标；X _if为采集的日志数据的脉冲指标；X _kv为采集的日志数据的峭度指标。 Among them, x _i is the log data collected by a single agent; N is the number of log data collection;

Is the arithmetic mean of the collected log data; X _rms is the effective value of the collected log data; X _p is the peak value of the collected log data; X _r is the square root amplitude of the collected log data; X _ws is the collected log The waveform index of the data; X _if is the impulse index of the collected log data; X _kv is the kurtosis index of the collected log data.

Use the Pearson correlation coefficient to filter out the effective features. Specifically, the above feature values are calculated with the original data for the Pearson correlation coefficient, and the calculated correlation coefficient is compared with the correlation threshold. If it is higher than the correlation threshold, it is considered It is valid data. If it is lower than the correlation threshold, it is considered invalid data and needs to be eliminated so that valid data can be filtered out. For example, if the correlation threshold is 0.7, the correlation coefficient between the square root amplitude and the original data is 0.2, it indicates that the square root amplitude is invalid data, and the correlation coefficient between the kurtosis index and the original data is 0.85, and the kurtosis index is deemed valid data. Among them, the formula of Pearson's correlation coefficient is as follows:

是日志数据x ₁，x ₂...,x _n的算数平均值；

是y ₁，y ₂...,y _n的算数平均值；N为数据采集的次数。 Among them, x _i is the value of data collected by a single agent; y _i is a characteristic value extracted from the data collected by a single agent;

Is the arithmetic mean of log data x ₁ , x ₂ ..., x _n ;

Is the arithmetic mean of y ₁ , y ₂ ..., y _n ; N is the number of data collections.

In an optional embodiment, as shown in FIG. 4, the log collection module 501 further includes an agent setting unit 5011, which is configured to set a plurality of first-level agents and a second-level agent for Flume, and each first-level agent The level agents respectively collect log data of a server, and the log data collected by multiple first level agents are collected to the second level agent, and the second level agent is transmitted to HDFS.

It should be noted that the specific implementation of the cluster log feature extraction device of the present application is substantially the same as the specific implementation of the cluster log feature extraction method and the electronic device described above, and will not be repeated here.

In addition, the embodiment of the present application also proposes a computer non-volatile readable storage medium. The computer readable storage medium may be a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a read-only memory (ROM), a Erasing programmable read only memory (EPROM), portable compact disk read only memory (CD-ROM), USB memory, etc., or any combination of several. The computer non-volatile readable storage medium includes a cluster log feature extraction program, etc., and the cluster log feature extraction program 50 implements the following operations when executed by the processor 22:

In step S10, logs of the server cluster are collected through the flume client and sent to the Hbase database server. Flume takes the Agent component as the smallest independent operating unit, and an Agent component is a complete data collection tool. Flume collects log data corresponding to each server through multiple agents. Set up an Agent for each server, collect log data on the corresponding server regularly and send it to the background through the API interface.

Step S30: Use Hadoop to perform data cleaning on the log data to filter out the original data, where the original data includes at least server disk occupancy rate, memory usage rate, cpu occupancy rate, and business interface call volume.

In step S50, the original data is extracted with feature values including average value, effective value, peak value, root square amplitude, waveform index, impulse index, and kurtosis index.

Step S70: Use the Pearson correlation coefficient to filter out the effective features, and calculate the Pearson correlation coefficient between the extracted feature values and the original data, and compare the calculated correlation coefficient with the correlation threshold. If it is higher than the correlation threshold, it is considered Data is valid, and data below the correlation threshold is considered invalid data and removed.

The specific implementation of the computer non-volatile readable storage medium of the present application is substantially the same as the specific implementation of the cluster log feature extraction method and the electronic device 2 described above, and will not be repeated here.

The foregoing descriptions are only preferred embodiments of the application, and are not intended to limit the application. For those skilled in the art, the application may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included in the protection scope of this application.

Claims (20)

A cluster log feature extraction method, applied to electronic equipment, is characterized in that it includes the following steps: Logs of the server cluster are collected through the flume client and sent to the Hbase database. The flume client collects the logs of each server in the server cluster through multiple agent processes. The agent regularly collects and passes the log data on the corresponding server. API interface is sent to Hbase database; Use Hadoop to clean the log data and filter out the original data. The original data includes at least the server disk occupancy rate, memory usage rate, cpu occupancy rate, and business interface call volume; Extract the feature value of the original data including mean value, effective value, peak value, square root amplitude, waveform index, impulse index, and kurtosis index; Use Pearson correlation coefficient to filter out effective features: Perform Pearson correlation coefficient calculations on the extracted feature values with the original data, and compare the calculated correlation coefficient with the correlation threshold. If the correlation coefficient is higher than the correlation threshold, it is valid data. Data below the correlation threshold is invalid data and will be eliminated. The cluster log feature extraction method according to claim 1, characterized in that: In the data cleaning process, the Laida criterion is used to remove data with gross errors, including the following steps: For log data x ₁ ,x ₂ ...,x _n , calculate the arithmetic average

And residual error

Among them, x _i is the log data collected by a single agent; Calculate the standard deviation S _x ,

If the residual error v _b of x _b in the log data (1≤b≤n), it satisfies the formula

It is determined that x _b is a singular value with a gross error value, and the singular value is eliminated. The cluster log feature extraction method according to claim 2, characterized in that: The singular value of the log data is replaced with a median value, where the median value refers to arranging each log data x ₁ , x ₂ ..., x _n in order of size, and the value in the middle position is called the median value. The cluster log feature extraction method according to claim 2, characterized in that: The original data is extracted with feature values including mean value, effective value, peak value, root square amplitude, waveform index, impulse index, and kurtosis index, among which, The effective value is calculated using the following formula:

The peak value is calculated using the following formula: X _p =max(x _i ) The square root amplitude is calculated using the following formula:

The waveform index is calculated using the following formula:

The impulse index is calculated using the following formula:

The kurtosis index is calculated using the following formula:

Among them, x _i is the log data collected by a single agent; N is the number of data collection;

Is the arithmetic mean of the collected log data; X _rms is the effective value of the collected log data; X _p is the peak value of the collected log data; X _r is the square root amplitude of the collected log data; X _ws is the collected log The waveform index of the data; X _if is the impulse index of the collected log data; X _kv is the kurtosis index of the collected log data. The cluster log feature extraction method according to claim 2, characterized in that: The formula of Pearson's correlation coefficient is as follows:

Among them, x _i is the log data collected by a single agent; y _i is a characteristic value extracted from the data collected by a single agent;

Is the arithmetic mean of log data x ₁ , x ₂ ..., x _n ;

Is the arithmetic mean of y ₁ , y ₂ ..., y _n ; N is the number of log data collections. The cluster log feature extraction method according to claim 1, characterized in that: Flume includes multiple first-level agents and one second-level agent. Each first-level agent collects log data from a server respectively. The log data collected by multiple first-level agents are collected to the second-level agent, and the The second-level agent is transmitted to HDFS. A cluster log feature extraction device, which is characterized by comprising: a log collection module, a data cleaning module, a feature extraction module, and an effective feature screening module, Among them, the log collection module is used to collect the logs of the server cluster through the flume client and send them to the Hbase database. Among them, the flume client collects the logs of each server in the server cluster through multiple agent processes, and the agent periodically sends the corresponding server The log data on the system is collected and sent to the Hbase database through the API interface; The data cleaning module is used to use Hadoop to clean the log data and filter out the original data. The original data includes at least the server disk occupancy rate, memory usage rate, cpu occupancy rate, and business interface call volume; The feature extraction module is used to extract the feature value of the original data including mean value, effective value, peak value, square root amplitude, waveform index, impulse index, and kurtosis index; The effective feature screening module uses the Pearson correlation coefficient to filter out the effective features: the extracted feature value is calculated with the original data for the Pearson correlation coefficient, and the calculated correlation coefficient is compared with the correlation threshold. If it is higher than the correlation threshold The data is valid, and the data is invalid if it is lower than the correlation threshold, and it will be eliminated. The cluster log feature extraction device according to claim 7, wherein the data cleaning module includes a Laida criterion determining unit, and the Laida criterion determining unit uses the Laida criterion to eliminate data with gross errors, including the following step: For log data x ₁ , x ₂ ..., x _n , calculate its arithmetic average

And residual error

Among them, x _i is the data value collected by a single agent; Calculate the standard deviation S _x ,

If the residual error v _b (1≤b≤n) of the data x _b satisfies the following formula

It is considered that x _b is a singular value with a gross error value, and the singular value is eliminated. The cluster log feature extraction device according to claim 8, wherein the data cleaning module further comprises a singular value replacement unit, and the singular value replacement unit replaces the singular value of the log data with a median value, wherein the median value is Refers to arranging each log data x ₁ , x ₂ ..., x _n in order of size, and the value in the middle position is called the median value. The cluster log feature extraction device according to claim 8, wherein the feature extraction module includes a mean value extraction unit, an effective value extraction unit, a peak value extraction unit, a square root amplitude extraction unit, a waveform index extraction unit, and an impulse index extraction unit , The kurtosis index extraction unit separately extracts the characteristic values of the original data including the mean value, the effective value, the peak value, the square root amplitude, the waveform index, the impulse index, and the kurtosis index. The effective value is calculated using the following formula:

The peak value is calculated using the following formula: X _p =max(x _i ) The square root amplitude is calculated using the following formula:

The waveform index is calculated using the following formula:

The impulse index is calculated using the following formula:

The kurtosis index is calculated using the following formula:

Among them, x _i is the log data collected by a single agent; N is the number of log data collection;

Is the arithmetic mean of the collected log data; X _rms is the effective value of the collected log data; X _p is the peak value of the collected log data; X _r is the square root amplitude of the collected log data; X _ws is the collected log The waveform index of the data; X _if is the impulse index of the collected log data; X _kv is the kurtosis index of the collected log data. The cluster log feature extraction device according to claim 8, wherein the formula of Pearson correlation coefficient is as follows:

Among them, x _i is the log data collected by a single agent; y _i is a characteristic value extracted from the data collected by a single agent;

Is the arithmetic mean of log data x ₁ , x ₂ ..., x _n ;

Is the arithmetic mean of y ₁ , y ₂ ..., y _n ; N is the number of log data collections. The cluster log feature extraction device according to claim 7, wherein the log collection module further comprises an agent setting unit, which is used to set a plurality of first-level agents and a second-level agent for Flume. The first-level agents respectively collect log data of a server, and the log data collected by multiple first-level agents are collected to the second-level agent, and the second-level agent is transmitted to HDFS. An electronic device, characterized in that it includes a memory and a processor, the memory stores a cluster log feature extraction program, and the cluster log feature extraction program is executed by the processor to implement the following steps: Logs of the server cluster are collected through the flume client and sent to the Hbase database. The flume client collects the logs of each server in the server cluster through multiple agent processes. The agent regularly collects and passes the log data on the corresponding server. API interface is sent to Hbase database; Use Hadoop to clean the log data and filter out the original data. The original data includes at least the server disk occupancy rate, memory usage rate, cpu occupancy rate, and business interface call volume; Extract the feature value of the original data including mean value, effective value, peak value, square root amplitude, waveform index, impulse index, and kurtosis index; Use the Pearson correlation coefficient to filter out the effective features: the extracted feature values are calculated with the original data for the Pearson correlation coefficient, and the calculated correlation coefficient is compared with the correlation threshold. If it is higher than the correlation threshold, it is valid data. Data below the correlation threshold is invalid data and will be eliminated. The electronic device according to claim 13, wherein: In data cleaning, the Laida criterion is used to eliminate data with gross errors, including the following steps: For log data x ₁ ,x ₂ ...,x _n , calculate the arithmetic average

And residual error

Among them, x _i is the data value collected by a single agent; Calculate the standard deviation S _x ,

If the residual error v _b of x _b in the log data (1≤b≤n), it satisfies the formula

It is considered that x _b is a singular value with a gross error value, and the singular value is eliminated. The electronic device according to claim 14, wherein: The singular value in the log data is replaced with a median value, where the median value refers to arranging each log data x ₁ , x ₂ ..., x _n in order of size, and the value in the middle position is called the median value. The electronic device according to claim 13, wherein Flume includes a plurality of first-level agents and a second-level agent, and each first-level agent collects log data of a server respectively, and multiple first-level agents The log data collected by the agent is collected to the second-level agent and transmitted to the HDFS by the second-level agent. A computer nonvolatile readable storage medium, wherein the computer nonvolatile readable storage medium stores a computer program, the computer program includes program instructions, and when the program instructions are executed by a processor, Implement the cluster log feature extraction method of claim 1. 18. The computer non-volatile readable storage medium of claim 17, wherein in the data cleaning process, using the Laida criterion to remove data with gross errors includes the following steps: For log data x ₁ ,x ₂ ...,x _n , calculate the arithmetic average

And residual error

Among them, x _i is the log data collected by a single agent; Calculate the standard deviation S _x ,

If the residual error v _b of x _b in the log data (1≤b≤n), it satisfies the formula

It is determined that x _b is a singular value with a gross error value, and the singular value is eliminated. The computer non-volatile readable storage medium according to claim 18, wherein the singular value of the log data is replaced with a median value, wherein the median value refers to each log data x ₁ , x ₂ .. ., x _n are arranged in order of size, and the value in the middle is called the median. The computer non-volatile readable storage medium of claim 17, wherein Flume comprises a plurality of first-level agents and a second-level agent, and each first-level agent collects a log of a server respectively Data, the log data collected by multiple first-level agents are collected to the second-level agent, and transmitted to the HDFS by the second-level agent.

Images