CN114036120A - A real-time analysis method and system based on massive log data - Google Patents

Abstract

The present application discloses a real-time analysis method based on massive log data. The method includes: acquiring input query conditions, combining the query conditions into keywords, performing a search based on the stored big data according to the keywords, and performing a search on the searched result set. The analysis is performed according to the set analysis strategy, and the analysis result is obtained; wherein, the big data is obtained in the following manner: the data associated with the business in each business process of each business point is recorded, the log data of each business point is generated, and the real-time log data of each business point is generated. The log data of each business point is collected and cached, and the cached log data is filtered and structured to obtain big data for persistent storage, and the big data is stored. The present application can satisfy operators' real-time analysis of call logs, thereby minimizing log analysis time and quickly locating problems.

Description

A real-time analysis method and system based on massive log data

technical field

The present invention relates to the field of communications, and in particular, to a real-time analysis method based on massive log data.

Background technique

With the development of communication technology, more and more users communicate and communicate by means of communication technologies such as mobile phones, fixed telephones, and networks. The use of users involves the invocation of various devices such as voice, video, and audio.

Users hope to have a safe and stable call environment to reduce and eliminate the occurrence of abnormal calls; operators, etc. also hope to eliminate abnormal calls from the source through technical means, provide the public with a safe and stable call environment, and be able to The user's usage habits are analyzed and summarized to improve the user experience.

At present, the storage and analysis technologies for business logs mainly focus on the following:

1. Implemented through a text file

Save a text file at each service point, the text file includes time, service name, machine name, automatic machine number, calling number, called number, original called, message type and content; when there is an analysis requirement, the The text files of each business point are scanned, and the required information is filtered out, and then analyzed and sorted.

2. Implemented through relational database

The business logs on each business point are aggregated to the relational database in real time, and the data is indexed according to the requirements; when there is an analysis requirement, the data that meets the conditions such as time period and call number is scanned and queried in full to obtain the required data. information.

3. Realization based on the combination of relational database and text file

By classifying and summarizing the data transmitted from each service point according to the calling number, called number, service name, and automatic machine number, each call generates a unique task identifier (ID), and then the different data of each call are stored separately. In different data files, the corresponding relationship between calling number, called number, service name, automaton number, task ID and data file is stored in the database; when using, find the task ID according to the query conditions, obtain the list of data files, read Take all files, summarize data, perform data analysis and display.

Among them, the service point is each telecommunication equipment and application software used in the communication process.

The above methods have some drawbacks, as follows:

For the method implemented through text files, this method needs to read and query files one by one, and has a high lag in data analysis.

For the method implemented by the relational database, all data in this method is stored in the relational database. For the ever-increasing mass of logs, such as hundreds of millions of logs, indexing and analyzing queries will affect the response and slow down.

For the method of combining relational database and text file, this method performs data preprocessing based on the calling number, called number, service name, and automatic machine number for each piece of data. The data preprocessing node will form a bottleneck.

At the same time, the common defects of all the above methods are:

1. Data storage management is inconvenient. Whether it is in the form of a file or stored in a database, when accumulated for a period of time, the massive log increases the difficulty of storage and backup. Improper handling will result in data loss.

2. In the case of massive logs, the data query response is seriously delayed, which cannot meet the basic needs of real-time analysis of data.

SUMMARY OF THE INVENTION

The present invention provides a real-time analysis method based on massive log data to reduce log analysis time.

A real-time analysis method based on massive log data provided by the present invention is implemented as follows:

Get the input query conditions, combine the query conditions into keywords,

Based on the stored big data, search by keywords,

Analyze the searched result set according to the set analysis strategy, and obtain the analysis result;

in,

The big data is obtained as follows:

Record the data associated with the business in each business process of each business point, and generate log data of each business point,

Collect the log data of each business point in real time and cache it,

Filter and structure the cached log data to obtain big data for persistent storage,

Store big data.

Preferably, the big data is stored in a distributed cluster manner, wherein hot data is stored in the first distributed storage device, all data is stored as cold data in the second distributed storage device, and when the cold data is accessed, it is stored as hot data. data is transferred to a first distributed storage device, the performance of the first distributed storage device is higher than the performance of the second distributed storage device,

Based on stored big data, search by keywords, including,

Prioritize the search for hot data,

When the hot data cannot be searched, the cold data is searched.

Optionally, the method further includes displaying the analysis results in a graph or in a table, wherein the graphical presentation is a time-series two-dimensional flowchart, and each step of the process displays detailed information of the process.

Optionally, the real-time collection of log data of each service point and buffering, including,

Collect log data of each business point from each business point,

According to the type of log data, it is cached to the corresponding queue in a distributed cluster mode, and the cached data is managed in the mode of producer and consumer. When the cached data is used by all consumers, the data is deleted.

The present invention also provides a real-time analysis system based on massive log data, the system comprising:

A log data generating device is used to record the data associated with the business in each business process of each business point, and generate log data of each business point,

The log data collection device is used to collect the log data of each business point in real time,

The big data cache device is used to cache the collected log data of each business point,

The data filtering confirmation device is used to filter and structure the cached log data to obtain big data for persistent storage.

Big data storage device for storing big data,

The log data analysis device is used to obtain the input query conditions, combine the query conditions into keywords, search according to the keywords based on the stored big data, analyze the searched result set according to the set analysis strategy, and obtain the analysis result.

Preferably, the system further includes,

The analysis result display device is used to display the analysis results in a graph or in a table, wherein the graphical display is a time-series two-dimensional flowchart, and each step of the process displays the detailed information of the process.

Optionally, the log data generating device includes,

The log data sorting unit is used for interacting with each business point according to the standard protocol, sorting the data provided by each business point, and generating log data;

The log data file unit is used to file the log data generated by the log data sorting unit to form a file suitable for reading;

The log data collection device includes:

The log data collection unit is used to track log files and provide event data to the log data aggregation unit for use;

The log data aggregation unit is used to process the data collected by the log data collection unit and transmit it to the first log data output unit at least once, and in the case that the first log data output unit is blocked and all transmitted events are not confirmed, continue Attempting to send data to the first log data output unit until the log data output unit outputs a confirmation that the event has been received;

The first log data output unit is used to interact with the big data cache device, obtain data from the log data aggregation unit, transmit the acquired data to the big data cache device, and return to the log when receiving a confirmation event from the big data cache device Data aggregation unit.

Optionally, the big data cache device includes:

The producer unit is used to interact with the first log data output unit, obtain log data, and publish the data to the corresponding data queue unit according to the type;

The data queue unit is used to receive the data sent by the producer unit for storage, interact with the consumer unit, read data from the queue and provide it to the consumer unit;

The consumer unit is used to interact with the data queue unit, read data from the queue, and transmit the data to the data filtering confirmation device;

The data filtering and confirming device includes:

The log data input unit is used to interact with the big data cache device, and obtain data from the big data cache device in a continuous stream transmission mode;

The log data filtering unit is used to filter, reorganize, confirm one or any combination of data processing, and convert the general format;

a second log data output unit, configured to output the data generated by the log data filtering unit to the big data storage device;

The big data storage device includes,

The first data interface unit is used to input, store, query and output big data.

The search and analysis unit is configured to acquire corresponding data from the data storage unit according to the data query condition provided by the first data interface unit.

A data storage unit for storing log data.

Optionally, the log data analysis device includes:

The second data interface unit is used for interacting with the big data storage device, providing a unified reading interface for querying the data in the big data storage device, and obtaining the data required for the calculation of the data analysis unit;

a data analysis unit for interacting with the second data interface unit and the query interface unit to provide real-time computing and mining computing capabilities;

The query interface unit is used to interact with the analysis result display device, analyze the query conditions, form a suitable query statement, send it to the data analysis unit for query, and organize the query results into a result set in a specified format, and return it to the analysis result display device;

The analysis result display device includes:

The query control unit is used for the start, stop, input of query conditions and normative detection of log data query tasks,

The query request unit is used to receive the query request of the query control unit, interact with the log data analysis device, and obtain the data required for the calculation of the data processing unit; interact with the data processing unit, and hand over the queried data to the data processing unit for processing .

A data processing unit for converting and/or processing the data from the query request unit according to the requirements of the query conditions;

The result display unit is used to display the log data organized by the data processing unit in a graphical or tabular form.

Optionally, the big data cache device is a distributed cluster cache device, and the device further includes:

a first cluster management unit, used for cluster management of the producer unit, the data queue unit, and the consumer unit;

The big data storage device is a distributed cluster storage device, and the device further includes:

The second cluster management unit is used to manage each node and provides search and analysis functions on all nodes.

This application aggregates the log data generated in each service point into big data for storage, and searches according to the query conditions based on the big data, which improves the efficiency of finding the cause of abnormal calls from service logs, and avoids untimely storage and analysis of service logs. slow problem. The real-time analysis method and system based on massive log data of the present application can satisfy the operator's real-time analysis of call logs after deployment by the operator, thereby minimizing the time for log analysis, quickly locating problems, and avoiding operators and Users suffer losses, and truly provide a safe call environment for phone users. For example, it is possible to locate and deal with the faults encountered by telephone users in real-time and effectively in a timely manner, thereby greatly reducing the number of faulty calls in the communication network, reducing the losses of telephone users and operators due to telephone faults, and effectively improving operations. provider's service quality.

Description of drawings

FIG. 1 is a schematic structural diagram of a real-time analysis system based on massive logs according to an embodiment of the present application.

FIG. 2 is a schematic diagram of a log data generating apparatus.

FIG. 3 is a schematic diagram of a log data collection device.

FIG. 4 is a schematic diagram of a big data cache device.

FIG. 5 is a schematic diagram of a data filtering and confirming device.

FIG. 6 is a schematic diagram of a big data storage device.

FIG. 7 is a schematic diagram of an apparatus for analyzing log data.

FIG. 8 is a schematic diagram of a device for analyzing results.

FIG. 9 is a schematic flowchart of a real-time analysis method based on massive logs of the present application.

Detailed ways

In order to make the objectives, technical means and advantages of the present application more clear, the present application will be further described in detail below with reference to the accompanying drawings.

Referring to FIG. 1 , FIG. 1 is a schematic structural diagram of a real-time analysis system based on massive logs according to an embodiment of the present application. The system includes: a log data generating device for formatted output of log data, a log data acquisition device for real-time collection of log data, a large data cache device for temporary storage of log data, and a log data for sorting, filtering and confirming. A data filtering confirmation device, a big data storage device for persistently storing log data, a log data analysis device for analyzing log data, and an analysis result display device for responding to a query; wherein:

The log data generating device is used to realize the log data output function at each service point, and specifically, according to the specified format, time interval, and specified path, the log file is generated. The business module, the use process of each business module, and the information content and information status transmitted between each business module.

The log data collection device is used for interacting with the log data generation device located at each business point, and preliminarily organizes the log data collected from the log data generation device to generate standard unified log data, that is, structured log data; and The big data cache device interacts and pushes the structured log data to the big data cache device for storage. Among them, in view of the fact that the log data collection device obtains a lot of log data from the log data generation device of each service node in real time, for example, in the telecommunications call service, multiple log data will be generated in one call. It will also increase. If the data is not processed, it will be very troublesome to query all the records of a call in the massive data in the future. Therefore, when recording the log data, according to the time, business, number, etc. Generate a unique ID, recorded in each line of log data. Considering that there are various situations in the log data, such as the same behavior data is stored in multiple rows, the fields are empty, etc., the log data collection device will sort out these situations, and submit the data in the same format to the big data cache device.

The big data cache device is responsible for temporarily storing and transferring log data, interacting with the log data collection device, receiving log data reported by the log data collection device, storing the log data in real time, and interacting with the data filtering and confirming device, The log data is read from the device, and according to the successfully read log data, the storage of the read log data in the big data cache device is abandoned. The big data cache device can adopt the subscription mode to provide data to multiple subscribers at the same time. After the subscribers of each data "consume" the data, the data will be cleared from the cache queue, which ensures the security of log data. The data filtering confirmation device can convert the data according to the format of the big data storage device after obtaining the log data from the cache, and filter out the useless information.

The data filtering confirmation device is used to interact with the big data cache device, obtain log data from the big data cache device, filter, reorganize and confirm the acquired log data, and feed back the new log data generated after confirmation to the big data cache device. data storage device.

The big data storage device is the core device of the system, which is used to interact with the data filtering and confirming device, and persistently store the log data from the data filtering and confirming device. At the same time, it interacts with the log data analysis device for the log data analysis device to query data. The big data storage device indexes and distributes the data, for example, indexes the unique identifier of each call in the log data, and classifies the data according to the time of hot and cold storage. The data is stored on the first distributed storage device with high performance. At the same time, all data is stored as cold data on the second distributed storage device with average data storage performance. When the cold data is accessed, it will be stored as hot data. Dumping to the first distributed storage device can not only solve the real-time problem, but also solve the cost problem caused by mass data storage.

The log data analysis device is used to interact with the big data storage device, obtain real-time log data, analyze and summarize the log data, and send the result set to the analysis result display device.

The analysis result display device is used to interact with the log data analysis device, issue a query request to the log data analysis device, obtain the result set returned by the log data analysis device, and visualize the result set in the form of graphs or tables.

Referring to FIG. 2, FIG. 2 is a schematic diagram of a log data generating apparatus. The log data generating means is composed of a log data sorting unit and a log data file unit. in:

The log data sorting unit is used to interact with each service point according to the standard protocol, sort out the communication message data provided by each service point, and generate a time, service type, automatic machine number, call unique code, host number, called number. , message data and other log data;

The log data file unit is used to file the log data generated by the log data sorting unit to form a file suitable for reading. The log data file unit supports the formation of files with various suffixes such as txt and log in time periods such as minutes, hours, and days. It also supports data sources in multiple formats, so that multiple businesses can use log data for real-time analysis at the same time.

Referring to FIG. 3 , FIG. 3 is a schematic diagram of a log data collection device. The log data collection device is composed of a log data collection unit, a log data aggregation unit and a first log data output unit. in:

The log data collection unit is used to track log files and provide event data to the log data aggregation unit for use;

The log data aggregating unit is configured to process the data collected by the log data collection unit and transmit it to the first log data output unit at least once to ensure that no data is lost. In the event that the first log data output unit is blocked and all transmitted events are not acknowledged, the log data aggregation unit will continue to attempt to send data to the first log data output unit until the first log data output unit outputs an acknowledgement of received events ; By adopting the data filtering and sorting technology, the setting can support the collection of files of various formats, and can summarize the data of multiple lines.

The first log data output unit is used to interact with the big data cache device, obtain data from the log data aggregation unit, and then transmit the obtained data to the big data cache device, and return to the big data cache device when receiving the confirmation event. Log data aggregation unit to complete a data transfer.

Referring to FIG. 4, FIG. 4 is a schematic diagram of a large data cache device. The big data cache device is composed of a producer unit, a data queue unit, a consumer unit and a first cluster management unit. in:

The producer unit is used to interact with the log data output unit of the first log data collection device, obtain log data, and publish the data to the data queue unit; according to the type of log data, the log data will be published to the corresponding The data queue unit, for example, uses different keywords to classify and process log data of different services to form independent transmission and storage.

The data queue unit is used to receive the data sent by the producer unit for storage, interact with the consumer unit, read data from the queue, and the same data can be used by multiple consumer units;

The consumer unit is used to interact with the data queue unit, read data from the queue, and transmit the data to the data filtering confirmation device;

The first cluster management unit is used for the big data cache. The device is a distributed and partitioned system, and the cluster management unit is used for cluster management of the producer unit, the data queue unit and the consumer unit.

In this way, the big data cache device stores and manages log data in the mode of producers and consumers, the log data collection device acts as a producer to store log data in the big data cache device, and the data filtering confirmation device acts as a consumer from The big data cache device consumes data.

Referring to FIG. 5 , FIG. 5 is a schematic diagram of a data filtering and confirming device. The data filtering and confirmation device is the final arrangement before the persistent storage of log data. The device obtains the log data from the consumer unit of the big data cache device, sorts, filters, and confirms the log data according to the established format, and then transmits it to the large data storage company who is brave in persistent storage. data storage device. The data filtering confirmation device is composed of a log data input unit, a log data filtering unit and a second log data output unit. in,

The log data input unit is used to interact with the big data cache device, and obtain various data from the big data cache device in a continuous stream transmission mode.

Log data filtering unit for data processing and transformation, parsing individual events, identifying named fields to build structures, and converting them into a common format.

The second log data output unit is configured to output the data generated by the log data filtering unit to the big data storage device.

Referring to FIG. 6, FIG. 6 is a schematic diagram of a big data storage device. The big data storage device stores log data persistently and provides functions for search and analysis. The big data storage device is composed of a first data interface unit, a search analysis unit and a data storage unit. in:

The first data interface unit is used to input, store, query and output data to the big data storage device.

The search and analysis unit is configured to acquire corresponding data from the data storage unit according to the data query condition provided by the first data interface unit.

The data storage unit is used to store the log data; preferably, by classifying and summarizing the data, when the data is stored in the database for persistence, a globally unique label is set for the same group of data, so as to greatly improve the query speed.

The second cluster management unit is configured to manage each node in the case of a cluster composed of a plurality of nodes in the data storage device, and provide a search and analysis function on all nodes.

Referring to FIG. 7 , FIG. 7 is a schematic diagram of an apparatus for analyzing log data. The device is used for arranging data query conditions, searching for data in the big data storage device with optimal conditions, and processing the search results as required. The log data analysis device is composed of a second data interface unit, a data analysis unit and a query interface unit. in:

The second data interface unit is used for interacting with the big data storage device, providing a unified reading interface for querying data in the big data storage device, and obtaining the data required for the calculation of the data analysis unit.

The data analysis unit is configured to interact with the second data interface unit and the query interface unit to provide real-time computing and mining computing capabilities.

The query interface unit is used to interact with the analysis result display device, analyze the query conditions, form a suitable query statement, send it to the data analysis unit for query, and organize the query results into a result set in a specified format, and return it to the analysis result display device.

Referring to FIG. 8 , FIG. 8 is a schematic diagram of an apparatus for analyzing results. The analysis result device is the control center of the system, responsible for the initiation and stop of the query process and the display of query results, and provides graphical and tabular display effects for the query results. The analysis result display device is composed of a query control unit, a query request unit, a data processing unit, and a result display unit. in:

The query control unit is used for the start, stop, input of query conditions and normative detection of log data query tasks.

The query request unit is used to receive the query request from the query control unit, interact with the log data analysis device, and obtain the data required for the calculation of the data processing unit; interact with the data processing unit, and hand over the queried data to the data processing unit for processing .

The data processing unit is used to convert and process the data from the query request unit according to the requirements of the query conditions.

The result display unit is used to display the log data organized by the data processing unit in the form of graphs or tables.

The above system realizes the following functions.

1. Real-time collection of log data

According to the analysis needs of the operator, all or part of the log data generated during the communication process is transmitted to the system for analysis and processing of the communication situation.

Through the log data collection device, according to the preset strategy, the communication information is preliminarily sorted in real time, the invalid information is filtered and removed, and the log data is pushed to the big data cache device for storage.

2. Data cache function

For the parallel massive data from the log data acquisition device, it is necessary to have preventive measures against data loss caused by untimely data processing. Therefore, before data processing, a data cache device is added. This device is a distributed, subscribed message queue system, with fast Persistence, high throughput, load balancing and other functions.

3. Data conversion filtering function

Before log data is stored in the database for persistent storage, the data must be processed and transformed to ensure that the data is in the optimal format for querying; the data conversion filtering function can add fields, remove fields, and segment data through regular expressions. Different data processing methods can also be performed according to conditional judgment.

4. Real-time analysis of log data

Immediately after the log data is generated, it is collected by the log data collection device, transmitted, and finally stored in the big data storage device, so that the data required for the query can be obtained from the big data storage device in real time, and analyzed according to the set analysis strategy. For example, identification and analysis are performed in multiple dimensions according to the calling number and the called number, the specified time period, and the service type, and the analysis result is provided to the analysis result display device.

5. Analysis result display function

After the analysis result is obtained, the analysis result display device can display the analysis result graphically or in a table according to the selected method, wherein the graphical display is a sequential two-dimensional flow chart, and each step of the flow can display the detailed information of the flow.

In this embodiment, log data is collected online in real time, and the big data technology is used to analyze the log, which has high analysis efficiency; the cache mechanism is adopted, and when the concurrent amount of collected log data exceeds the storage amount, the log can be cached first to prevent log loss. ; Classify the hot data and cold data that have been stored in the database for persistence to reduce the amount of data scanned by each query, which is beneficial to improve the query speed.

Referring to FIG. 9 , FIG. 9 is a schematic flowchart of a real-time analysis method based on massive logs of the present application. The method includes,

Step 901, perform a normalization check on the input query condition, after the check is passed, execute Step 902, otherwise, output a prompt,

Step 902, according to the rules of the unique identification of the log data, for example, different identifications used for classification, combine the query conditions into keywords, and perform a search based on the data stored in the big data storage device,

Among them, the hot data has a higher search priority, that is, the hot data is searched first,

If the hot data cannot be searched, then go to the cold data to search.

If the final search result is empty, the query result will be sent directly;

If the final search result is not empty, a result set is obtained, and step 903 is executed to perform data analysis and sorting on the search result.

Step 903, arranging the result set, and then displaying the results graphically or in a table; the graphical display is a time-series two-dimensional flow chart, the detailed information of the process can be displayed in each step of the process, and the tabular display can display the result set. The details of the list are displayed in pagination.

In this document, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such existence between these entities or operations. The actual relationship or sequence. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the present invention. within the scope of protection.

Claims (10)

1. A real-time analysis method based on mass log data is characterized by comprising the following steps,

acquiring input query conditions, combining the query conditions into keywords,

based on the stored big data, searching is carried out according to the keywords,

analyzing the searched result set according to a set analysis strategy to obtain an analysis result;

wherein,

the big data is obtained as follows:

recording the data associated with the service in each service process of each service point to generate the log data of each service point,

collecting log data of each service point in real time, caching,

filtering and structuring the cached log data to obtain big data for persistent storage,

and storing the big data.

2. The real-time analysis method of claim 1, wherein the big data is stored in a distributed cluster, wherein hot data is stored in a first distributed storage device, all data is stored in a second distributed storage device as cold data, and the cold data is transferred to the first distributed storage device as hot data when accessed, wherein the performance of the first distributed storage device is higher than the performance of the second distributed storage device,

based on the stored big data, a search is performed according to the keywords, including,

the hot data is preferentially searched for,

in the case that the hot data cannot be searched, the cold data is searched again.

3. The real-time analysis method of claim 1, further comprising graphically displaying or tabulating the analysis results, wherein the graphical display is a time-sequenced two-dimensional flow chart, and each flow shows the detailed information of the flow.

4. The real-time analysis method of claim 1, wherein the collecting log data of each service point in real-time and buffering comprises,

collecting log data of each service point from each service point,

according to the type of the log data, caching the log data in corresponding queues in a distributed cluster mode, managing the cached data in a mode of a producer and a consumer, and deleting the cached data after the cached data is used by all consumers.

5. A real-time analysis system based on mass log data is characterized in that the system comprises,

a log data generating device for recording the data associated with the service in each service process of each service point to generate the log data of each service point,

a log data acquisition device for acquiring log data of each service point in real time,

the big data caching device is used for caching the collected log data of each service point,

the data filtering and confirming device is used for filtering and structuring the cached log data to obtain big data for persistent storage,

a big data storage device for storing big data,

and the log data analysis device is used for acquiring input query conditions, combining the query conditions into keywords, searching according to the keywords based on the stored big data, and analyzing the searched result set according to a set analysis strategy to obtain an analysis result.

6. The real-time analysis system of claim 5, further comprising,

and the analysis result display device is used for carrying out graphic display or table display on the analysis result, wherein the graphic display is a time-sequenced two-dimensional flow chart, and each step of the flow displays the detailed information of the flow.

7. The real-time analysis system of claim 6, wherein the log data generation means comprises,

the log data sorting unit is used for interacting with each service point according to a standard protocol, sorting the data provided by each service point and generating log data;

the log data file unit is used for performing file processing on the log data generated by the log data sorting unit to form a file suitable for reading;

the log data acquisition device comprises a log data acquisition device,

the log data collection unit is used for tracking the log file and providing the event data for the log data aggregation unit to use;

the log data gathering unit is used for processing the data collected by the log data collecting unit and transmitting the processed data to the first log data output unit at least once, and under the condition that the first log data output unit is blocked and all transmitted events are not confirmed, continuously trying to send data to the first log data output unit until the log data output unit outputs and confirms the received events;

and the first log data output unit is used for interacting with the big data cache device, acquiring data from the log data aggregation unit, transmitting the acquired data to the big data cache device, and returning the acquired data to the log data aggregation unit when receiving a confirmation event of the big data cache device.

8. The real-time analysis system of claim 7, wherein the big data caching means comprises,

the producer unit is used for interacting with the first log data output unit, acquiring log data and issuing the data to the corresponding data queue unit according to types;

the data queue unit is used for receiving and storing the data sent by the producer unit, interacting with the consumer unit, reading the data from the queue and providing the data to the consumer unit;

the consumer unit is used for interacting with the data queue unit, reading data from the queue and transmitting the data to the data filtering and confirming device;

the data filtering and confirming device comprises a data filtering and confirming device,

the log data input unit is used for interacting with the big data caching device and acquiring data from the big data caching device in a continuous stream transmission mode;

the log data filtering unit is used for carrying out one or any combination of processing of filtering, recombining and confirming the data and converting the universal format;

a second log data output unit for outputting the data generated by the log data filtering unit to the big data storage device;

the big data storage device comprises a big data storage device,

and the first data interface unit is used for carrying out data input storage and query output on the big data.

And the search analysis unit is used for acquiring corresponding data from the data storage unit according to the data query condition provided by the first data interface unit.

And the data storage unit is used for storing the log data.

9. The real-time analysis system of claim 8, wherein the log data analysis means comprises,

the second data interface unit is used for interacting with the big data storage device, providing a uniform reading interface for inquiring data in the big data storage device and acquiring data required by the calculation of the data analysis unit;

the data analysis unit is used for interacting with the second data interface unit and the query interface unit and providing real-time calculation and mining calculation capacity;

the query interface unit is used for interacting with the analysis result display device, analyzing the query conditions to form a proper query statement, submitting the query statement to the data analysis unit for query, sorting the query result into a result set with a specified format, and returning the result set to the analysis result display device;

the analysis result display device comprises a display device,

the query control unit is used for starting and stopping a log data query task, recording query conditions and detecting normalization,

the query request unit is used for receiving a query request of the query control unit, interacting with the log data analysis device and acquiring data required by the calculation of the data processing unit; interacting with the data processing unit, and handing the inquired data with the data processing unit for processing;

the data processing unit is used for converting and/or processing the data from the query request unit according to the requirements of the query conditions;

and the result display unit is used for displaying the log data finished by the data processing unit in a graphical or tabular form.

10. The real-time analysis system of claim 9, wherein the big data caching apparatus is a distributed cluster caching apparatus, the apparatus further comprising,

the first cluster management unit is used for cluster management of the producer unit, the data queue unit and the consumer unit;

the big data storage device is a distributed cluster storage device, the device further comprises,

and the second cluster management unit is used for managing each node and providing a search analysis function on all the nodes.

Images