CN119357142A

CN119357142A - Log processing method, device and electronic equipment

Info

Publication number: CN119357142A
Application number: CN202411381540.9A
Authority: CN
Inventors: 胡九有
Original assignee: Guangzhou Overseas Shoulder Sub Network Technology Co ltd
Current assignee: Guangzhou Overseas Shoulder Sub Network Technology Co ltd
Priority date: 2024-09-29
Filing date: 2024-09-29
Publication date: 2025-01-24

Abstract

The present application belongs to the field of computer technology, and provides a log processing method, device and electronic device. The method determines the attribute information of the collected log; based on the attribute information, the log content of the log is stored in the corresponding partition in the message middleware; the log data in each partition is deduplicated, and the deduplicated log data is stored in the database. The above method can completely delete redundant logs, improve the accuracy of log deduplication, and reduce the occupancy of the database by redundant logs.

Description

Log processing method and device and electronic equipment

Technical Field

The present application belongs to the field of computer technology, and in particular, to a method and an apparatus for processing logs, and an electronic device.

Background

The log may record state information of the application program during running, operation behavior of the user during using the application program, and the like. By analyzing the log, valuable information can be mined. In order to avoid missing the collected logs due to network problems, the logs are usually collected in batches periodically, and a large number of logs are often collected repeatedly in this way, so that the redundant logs occupy the storage space, and therefore, the problem to be solved is urgent to delete the redundant logs.

In the related art, the redundant logs in each storage space are usually deleted, however, because the collected redundant logs may be distributed in different storage spaces, the related art cannot completely delete the redundant logs, which is not beneficial to analysis of the logs.

Disclosure of Invention

The application provides a log processing method, a log processing device and electronic equipment, which aim to solve the technical problem that redundant logs cannot be completely deleted.

The first aspect of the embodiment of the application provides a log processing method, which comprises the steps of determining attribute information of an acquired log, storing log content of the log to a corresponding partition in a message middleware based on the attribute information, performing duplicate removal processing on log data in each partition, and storing the duplicate removed log data in a database.

According to the embodiment of the application, before determining the attribute information of the collected log, the method further comprises the step of collecting the log based on the pre-configured white list information and the pre-configured black list information.

According to the embodiment of the application, the storage of the log content of the log into the corresponding partition in the message middleware based on the attribute information comprises the steps of determining the partition corresponding to the attribute information from the message middleware and storing a key value pair formed by the attribute information and the log content based on the partition corresponding to the attribute information.

According to the embodiment of the application, each partition comprises a master copy and a slave copy, and the method further comprises the step of returning a response code to the source terminal of the log if the master copy and the slave copy both receive the key value pair, wherein the response code is used for prompting that the key value pair is successfully stored.

According to the embodiment of the application, the log data in each partition is subjected to de-duplication processing, which comprises the steps of grouping the log data in each partition based on a sliding window to obtain a plurality of log groups, repeatedly detecting the log data in the plurality of log groups, and deleting the detected repeated log data.

According to the embodiment of the application, after the log data in each partition is subjected to the deduplication processing, the method further comprises the steps of identifying the log data with the abnormal information from the log data subjected to the deduplication based on a preset rule, and moving the log data with the abnormal information to a designated partition of the message middleware.

According to the embodiment of the application, the log data after the duplication removal is stored in a database, and the method comprises the steps of creating a data table corresponding to the configured table information in the database based on the configured table information, and writing the log data after the duplication removal into the data table.

According to the embodiment of the application, if the total amount of the logs in the message middleware is the same as the number of the collected logs, performing the duplicate removal processing on the log data in each partition, and storing the duplicate removed log data into a database, and if the total amount of the logs in the message middleware is different from the number of the collected logs, generating prompt information, wherein the prompt information is used for indicating that the collected logs are abnormal.

The second aspect of the embodiment of the application provides a log processing device which comprises a determining unit, a storage unit and a processing unit, wherein the determining unit is used for determining attribute information of an acquired log, the storage unit is used for storing log content of the log to a corresponding partition in a message middleware based on the attribute information, and the processing unit is used for carrying out duplicate removal processing on log data in each partition and storing the duplicate removed log data to a database.

A third aspect of the embodiments of the present application provides an electronic device including a memory storing computer readable instructions, and a processor executing the computer readable instructions stored in the memory to implement the log processing method.

A fourth aspect of the embodiments of the present application provides a computer-readable storage medium having stored therein computer-readable instructions that are executed by a processor in an electronic device to implement the log processing method.

In the embodiments of the present application, based on the attribute information, the log content in the collected log is stored in the partition corresponding to the message middleware, and because the attribute information corresponding to the log with the same log content is generally the same, the log with the same log content can be ensured to be stored in the same partition, so that when the log data in each partition is subjected to the deduplication process, the redundant log can be completely deleted, and the accuracy of log deduplication is improved. In addition, since log data in each partition is subjected to respective deduplication, deduplication efficiency can be improved. By carrying out deduplication processing on the log data in each partition, occupation of redundant logs on a database can be reduced, and analysis of the logs can be facilitated.

Drawings

Fig. 1 is a schematic structural diagram of an electronic device for implementing a log processing method according to an embodiment of the present application.

Fig. 2 is a flowchart of a log processing method according to an embodiment of the present application.

Fig. 3 is a flowchart of a log processing method according to another embodiment of the present application.

Fig. 4 is a schematic diagram of a configuration interface for log collection according to an embodiment of the present application.

Fig. 5 is a flowchart of a log processing method according to another embodiment of the present application.

Fig. 6 is a flowchart of a log processing method according to another embodiment of the present application.

Fig. 7 is a functional block diagram of a log processing device according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described in detail with reference to the accompanying drawings and specific embodiments.

In the present application, "at least one" means one or more, and "a plurality" means two or more. "and/or" describes an association relationship of associated objects, meaning that there may be three relationships, e.g., A and/or B may mean that A alone exists, while A and B together exist, and B alone exists, where A, B may be singular or plural. The terms "first," "second," "third," "fourth" and the like in the description and in the claims and drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order.

In embodiments of the application, words such as "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "e.g." in an embodiment should not be taken as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion. The following embodiments and features of the embodiments may be combined with each other without conflict.

In modern information systems, journals may record state information of an application during running, operational behavior of a user during use of the application, and the like. By analyzing the log, valuable information can be mined. For example, the log is analyzed to assist operation and maintenance personnel in rapidly removing faults, and for example, the log is analyzed to assist sales personnel in rapidly knowing portrait information of users.

In order to avoid missing the collected logs due to network problems, the logs are usually collected in batches periodically, and a large number of logs are often collected repeatedly in the mode, so that the redundant logs occupy the storage space, and meanwhile, the log processing efficiency is also affected.

Based on the above, the embodiment of the application provides a log processing method, which stores the logs with the same log content into the same partition, so that redundant logs can be completely deleted when the log data in each partition is subjected to the duplicate removal processing, and the accuracy of log duplicate removal is improved.

In addition, the related art has difficulty in precisely controlling the timing and range of collection when collecting logs, resulting in that critical log information cannot be collected timely or precisely, thereby affecting the effect of log analysis.

Based on the above, the embodiment of the application provides another log processing method, which is based on the pre-configured white list information and the pre-configured black list information to collect logs, and because the information to be collected is recorded in the white list information and the information not to be collected is recorded in the black list information, the logs can be accurately collected, so that the key logs are prevented from missing to collect, or irrelevant logs are collected, and the analysis of the logs is facilitated.

In the embodiment of the present application, the log processing method is applied to one or more electronic devices 1, where the electronic devices 1 are devices capable of executing computer readable instructions to automatically perform numerical computation and/or information processing, and the hardware includes, but is not limited to, a microprocessor, an Application SPECIFIC INTEGRATED Circuit (ASIC), a Programmable gate array (Field-Programmable GATE ARRAY, FPGA), a digital signal Processor (DIGITAL SIGNAL Processor, DSP), an embedded device, and the like.

The electronic device 1 may be any electronic product that can perform man-machine interaction with a user, such as a Personal computer, a tablet computer, a smart phone, a Personal digital assistant (Personal DIGITAL ASSISTANT, PDA), a game console, an interactive internet protocol television (Internet Protocol Television, IPTV), a smart wearable device, etc.

The electronic device 1 may comprise a network device and/or a user device. Wherein the network device includes, but is not limited to, a single network electronic device, a group of electronic devices made up of multiple network electronic devices, or a Cloud based Cloud Computing (Cloud Computing) made up of a large number of hosts or network electronic devices.

The network in which the electronic device 1 is located includes, but is not limited to, the internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (Virtual Private Network, VPN), and the like.

In an embodiment of the application, the electronic device 1 includes, but is not limited to, a memory 12, a processor 13, and computer readable instructions, such as a log handler, stored in the memory 12 and executable on the processor 13.

It will be appreciated by those skilled in the art that the schematic diagram is merely an example of the electronic device 1 and does not constitute a limitation of the electronic device 1, and may include more or fewer components than shown, or may combine certain components, or different components, e.g. the electronic device 1 may also include input-output devices, network access devices, buses, etc.

The Processor 13 may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. The general purpose processor may be a microprocessor or a processor, or any conventional processor, etc., and the processor 13 is an operation core and a control center of the electronic device 1, connects various parts of the entire electronic device 1 using various interfaces and lines, and executes an operating system of the electronic device 1 and various applications, program codes, etc. installed.

The memory 12 may be an external memory and/or an internal memory of the electronic device 1. Further, the memory 12 may be a physical memory, such as a memory bank, a TF card (Trans-FLASH CARD), or the like.

In connection with fig. 2, 3, 5, and 6, the memory 12 in the electronic device 1 stores computer readable instructions, and the processor 13 can execute the computer readable instructions stored in the memory 12 to implement a plurality of processes as shown in fig. 2, 3, 5, and 6 to implement a log processing method.

Fig. 2 is a flowchart of a log processing method according to an embodiment of the present application. The order of the steps in the flowchart may be changed and some steps may be omitted according to various needs. The log processing method provided by the embodiment of the application comprises the following steps of.

S201, determining attribute information of the collected logs.

In at least one embodiment of the application, the attribute information includes, but is not limited to, log record time, log grade, address information corresponding to the source terminal of the log, server name corresponding to the consuming server of the log, function name, and location information stored in the source terminal of the log. The log levels may include, among other things, debug levels, information levels, alert levels, error levels, and the like. The source terminal may represent a terminal device that generates a log and the consumption server may represent a server that consumes the log.

In at least one embodiment of the present application, the electronic device may collect, from a header of the log, a log recording time, a log level, address information corresponding to a source terminal of the log, and a server name corresponding to a consumption server of the log. The electronic device may collect the function name from the method body of the log. The electronic equipment takes the acquisition position of the log as position information.

And S202, storing the log content of the log to a corresponding partition in the message middleware based on the attribute information.

In at least one embodiment of the application, the log content may include information in the method body of the log. The partitions corresponding to different attribute information are different, for example, if the log level of the log a is a debug level, the partition corresponding to the log a in the message middleware may be part 0, and if the log level of the log B is an error level, the partition corresponding to the log a in the message middleware may be part 1. For another example, if the location information of the log C is IP1, the partition corresponding to the log C in the message middleware may be part 0, and if the location information of the log D is IP2, the partition corresponding to the log D in the message middleware may be part 1.

In at least one embodiment of the application, the electronic device stores the log content of the log to the corresponding partition in the message middleware based on the attribute information, and comprises the steps of determining the partition corresponding to the attribute information from the message middleware, and storing a key value pair consisting of the attribute information and the log content based on the partition corresponding to the attribute information.

The message middleware stores a correspondence between attribute information and partition identifiers, where the partition identifiers are used to uniquely identify different partitions, for example, partition identifier topic0 part 0, partition identifier topic0 part 1 and partition identifier topic1part 0 respectively represent different partitions.

For example, the attribute information includes log level and location information, the log level included in the attribute information s1 is debug level, the location information included in the attribute information s1 is IP1, the log level included in the attribute information s2 is error level, the location information included in the attribute information s2 is IP2, the partition corresponding to the attribute information s1 is topic0 partition0, and the partition corresponding to the attribute information s2 is topic0 partition1. If the log level of the log A is the debug level, the position information of the log A is IP1, the log level of the log B is the error level, and the position information of the log B is IP2. The log level (debug level) and location information (IP 1) of the log a are used as a first key, the log content of the log a is used as a first value, and a first key value pair (including a first key value pair corresponding to the log A key and first value) to partial 0. The log level (error level) and the position information (IP 2) of the log B are used as a second key, the log content of the log B is used as a second value, and a second key value pair (including the second key and the second value) of the log B is stored in the partial 0 part 1.

In the embodiment, the corresponding partition is determined by the attribute information, and the same log content and the same attribute information are corresponding to the same log, so that the log with the same log content can be ensured to be stored in the same partition.

In another embodiment, the electronic device stores the log content of the log into a corresponding partition in the message middleware based on the attribute information, and further includes converting the attribute information into a hash value, determining the partition corresponding to the hash value from the message middleware, and storing a key value pair consisting of the hash value and the log content based on the partition corresponding to the hash value. Wherein hash values corresponding to different attribute information are different. The message middleware stores the corresponding relation between the hash value and the partition identification.

In the above example, the log level (debug level) and the position information (IP 1) of the log a are converted into the first hash value, and the log level (error level) and the position information (IP 2) of the log B are converted into the second hash value. And forming a first key value pair by the first hash value and the log content of the log A, forming a second key value pair by the second hash value and the log content of the log B, storing the first key value pair into the partial 0 of the topic, and storing the second key value pair into the partial 1 of the topic.

In at least one embodiment of the present application, key-value pairs are stored sequentially according to the order of log-recording times.

In at least one embodiment of the application, each partition comprises a master copy and a slave copy, and the method further comprises the step that if the master copy and the slave copy both receive a key value pair, the electronic device returns a response code to the source terminal of the log, wherein the response code is used for prompting that the key value pair is successfully stored. In the embodiment, when the key value pairs are received by both the master copy and the slave copy, the answer code prompt key value pair is returned to be successfully stored, so that successful log storage and backup can be ensured, and the validity of log storage is ensured.

S203, performing duplicate removal processing on the log data in each partition, and storing the duplicate removed log data in a database.

In at least one embodiment of the application, the database may be a database supporting idempotent writing or transactional storage, e.g., the database may include a MySQL database, an Oracle database, or the like. According to the embodiment, the log data after the duplication removal is stored in the database supporting idempotent writing, so that the phenomenon that the log data are repeatedly written into the database due to network or system faults can be avoided, and the duplication removal effectiveness is improved. According to the embodiment, by storing the log data after the duplication elimination in the database supporting transactional storage, two-stage submission (Two-phase Commit Protocol,2 PC) can be combined, so that the log data after the duplication elimination is ensured to be successfully stored in the database, and log storage failure caused by network problems or system faults is avoided.

In at least one embodiment of the application, the deduplication processing is performed on the log data in each partition, and the method comprises the steps of grouping the log data in each partition based on a sliding window to obtain a plurality of log groups, repeatedly detecting the log data in the plurality of log groups, and deleting the detected repeated log data.

The sliding window may be a time window, the length of the sliding window may be set to a preset duration, the preset duration may be set and adjusted according to requirements, for example, the starting time point of the first sliding window is 9:00, the preset duration is 1 hour, the first sliding window may be 9:00-10:00, and the second sliding window may be 10:00-11:00.

In order to avoid missing the log collection due to network problems, the log collection is usually performed periodically, for example, the period set in the first log collection may be 9:00-9:30, and the period set in the second log collection may be 9:10-9:40, so if no network problem occurs, the log collection of 9:10-9:30 will be repeated. In order to delete the redundant logs of 9:10-9:30, the preset duration can be determined according to the period set when the logs are collected. According to the above example, the preset time period may be set to a time period of 9:10 to 9:30, that is, 20 minutes. The preset duration is determined by the period set when the logs are collected, so that the same log content can be ensured to be in the same log group.

Each partition is provided with a plurality of log groups, the number of the logs in each log group can be the same, and the number of the logs in each log group can also be different. For example, for partition topic0 partition1, there may correspond to 2 log packets, the first log packet may include 100 logs, and the second log packet may include 120 logs.

According to the embodiment, the log data in each partition are grouped through the sliding window, so that a plurality of log groups are obtained, the log groups are respectively and repeatedly detected in parallel, the detection efficiency of the redundant log can be improved, and the deletion efficiency of the redundant log is improved. In addition, since the same log content is stored in the same partition and the logs are stored according to the log recording time, the same log content is usually stored in the same log packet, and further by repeatedly detecting the log packet, the detection of the redundant logs can be further improved due to the relatively small number of the logs in the log packet, so that the deletion efficiency of the redundant logs is improved.

In at least one embodiment of the application, storing the de-duplicated log data in a database includes creating a data table corresponding to the configured table information in the database based on the configured table information, and writing the de-duplicated log data into the data table.

The configured table information may include, but is not limited to, an item name, a database name, a table name, and a separator between any two log data, wherein the item name may indicate an item to which the log belongs, the database name may indicate a database in which the log stores, and the table name may indicate a name of a created data table.

According to the embodiment, through the configured table information, the data table corresponding to the configured table information can be created in the database, so that log data after duplicate removal is stored when the data table is not included in the database.

In an embodiment, a log is collected from a local directory of a source terminal, the collected log is stored in a server of a corresponding area, the logs in the servers corresponding to a plurality of areas are stored in a central server, the logs are stored in corresponding partitions in a message middleware based on attribute information of the logs, duplicate removal processing is performed on log data in each partition, and the duplicate removed log data is stored in a database. For example, the areas where the source terminals Z1 and Z2 are located are the area Q1, the areas where the source terminals Z3, Z4 and Z5 are located are the area Q2, the logs collected from the source terminals Z1 and Z2 are stored in the server F1 corresponding to the area Q1, the logs collected from the source terminals Z3, Z4 and Z5 are stored in the server F2 corresponding to the area Q2, the server F1 and the server F2 respectively store the logs in the central server, and the central server may be a server corresponding to any designated area.

In the embodiments of the present application, the collected log content in the log is stored in the corresponding partition in the message middleware based on the attribute information, and because the attribute information corresponding to the log with the same log content is generally the same, the log with the same log content can be ensured to be stored in the same partition, so that when the log data in each partition is subjected to the deduplication process, the redundant log can be completely deleted, and the accuracy of log deduplication is improved. In addition, since log data in each partition is subjected to respective deduplication, deduplication efficiency can be improved. By carrying out deduplication processing on the log data in each partition, occupation of redundant logs on a database can be reduced, and analysis of the logs can be facilitated. For example, abnormal conditions in the system can be timely captured and recorded through log collection, so that an operation and maintenance user can rapidly locate the system problem, the influence of downtime and service interruption of the system is reduced, the operation and maintenance user can be helped to rapidly analyze faults, the repair process is accelerated, and the service availability and stability are improved. For another example, by collecting logs related to system performance, bottlenecks in the system, such as high-load resources, slow queries, memory leaks, etc., can be identified, thereby providing data support for system performance tuning. The health condition of each part of the system can be monitored, and potential performance problems can be found in time, so that preventive maintenance can be performed. For another example, the collected logs may be used to track user behavior, detect abnormal activity, and potential security threats, and timely identify and respond to security events. For another example, during the development and testing phases, the log may help the developer understand code behavior and verify functional correctness.

Fig. 3 is a flowchart of a log processing method according to another embodiment of the present application. The order of the steps in the flowchart may be changed and some steps may be omitted according to various needs. The log processing method provided by the embodiment of the application comprises the following steps of.

S301, collecting logs based on pre-configured white list information and black list information.

In at least one embodiment of the present application, the white list information and the black list information may be set according to the acquisition requirements. For easy understanding, as shown in fig. 4, fig. 4 is a configuration interface for log collection provided in an embodiment of the present application, and a user may input white list information and black list information in the configuration interface for log collection. For example, the white list information may be key=my_product_name_env, value= hago as shown in fig. 4. The user clicks an 'add' button at the environment variable white list in the configuration interface for log acquisition, and an input box of white list information can be displayed. The user clicks an 'add' button at the environment variable blacklist in the configuration interface of log collection, and an input box of blacklist information can be displayed.

In at least one embodiment of the present application, the electronic device detects whether the log includes white list information, if the log includes white list information, detects whether the log includes black list information, and if the log does not include black list information, acquires the log. If the log does not include white list information or the log includes black list information, the collection of the log is skipped. According to the embodiment, through the pre-configured white list information and the pre-configured black list information, the logs can be acquired according to actual acquisition requirements, and individuation of log acquisition is improved.

In at least one embodiment of the present application, a log agent is deployed on a source terminal of a log, the log is collected in real time, and the collected log is sent to a message queue system. The log agents may include, but are not limited to Fluentd agents, logstack agents, filebeat agents.

S302, determining attribute information of the collected logs.

And S303, storing the log content of the log into a corresponding partition in the message middleware based on the attribute information.

S304, performing duplicate removal processing on the log data in each partition, and storing the duplicate removed log data in a database.

For details of steps S302-S304, reference is made to the above detailed description of steps S201-S203 in fig. 2, and the description is not repeated here.

In this embodiment, the logs can be collected according to the actual collection requirement by the pre-configured white list information and black list information, so that the key logs can be collected, and analysis of the logs is facilitated. And the irrelevant logs can be prevented from being collected, and analysis of the logs is prevented from being influenced.

Fig. 5 is a flowchart of a log processing method according to another embodiment of the present application. The order of the steps in the flowchart may be changed and some steps may be omitted according to various needs. The log processing method provided by the embodiment of the application comprises the following steps of.

S501, determining attribute information of the collected logs.

S502, based on the attribute information, storing the log content of the log into a corresponding partition in the message middleware.

S503, performing duplication elimination processing on the log data in each partition, and storing the duplicated log data in a database.

For details of steps S501-S503, reference is made to the above detailed description of steps S201-S203 in fig. 2, and the description is not repeated here.

S504, identifying the log data with abnormal information from the log data after the duplication removal based on a preset rule.

In at least one embodiment of the application, the electronic equipment identifies the log data with the abnormal information from the log data after the duplication removal based on the preset rule, and comprises the steps of detecting whether the log data after the duplication removal comprises the set information in the preset rule or not, and if the log data after the duplication removal comprises the set information in the preset rule, determining the log data after the duplication removal as the log data with the abnormal information.

S505, the log data with the abnormal information is moved to a designated partition of the message middleware.

In at least one embodiment of the present application, log data with exception information is stored in a designated partition, which is used to analyze exception behavior, system anomalies, and the like.

In this embodiment, through a preset rule, log data with abnormal information can be accurately identified from the log data after duplication removal, and then the log data with abnormal information is moved to a designated partition of the message middleware, and the log data with abnormal information is stored in the designated partition in a special manner, so that analysis of the log by an operation and maintenance user is facilitated.

Fig. 6 is a flowchart of a log processing method according to another embodiment of the present application. The order of the steps in the flowchart may be changed and some steps may be omitted according to various needs. The log processing method provided by the embodiment of the application comprises the following steps of.

S601, determining attribute information of the collected logs.

And S602, storing the log content of the log to a corresponding partition in the message middleware based on the attribute information.

For details of steps S601-S602, reference is made to the above detailed description of steps S201-S202 in fig. 2, and the description is not repeated here.

S603, detecting whether the total amount of the logs in the message middleware is the same as the number of the collected logs.

In at least one embodiment of the present application, if the total amount of logs in the message middleware is the same as the number of collected logs, S604 is performed, and if the total amount of logs in the message middleware is different from the number of collected logs, S605 is performed.

S604, performing deduplication processing on the log data in each partition, and storing the log data subjected to deduplication into a database.

For details of step S604, reference is made to the above detailed description of step S203 in fig. 2, and the description thereof will not be repeated here.

S605, generating prompt information.

In at least one embodiment of the present application, the prompt information is used to indicate that the collected log has an abnormal condition. If the total amount of the logs in the message middleware is smaller than the number of the collected logs, generating prompt information for indicating that the logs are lost. If the total amount of the logs in the message middleware is larger than the amount of the logs in the source terminal, generating prompt information for indicating log redundancy. Since logs are typically collected periodically (to address network problems resulting in data loss), the number of logs collected is typically greater than the number of logs in the source terminal.

In this embodiment, when the total amount of the logs in the message middleware is the same as the number of the collected logs, it may be indicated that the collected logs have no abnormal condition of log loss, and then the log data in each partition is further subjected to the deduplication processing, so that the log data cannot be comprehensively subjected to the deduplication processing is avoided, and thus the deduplication effectiveness is improved. When the total amount of the logs in the message middleware is different from the collected logs, prompt information can be generated for prompting whether the logs are lost or not.

Fig. 7 is a functional block diagram of a log processing device according to an embodiment of the present application. The log processing device 11 includes a determination unit 110, a storage unit 111, a processing unit 112, an acquisition unit 113, a transmission unit 114, an identification unit 115, a movement unit 116, and a generation unit 117. The module/unit referred to herein is a series of computer readable instructions capable of being retrieved by the processor 13 and performing a fixed function and stored in the memory 12.

In one embodiment, the determining unit 110 is configured to determine attribute information of the collected log, the storage unit 111 is configured to store log content of the log to a corresponding partition in the message middleware based on the attribute information, and the processing unit 112 is configured to perform deduplication processing on log data in each partition and store the log data after deduplication to the database.

In one embodiment, before determining the attribute information of the collected log, the collection unit 113 is configured to collect the log based on the preconfigured white list information and black list information.

In one embodiment, the storage unit 111 is specifically configured to determine a partition corresponding to the attribute information from the message middleware, and store a key value pair formed by the attribute information and the log content based on the partition corresponding to the attribute information.

In one embodiment, each partition includes a master copy and a slave copy, and the sending unit 114 is configured to return a response code to the source terminal of the log if the master copy and the slave copy both receive the key value pair, where the response code is used to prompt that the key value pair is stored successfully.

In one embodiment, the processing unit 112 is specifically configured to group the log data in each partition based on a sliding window to obtain a plurality of log groups, repeatedly detect the log data in the plurality of log groups, and delete the detected repeated log data.

In one embodiment, after performing the deduplication process on the log data in each partition, the identifying unit 115 is configured to identify log data having anomaly information from the log data after the deduplication based on a preset rule, and the moving unit 116 is configured to move the log data having anomaly information to a specified partition of the message middleware.

In one embodiment, the processing unit 112 is specifically configured to create a data table corresponding to the configured table information in the database based on the configured table information, and write the log data after deduplication to the data table.

In an embodiment, the processing unit 112 is further configured to perform deduplication processing on the log data in each partition and store the deduplicated log data in the database if the total amount of the logs in the message middleware is the same as the number of the collected logs, and the generating unit 117 is configured to generate prompt information if the total amount of the logs in the message middleware is different from the number of the collected logs, where the prompt information is used to indicate that the collected logs are abnormal.

In the embodiments of the present application, the collected log content in the log is stored in the corresponding partition in the message middleware based on the attribute information, and because the attribute information corresponding to the log with the same log content is generally the same, the log with the same log content can be ensured to be stored in the same partition, so that when the log data in each partition is subjected to the deduplication process, the redundant log can be completely deleted, and the accuracy of log deduplication is improved. In addition, since log data in each partition is subjected to respective deduplication, deduplication efficiency can be improved. By carrying out deduplication processing on the log data in each partition, occupation of redundant logs on a database can be reduced, and analysis of the logs can be facilitated.

The integrated modules/units of the electronic device 1 may be stored in a computer readable storage medium if implemented in the form of software functional units and sold or used as a stand alone product. Based on such understanding, the present application may implement all or part of the flow of the method of the above-described embodiments, or may be implemented by means of computer readable instructions to instruct related hardware, where the computer readable instructions may be stored in a computer readable storage medium, where the computer readable instructions, when executed by a processor, implement the steps of the method embodiments described above.

The computer readable instructions include computer readable instruction code, which may be in the form of source code, object code, executable files, or some intermediate form, etc. The computer readable medium may include any entity or device capable of carrying computer readable instruction code, a recording medium, a USB flash disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory).

In particular, the specific implementation method of the processor 13 on the computer readable instructions may refer to descriptions of related steps in the corresponding embodiments of fig. 2, 3,5 and 6, which are not repeated herein.

In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of modules is merely a logical function division, and other manners of division may be implemented in practice.

The modules illustrated as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, may be located in one place, or may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional module in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units can be realized in a form of hardware or a form of hardware and a form of software functional modules.

The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.

Furthermore, it is evident that the word "comprising" does not exclude other elements or steps, and that the singular does not exclude a plurality. Also, the plurality of units or means of (a) may be implemented by one unit or means by software or hardware. The terms first, second, etc. are used to denote a name, but not any particular order.

Finally, it should be noted that the above-mentioned embodiments are merely for illustrating the technical solution of the present application and not for limiting the same, and although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made to the technical solution of the present application without departing from the spirit and scope of the technical solution of the present application.

Claims

1. A log processing method, the method comprising:

Determining attribute information of the collected logs;

based on the attribute information, storing the log content of the log to a corresponding partition in the message middleware;

and carrying out de-duplication processing on the log data in each partition, and storing the log data subjected to de-duplication into a database.

2. The log processing method as defined in claim 1, wherein before determining the attribute information of the collected log, the method further comprises:

The log is collected based on the pre-configured white list information and black list information.

3. The method according to claim 1, wherein storing the log content of the log to a corresponding partition in a message middleware based on the attribute information comprises:

determining a partition corresponding to the attribute information from the message middleware;

and storing key value pairs formed by the attribute information and the log content based on the partition corresponding to the attribute information.

4. The log processing method as defined in claim 3 wherein each partition comprises a master copy and a slave copy, the method further comprising:

And if the master copy and the slave copy both receive the key value pair, returning a response code to the source terminal of the log, wherein the response code is used for prompting that the key value pair is successfully stored.

5. The log processing method as set forth in claim 1, wherein said performing a deduplication process on the log data in each partition comprises:

grouping the log data in each partition based on the sliding window to obtain a plurality of log groups;

repeatedly detecting the log data in the plurality of log packets, and deleting the detected repeated log data.

6. The log processing method as defined in claim 1, wherein after performing the deduplication processing on the log data in each partition, the method further comprises:

Identifying log data with abnormal information from the log data after duplication removal based on a preset rule;

and moving the log data with the abnormal information to a designated partition of the message middleware.

7. The log processing method as set forth in claim 1, wherein storing the deduplicated log data in a database comprises:

Creating a data table corresponding to the configured table information in the database based on the configured table information;

And writing the log data subjected to the duplication removal into the data table.

8. The log processing method according to any one of claims 1 to 7, characterized in that the method further comprises:

If the total amount of the logs in the message middleware is the same as the number of the collected logs, executing the duplicate removal processing on the log data in each partition, and storing the duplicate removed log data into a database;

if the total amount of the logs in the message middleware is different from the collected logs, generating prompt information, wherein the prompt information is used for indicating the collected logs to have abnormal conditions.

9. A log processing apparatus, the apparatus comprising:

The determining unit is used for determining attribute information of the collected logs;

The storage unit is used for storing the log content of the log to a corresponding partition in the message middleware based on the attribute information;

The processing unit is used for carrying out duplication elimination processing on the log data in each partition and storing the duplicated log data into the database.

10. An electronic device, comprising:

a memory storing computer readable instructions, and

A processor executing computer readable instructions stored in the memory to implement the log processing method of any one of claims 1 to 8.