CN114911817A - A data processing method, device, electronic device and storage medium - Google Patents

Abstract

The disclosure provides a data processing method and device, electronic equipment and a storage medium, and relates to the field of data processing, in particular to the field of databases. The specific implementation scheme is as follows: determining at least one target query statement meeting a specified condition from the at least one query statement; performing privacy removal processing on the at least one target query statement respectively to obtain at least one statement to be utilized; clustering the at least one sentence to be utilized according to the text content of the at least one sentence to be utilized to obtain at least one sentence cluster; and outputting a corresponding processing task aiming at each statement cluster in the at least one statement cluster. By the scheme, the user information can be considered not to be leaked, and the slow query statement can be efficiently processed.

Description

A data processing method, device, electronic device and storage medium

technical field

The present disclosure relates to the field of data processing, in particular to the field of databases, and in particular to a data processing method, apparatus, electronic device and storage medium.

Background technique

The business system can generate query statements for the database based on the query information given by the user, such as SQL (Structured Query Language, Structured Query Language) statements, and send the generated query statements to the database system, so that the database system can process data. Inquire.

However, for database query, there is a problem of slow query, for example, a problem of slow SQL (Structured QueryLanguage, structured query language). Because the execution time of slow queries is too long, it has always been an important factor affecting database performance, and greatly affects user experience.

SUMMARY OF THE INVENTION

The present disclosure provides a data processing method, apparatus, electronic device, and storage medium.

According to an aspect of the present disclosure, a data processing method is provided, comprising:

From at least one query statement, determine at least one target query statement that meets a specified condition; wherein, the specified condition includes that the execution duration exceeds a predetermined duration threshold;

Perform deprivation processing on the at least one target query statement respectively to obtain at least one statement to be utilized;

Clustering the at least one sentence to be used according to the text content of the at least one sentence to be used to obtain at least one sentence cluster;

For each sentence cluster in the at least one sentence cluster, a corresponding processing task is output.

According to a second aspect of the present disclosure, there is provided a data processing apparatus, comprising:

a first determination module, configured to determine, from at least one query statement, at least one target query statement that meets a specified condition; wherein the specified condition includes that the execution duration exceeds a predetermined duration threshold;

a first processing module, configured to perform deprivation processing on the at least one target query statement to obtain at least one to-be-utilized statement;

a clustering module, configured to perform clustering on the at least one sentence to be used according to the text content of the at least one sentence to be used to obtain at least one sentence cluster;

The output module is configured to output a corresponding processing task for each sentence cluster in the at least one sentence cluster.

According to a third aspect of the present disclosure, there is provided an electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform any one of the data processing methods.

According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform any data processing method.

According to a fifth aspect of the present disclosure, there is also provided a computer program product, comprising a computer program that, when executed by a processor, implements any one of the data processing methods.

It should be understood that what is described in this section is not intended to identify key or critical features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.

Description of drawings

The accompanying drawings are used for better understanding of the present solution, and do not constitute a limitation to the present disclosure. in:

1 is a flowchart of a data processing method provided by an embodiment of the present disclosure;

FIG. 2 is another flowchart of the data processing method provided by an embodiment of the present disclosure;

3 is a schematic diagram of a processing interface for a processing task provided by an embodiment of the present disclosure;

4 is a schematic diagram of a system architecture to which the data processing method provided by an embodiment of the present disclosure is applicable;

5 is a schematic structural diagram of a data processing apparatus provided by an embodiment of the present disclosure;

6 is another schematic structural diagram of a data processing apparatus provided by an embodiment of the present disclosure;

FIG. 7 is a block diagram of an electronic device used to implement the data processing method of an embodiment of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

As a user-oriented system, the business system can generate query statements for the database based on the query information given by the user, such as SQL (Structured Query Language, Structured Query Language) statements, and send the generated query statements to the database system. In order to make the database system perform data query. Exemplarily, the business system may be a library platform, a cloud storage platform, various information processing systems, and so on.

However, for database query, due to factors such as insufficient optimization of the code content of the business system, a slow query problem may be caused, such as a slow SQL (Structured Query Language, structured query language) problem. Because the execution time of slow queries is too long, it has always been an important factor affecting database performance, and greatly affects user experience. In addition, with the complexity of database architecture and the increase of massive data, the problems brought by slow queries are becoming more and more prominent: leading to synchronization delay of the entire database cluster, master-slave delay of business; large consumption of host and database resources, affecting the performance of other queries. Performance, database throughput capacity declines; query requests pile up, which eventually leads to slow or even completely unresponsive database services, and so on.

There is currently a simple way to deal with slow SQL, but it only identifies the slow SQL in the development environment and then notifies the corresponding handler, so as to carry out corresponding processing, and does not deal with the existing slow SQL problem in the online business system. And since the identified slow SQL may contain user information, user information may be leaked.

Based on the above problems, the embodiments of the present disclosure provide a data processing method, apparatus, electronic device, and storage medium, so as to achieve the purpose of efficiently processing slow query sentences while keeping user information from being leaked.

The following first introduces a data processing method provided by an embodiment of the present disclosure.

Wherein, a data processing method provided by an embodiment of the present disclosure may be applied to an electronic device, and the electronic device may be a terminal device or a server, and the present disclosure does not limit the specific form of the electronic device. In addition, a data processing method provided by an embodiment of the present disclosure may be applied to a cluster, a distributed application, or any other application scenario that requires processing a slow query statement, and the embodiment of the present disclosure does not limit the specific scenario.

In addition, the execution body of the data processing method provided by the embodiment of the present disclosure may be a data processing apparatus. Exemplarily, when the data processing method is applied to a terminal device, the data processing device may be functional software running in the terminal device, such as tool software for processing slow query sentences; It can be a plug-in in an existing client, eg, a plug-in in a client for software processing. Exemplarily, when the data processing method is applied to a server, the data processing apparatus may be a computer program running in the server, and the computer program may be used to initiate slow query statement processing.

Moreover, the slow query involved in the present disclosure refers to a query whose duration exceeds a predetermined time threshold, and correspondingly, a slow query statement is a query whose query duration exceeds the predetermined duration threshold. Wherein, the predetermined duration threshold can be set according to the actual situation, for example: for a query scenario that requires instant response, the predetermined duration threshold can be, for example, 0.5 seconds, 1 second, 2 seconds, etc.; for others, the response speed requirements are lower For the scenario of , the predetermined duration threshold may be, for example, 0.5 minutes, 1 minute, and so on.

In addition, exemplarily, for a database that uses SQL statements to query, the so-called slow query problem may be referred to as the slow SQL problem. The present disclosure does not specifically limit the types of query statements utilized by the slow query.

A data processing method provided by an embodiment of the present disclosure may include the following steps:

From at least one query statement, determine at least one target query statement that meets a specified condition; wherein, the specified condition includes that the execution duration exceeds a predetermined duration threshold;

Perform deprivation processing on the at least one target query statement respectively to obtain at least one statement to be utilized;

Clustering the at least one sentence to be used according to the text content of the at least one sentence to be used to obtain at least one sentence cluster;

For each sentence cluster in the at least one sentence cluster, a corresponding processing task is output.

In this solution, after at least one target query statement in at least one query statement that meets the specified conditions is determined, the at least one target query statement is deprived of privacy processing, so that information is prevented from being leaked; and, considering the deprivation of privacy The at least one sentence to be used after the transformation process contains a sentence with common content, therefore, the at least one sentence to be used is clustered, and the processing task is output according to each sentence cluster in the at least one sentence cluster obtained by the clustering, In this way, the number of processing tasks is greatly reduced compared to setting processing tasks for each target query statement. It can be seen that through this solution, slow query statements can be efficiently processed while taking into account that user information is not leaked.

A data processing method provided by the present disclosure will be exemplarily introduced below with reference to the accompanying drawings.

As shown in FIG. 1, a data processing method provided by the present disclosure may include the following steps:

S101: From at least one query statement, determine at least one target query statement that meets a specified condition;

Wherein, the specified condition includes that the execution duration exceeds a predetermined duration threshold;

In the embodiment of the present disclosure, log data is generated when the target business system is running, and the log data includes at least one query statement; then, at least one target query statement that meets the specified condition can be determined from the at least one query statement, so that the target query statement that meets the specified condition can be determined. At least one target query statement of the condition undergoes subsequent processing. Wherein, the specified condition includes that the execution time exceeds a predetermined time length threshold. Correspondingly, the target query statement may be a slow query statement. In addition, in practical applications, the slow query statement may also be referred to as a statement with a slow query state or a slow query statement. question statement. Exemplarily, the business system may be a library platform, a cloud storage platform, various information processing systems, and so on.

The specific implementation manner of determining at least one target query statement that meets the specified condition from the at least one query statement is not limited in the embodiment of the present disclosure, and any method that can identify whether the query statement meets the specified condition can be applied. in the embodiments of the present disclosure. Exemplarily, from at least one query statement, the method of determining at least one target query statement that meets the specified condition may include: obtaining at least one query statement; for each query statement, if based on the execution time and/or EXPLAIN of the query statement As a result, it is recognized that the query statement meets the specified condition, and the query statement is determined as the target query statement. Wherein, if it is determined whether the query statement meets the specified condition based on the execution time of the query statement, a predetermined duration threshold can be preset, and when the query statement is identified, the execution duration of the query statement is compared with the predetermined duration threshold. Yes, if it exceeds the predetermined duration threshold, it is determined that the query statement meets the specified conditions; if it is identified based on the EXPLAIN result of the query statement whether the query statement meets the specified conditions, the EXPLAIN command (used to query the query statement) can be executed. Check the query execution plan), get the grammatical analysis result, that is, the EXPLAIN result, and evaluate the EXPLAIN result based on whether the score meets the score conditions set for the slow query, that is, whether the score meets the specified conditions. The set score conditions are used to determine whether the query statement meets the specified conditions; and if the query statement meets the specified conditions based on the execution time of the query statement and the EXPLAIN result, it can be executed when the execution time exceeds the predetermined duration threshold and the score is evaluated. When the score condition set for the slow query is satisfied, it is determined that the query statement meets the specified condition. It can be understood that the EXPLAIN result can include multiple columns: query type (select_type), scan method (type), number of rows scanned (rows), index actually used (key), etc., which can affect the query efficiency. Different values of the column set different evaluation scores, so as to identify whether the specified conditions are met based on the total evaluation score.

In addition, in practical applications, considering that under various operating environments, the target business system can call the database system to generate log data, and the generated log data includes at least one query statement. Therefore, the desired business system can be flexibly selected. At least one query statement of the running environment, to process the slow query in the specified running environment. Then, the method of determining at least one target query statement that meets the specified condition from the at least one query statement may include: from at least one query statement generated under the specified operating environment, determining at least one target query statement that meets the specified condition; wherein, The specified operating environment includes at least one of a research and development environment, a test environment, a heterogeneous environment, and a production environment. It should be noted that, in this embodiment of the present disclosure, log data generated in a specified operating environment can be collected, which can be real-time or scheduled, which is reasonable; the so-called R&D environment, that is, the development environment, is The development or programming at the code level; the so-called test environment is the environment for testing the code content generated by the R&D environment; the so-called heterogeneous environment, also known as the access environment, is the further testing of the test environment, which is completely Simulate the production environment, and further test the code content generated in the R&D environment; in the production environment, the code content developed at this time has been launched for users to use.

Based on the above implementation manner, the target query statement that meets the specified conditions in any specified operating environment can be flexibly processed, and finally the business code can be flexibly optimized. Moreover, multiple operating environments can be selected, so as to intercept and optimize slow queries layer by layer to achieve the effect of comprehensive processing.

It should be noted that the above description of the manner of specifying the operating environment and determining each target query statement that meets the specified conditions is only an example, and should not constitute a limitation of the present disclosure.

S102: Perform deprivation processing on at least one target query statement to obtain at least one statement to be utilized;

Among them, each target query statement may contain content related to user information. At this time, in order to ensure the security of information and protect personal privacy, the target query statement can be deprived, that is, the target query statement can be deprived. de-private user information. It can be understood that by depriving the user information in the target query sentence, data desensitization of the target query sentence can be realized.

Exemplarily, the method of depriving the target query statement may be: hiding key information, or replacing key information, where the key information here is user information that needs to be deprived. For example, if the target query statement contains personal name, address, etc., this information can be hidden, this part of the content can be not displayed, or this part of the content can be replaced with fixed characters, such as "? or *".

In addition, the target field related to the user information can be determined in advance, and when deprivation is performed, the field content corresponding to the target field is deprived. For example: a slow query statement is SELECT*FROM Persons WHEREcity='Beijing'; the preset target fields are: FROM, WHERE, then after deprivation processing, it is SELECT*FROM? WHERE%? ='? ’.

It should be emphasized that the above-mentioned specific implementation manner of depriving the target query statement is only an example, and should not constitute a limitation to the embodiments of the present disclosure. In addition, in the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user information involved are all in compliance with relevant laws and regulations, and do not violate public order and good customs.

S103: According to the text content of the at least one sentence to be used, cluster the at least one sentence to be used to obtain at least one sentence cluster;

Since the query statement is a statement formed by assigning specific query information to a certain type of query type statement, and the query information usually involves user information, in order to prevent user information from leaking, and considering the deprivation process At least one target query statement contains a statement with common content, and at least one statement to be used can be clustered to obtain at least one statement cluster, so as to perform subsequent data processing according to the clustering result, and finally realize the slow query Efficient processing of statements.

It can be understood that if at least one to-be-utilized statement is the same, a statement cluster can be obtained; if at least one to-be-utilized statement has the same statement, the number of available statement clusters is less than the number of at least one to-be-utilized statement. .

It should be noted that, according to the text content of the at least one to-be-used sentence, there may be various manners for clustering the at least one to-be-used sentence.

Exemplarily, in an implementation manner, the character string of at least one to-be-utilized sentence may be compared, and target query sentences with the same character string are grouped into a cluster, so as to obtain at least one sentence cluster.

Exemplarily, in another implementation manner, according to the text content of the at least one to-be-used sentence, clustering at least one to-be-used sentence to obtain at least one sentence cluster may include: for the at least one to-be-used sentence Hash operation is performed on each to-be-utilized statement in the statement to obtain a signature corresponding to the to-be-utilized statement; and at least one statement cluster is obtained according to the signature corresponding to the to-be-utilized statement.

The signature obtained by hashing the statement to be used can be a character string of 8, 16, or 32 bits, and the signature can be used as the identification of the target query statement, so as to be used for retrieval and analysis of the statement to be used. Through the hash operation, a string of a fixed size can be obtained, so that the sentences to be used can be clustered more efficiently according to the results of each hash operation. Wherein, according to the signature corresponding to the to-be-utilized sentence, the condition for obtaining at least one sentence cluster may be: the signatures are the same, or the signature similarity is greater than a predetermined threshold, and so on.

S104: For each statement cluster in the at least one statement cluster, output the corresponding processing task;

Wherein, the processing task is a task for instructing to perform specified processing on the statement cluster.

After the clustering of each target query sentence after deprivation is completed, the output of the processing task can be performed for each sentence cluster obtained by the clustering. Compared with the prior art, this solution does not perform task output for each slow query statement, thus avoiding a lot of manpower to process each slow query statement, thereby realizing efficient processing of the slow query statement. It should be noted that, in practical applications, the number of sentence clusters obtained after clustering is much less than the number of sentences before clustering.

Wherein, the designated processing may be a processing of optimizing the content of the business code corresponding to the statement in the statement cluster, and correspondingly, the processing task may be used to instruct the processing personnel to perform the content of the business code corresponding to the statement in the statement cluster. In this way, by outputting the processing task for the statement cluster, the processing personnel can locate the code content corresponding to the statement in the statement cluster, thereby completing the optimization of the code content. Of course, the specified processing may also be other types of processing, such as: processing of only locating the business code content corresponding to the statement in the statement cluster, etc., which is not limited in this embodiment of the present disclosure.

Exemplarily, in an implementation manner, outputting a corresponding processing task for each statement cluster in at least one statement cluster includes: for each statement cluster, determining a statement identifier of the statement cluster as a task of the statement cluster. The content, for example: the signature of the statement in the statement cluster as the task content; for each statement cluster, output the processing task including the statement identifier of the statement cluster. The statement identifier of each statement cluster is associated with a to-be-used statement of the statement cluster. In this way, the processing personnel can use the statement identifier of each statement cluster to obtain information from the data table containing the mapping relationship between the statement identifier and each statement cluster. , locate a to-be-used statement of the statement cluster, and then use the located statement to locate and optimize the subsequent code content.

Exemplarily, in an implementation manner, for each statement cluster in at least one statement cluster, outputting a corresponding processing task includes: for each statement cluster, determining a processing object corresponding to the statement cluster, and outputting the corresponding processing task. task; wherein, the processing object is a statement to be used in the statement cluster. It should be noted that, for the purpose of efficiently and intuitively processing slow query statements, when outputting the processing task for each statement cluster, the statement content involved in each statement cluster may be output. In addition, since the sentences contained in the same sentence cluster are the same, for each sentence cluster, any to-be-used sentence in the sentence cluster, that is, any query sentence after deprivation, can be used to form the corresponding sentence of the sentence cluster. task content.

In addition, in the case of determining at least one target query statement that meets the specified conditions from at least one query statement generated under the specified operating environment, for each obtained statement cluster, output corresponding processing tasks, including:

For each statement cluster in the at least one statement cluster, the processing task corresponding to the statement cluster is output to the processing terminal corresponding to the specified operating environment. The processing terminal corresponding to the designated operating environment is the processing terminal used by the processing personnel of the designated operating environment. By outputting processing tasks to the processing terminal corresponding to the designated operating environment, the processing personnel of the designated operating environment can be informed of the processing tasks. , so as to perform corresponding processing on the processing task.

In this solution, after at least one target query statement in at least one query statement that meets the specified conditions is determined, the at least one target query statement is deprived of privacy processing, so that information is prevented from being leaked; and, considering the deprivation of privacy The at least one sentence to be used after the transformation process contains a sentence with common content, therefore, the at least one sentence to be used is clustered, and the processing task is output according to each sentence cluster in the at least one sentence cluster obtained by the clustering, In this way, the number of processing tasks is greatly reduced compared to setting processing tasks for each target query statement. It can be seen that through this solution, slow query statements can be efficiently processed while taking into account that user information is not leaked.

Based on the above data processing method, in another embodiment of the present disclosure, another data processing method is also provided, as shown in FIG. 2 , which may include the following steps:

S201: From at least one query statement, determine at least one target query statement that meets a specified condition;

Wherein, the specified condition includes that the execution duration exceeds a predetermined duration threshold;

S202: Perform deprivation processing on at least one target query statement respectively to obtain at least one statement to be utilized;

S203: According to the text content of the at least one sentence to be used, cluster the at least one sentence to be used to obtain at least one sentence cluster;

S204: For each statement cluster in the at least one statement cluster, output the corresponding processing task;

Wherein, the processing task is a task for instructing to perform specified processing on the statement cluster;

Steps S201-S204 are similar to the above-mentioned contents of S101-S104, and will not be repeated here.

S205: Determine the query statement to be executed;

Since there are not only already executed query statements, but also new and unexecuted query statements during system operation; therefore, when any unexecuted query statement is received, the unexecuted query statement can be regarded as pending execution the query statement, so that the subsequent data processing steps are performed for the query statement to be executed.

It should be noted that the to-be-executed query statement may be a query statement that has never been executed, or may be an improved query statement for a previously executed query statement, which is not limited herein.

S206: Perform deprivation processing on the query statement to be executed to obtain the statement to be analyzed;

When the target business system generates a business request with a query statement, that is, when a query statement to be executed is generated, before the query statement to be executed is sent to the database system, the user information in the query statement to be executed can be deprived of privacy processing. The statement to be analyzed is obtained, and the steps of the subsequent processing method are performed on the statement to be analyzed, so as to achieve real-time management and control of the slow query statement.

The manner of performing the deprivation processing on the user information in the query statement to be executed is the same as the manner in which the deprivation processing of the user information in the target query statement is performed above. Exemplarily, in an implementation manner, the way of depriving the query statement to be executed is: hide the part of the query statement to be executed that contains user information, or replace the information; Parts of information such as addresses are replaced with "? or *", etc., of course, it is not limited to this.

In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user information involved are all in compliance with relevant laws and regulations, and do not violate public order and good customs.

S207: If it is detected that the statement to be analyzed belongs to any one of at least one statement cluster, the query statement to be executed is intercepted and/or alarmed;

It should be noted that at least one statement cluster is obtained by clustering the target query statement above, and the target query statement is a slow query statement, that is, the target query statement is a statement that meets the specified conditions. Therefore, if the statement to be analyzed belongs to At least any one of the statement clusters, the query statement to be executed corresponding to the statement to be analyzed is a slow query statement, that is, the query statement to be executed is a statement that meets the specified conditions, and the query statement to be executed can be directly intercepted and/or or call the police. It can be understood that the so-called interception means that the query statement to be executed is intercepted outside the database system, so that the database system cannot receive the query statement to be executed; and, in practical applications, the alarm method may include: sending an alarm notification, Alarm signal, alarm sound, or any kind of alarm notification that can remind the corresponding processing personnel. Exemplarily, the method of detecting whether the statement to be analyzed belongs to any one of the at least one statement cluster may be: detecting whether the string of the statement to be analyzed is the same as the string of any one of the at least one statement cluster, or, detecting whether the statement to be analyzed is the same. Check whether the obtained result is the same as the hash result of any one of the at least one sentence cluster, and if it is the same, it is detected that the sentence to be analyzed belongs to any one of the at least one sentence cluster.

In addition, in order to avoid the loss of the target business system that may be caused by intercepting and/or alarming any slow query statement, it is possible to re-identify the statement to be analyzed, and then decide whether to intercept and/or alarm, thereby Implement flexible filtering of slow query statements to prevent accidental damage to the production environment.

Based on this processing idea, exemplarily, in an implementation manner, if it is detected that the statement to be analyzed belongs to any one of at least one statement cluster, the query statement to be executed is intercepted and/or alarmed, which may include:

If it is detected that the to-be-analyzed statement belongs to any one of the at least one statement cluster, and the specified evaluation index corresponding to any one of the statement clusters meets a predetermined threshold condition, intercept and/or intercept the to-be-executed query statement Call the police.

Exemplarily, the specified evaluation index may be execution time, locking time (time for slow query statement locking), EXPLAIN result, and the like. These results can be pre-recorded, or statistically or calculated, which is reasonable. For example, the specified evaluation index for slow queries is execution time. At this time, if the execution time corresponding to any one of the statement clusters exceeds a preset time-length threshold, the query statement to be executed can be intercepted and/or alarmed. It can be understood that the execution duration corresponding to any one of the statement clusters may be obtained by averaging the execution durations of the target query statements included in the any one of the statement clusters. In addition, it can be understood that, if the statement to be analyzed does not belong to any one of the at least one statement cluster, or, the statement to be analyzed belongs to any one of the at least one statement cluster but the specified judgment on slow query corresponding to any one statement cluster If the indicator does not meet the predetermined threshold condition, the query statement to be executed corresponding to the to-be-analyzed statement can be delivered, that is, the to-be-analyzed statement is delivered to the database system, so that the database system can respond to the to-be-analyzed statement.

The solution provided by this embodiment can not only efficiently process slow query statements while taking into account that user information is not leaked, but also filter the slow query statements newly generated by the business system, so as to achieve the purpose of real-time management and control of slow query statements . In addition, set predetermined threshold conditions to reasonably filter slow query statements to prevent accidental damage to the production environment.

Optionally, based on the above-mentioned embodiment, in the data processing method provided by another embodiment of the present disclosure, after outputting the corresponding processing task for each sentence cluster in the at least one sentence cluster, you can also include:

When a designated regulation instruction for any processing task is received, the processing task responds to the designated regulation instruction.

Wherein, the specified regulation instruction for any processing task is an instruction for regulating the processing personnel, status information, etc. of the processing task. Exemplarily, the specified regulation instruction may be: an instruction for characterizing the handover of a task to another person, an instruction for marking a task as an exempt task, an instruction for setting the status of a task to a resolved state, an instruction for setting a task Instructions for remarking information, etc. Among them, marked as exemption: After marking the exemption, such signatures will no longer be notified to the police and need to be reviewed by the processing personnel.

It can be understood that, in the output interface of the processing task, a trigger button corresponding to each specified regulation instruction may be set. In this way, by clicking the trigger button, the processing personnel table can issue a corresponding specified regulation instruction, so that the data processing device can respond to the specified regulation instruction for the processing task after receiving the specified regulation instruction.

In order to facilitate the understanding of the solution, FIG. 3 shows a schematic diagram of a processing interface for a processing task. As shown in FIG. 3 , for a processing task, the output interface can display the task content of the processing task and other information, including the current owner, status, type, and trigger buttons corresponding to each specified control instruction. Among them, if the processing task is a task determined based on log data generated in a production environment, the type is an online type; if the processing task is a task determined based on log data generated in other operating environments, the type Offline type.

It can be seen that the processing task is flexibly processed by the processing personnel through the response to the specified regulation instruction for any processing task.

The principle content of a slow query statement processing method provided by the present disclosure will be described in detail below with reference to another specific embodiment.

The system architecture diagram shown in Figure 4, Figure 4 provides the following five parts:

Simple database architecture 410: The database system includes master-slave nodes (master nodes and slave nodes), master-slave nodes, and proxy instances; wherein the proxy instances pass external write traffic to the master node, so that the master node completes data writing Process; pass the external read traffic to each slave node slave-01 to slave-n, so that the slave node completes the data reading process. Wherein, in the process of accessing the database by the business system, a log file (which contains the above at least one query statement) may be generated.

Slow query statement collection and processing 420: Real-time log collection can be performed from log files generated in the simple database architecture, and then slow query statement identification is performed on the collected logs (corresponding to the above-mentioned from at least one query statement, it is determined that the specified conditions are met. at least one target query statement), then the signature information is generated (corresponding to the above-mentioned deprivation of the target query statement, followed by a hash operation to obtain the signature corresponding to the target query statement), and finally related information is stored. This part can store the execution time of the query statement, the number of parsing lines: the number of lines of code for parsing the query statement, the number of returned lines: the number of lines of data returned for the query statement, the source IP: the IP of the source of the query statement, the request time: Information such as the request time of the query statement.

Slow query statement management 430: Receive the data pushed by the slow query statement collection and processing part, aggregate the slow query statements of the same type (corresponding to the above-mentioned clustering of at least one to-be-used statement to obtain at least one statement cluster), and store the aggregated information: i.e. Each sentence cluster obtained after the aggregation is stored, and then each sentence cluster is processed according to the result obtained by the aggregation (corresponding to the above for each sentence cluster in the at least one sentence cluster, the corresponding processing task is output). The so-called processing can include: notification and alarm: alarm the corresponding slow query statement and notify the corresponding handler; claim function: the handler claims the processing task that belongs to his own processing; handover function: the processing task that does not belong to his own processing Hand over to the corresponding handler; assign function: assign the corresponding processing task to the corresponding handler; set schedule: set the deadline for the processing task; set remarks: set the remarks of the processing task, for example: the special requirements of a processing task : Must be processed on the day and so on.

Slow query statement identification and interception 440: when any service request arrives (corresponding to the above-mentioned query statement to be determined to be executed), perform deprivation processing on the user information in the query statement in the service request, and obtain the statement to be analyzed (corresponding to the query statement to be executed). The above query statement to be executed is deprived to obtain the statement to be analyzed); the data of the management part of the slow query statement is requested, and then according to the data of the management part of the slow query statement, whether the statement to be analyzed is a slow query statement (corresponding to the above if If it is detected that the statement to be analyzed belongs to any one of at least one statement cluster, the query statement to be executed corresponding to the statement to be analyzed is a slow query statement). If it is not a slow query statement, it will be sent to the database system normally; if it is a slow query statement , and judge again whether the threshold is reached (corresponding to the above if it is detected that the statement to be analyzed belongs to any one of the at least one statement cluster, and the specified evaluation index on the slow query corresponding to any one statement cluster meets the predetermined threshold condition) if If the threshold is reached, the slow query statement will be intercepted and an alarm will be issued. If it is not reached, the slow query statement will be sent to the database normally.

Slow query statement interception 450: The first interception in the development environment; the second interception in the test environment; the third interception in the heterogeneous environment; the fourth interception in the production environment; Statements are intercepted before the production environment, and business code is optimized in time. Multi-environment interception can be reasonably allocated according to business needs. For example, only the development environment and test environment can be applied; or the development environment, test environment, heterogeneous environment, and production environment can be applied, but the proportion of development environment and test environment larger.

In this embodiment, it is possible to dynamically monitor the log data generated by the business, with stronger real-time performance, and then perform deprivation processing on the slow query statements, which can not only cluster the slow query statements belonging to the same category, but also ensure that The privacy of user data avoids potential security risks caused by user data leakage; it can efficiently process each type of slow query statement in any environment to avoid the situation that no one cares about online; and, set a threshold , more flexibly filter slow query statements reasonably to prevent accidental injury to the production environment; the multi-channel interception strategy of multiple environments can intercept slow query statements before the production environment, and optimize business code in time to avoid business impact; In addition, for complex master-slave architectures or distributed architectures, the improved slow query processing efficiency is particularly evident.

According to an embodiment of the present disclosure, the present disclosure also provides a data processing apparatus, as shown in FIG. 5 , the apparatus includes:

The first determination module 510 is configured to determine, from at least one query statement, at least one target query statement that meets a specified condition; wherein, the specified condition includes that the execution duration exceeds a predetermined duration threshold;

The first processing module 520 is configured to perform deprivation processing on the at least one target query statement respectively to obtain at least one to-be-utilized statement;

Clustering module 530, configured to perform clustering on the at least one sentence to be used according to the text content of the at least one sentence to be used to obtain at least one sentence cluster;

The output module 540 is configured to output a corresponding processing task for each sentence cluster in the at least one sentence cluster.

In this solution, after at least one target query statement in at least one query statement that meets the specified conditions is determined, the at least one target query statement is deprived of privacy processing, so that information is prevented from being leaked; and, considering the deprivation of privacy The at least one sentence to be used after the transformation process contains a sentence with common content. Therefore, the at least one sentence to be used is clustered, and the processing tasks are respectively output according to each sentence cluster in the at least one sentence cluster obtained by the clustering, In this way, the number of processing tasks is greatly reduced compared to setting processing tasks for each target query statement. It can be seen that through this solution, slow query statements can be efficiently processed while taking into account that user information is not leaked.

Optionally, the output module is specifically used for:

For each sentence cluster, a processing object corresponding to the sentence cluster is determined, and a corresponding processing task is output; wherein, the processing object is a to-be-used sentence in the sentence cluster.

Optionally, the determining module is specifically used for:

From at least one query statement generated under the specified operating environment, determine at least one target query statement that meets the specified condition;

The output module is specifically used for:

For each statement cluster in the at least one statement cluster, the processing task corresponding to the statement cluster is output to the processing terminal corresponding to the specified operating environment.

Optionally, the clustering module is specifically used for:

Hash operation is performed on each to-be-utilized statement in the at least one to-be-utilized statement to obtain a signature corresponding to the to-be-utilized statement;

At least one sentence cluster is obtained according to the signature corresponding to the to-be-utilized sentence.

Optionally, as shown in Figure 6, the device further includes:

a second determining module 650, configured to determine the query statement to be executed;

The second processing module 660 is configured to perform deprivation processing on the query statement to be executed to obtain the statement to be analyzed;

The interception module 670 is configured to intercept and/or alarm the query statement to be executed if it is detected that the statement to be analyzed belongs to any one of the at least one statement cluster.

Optionally, the interception module is specifically used for:

If it is detected that the to-be-analyzed statement belongs to any one of the at least one statement cluster, and the specified evaluation index corresponding to any one of the statement clusters meets a predetermined threshold condition, intercept and/or intercept the to-be-executed query statement Call the police.

Optionally, the device further includes:

The response module is configured to respond to the specified control instruction for any processing task when a specified control instruction is received for the processing task.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

An embodiment of the present disclosure provides an electronic device, including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform any one of the data processing methods.

Embodiments of the present disclosure provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute any data processing method.

The present disclosure provides a computer program product, including a computer program that, when executed by a processor, implements any data processing method.

FIG. 7 shows a schematic block diagram of an example electronic device 700 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 7 , the device 700 includes a computing unit 701 that can be executed according to a computer program stored in a read only memory (ROM) 702 or loaded into a random access memory (RAM) 703 from a storage unit 708 Various appropriate actions and handling. In the RAM 703, various programs and data necessary for the operation of the device 700 can also be stored. The computing unit 701 , the ROM 702 , and the RAM 703 are connected to each other through a bus 704 . An input/output (I/O) interface 705 is also connected to bus 704 .

Various components in the device 700 are connected to the I/O interface 705, including: an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc. ; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 709 allows the device 700 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

Computing unit 701 may be various general-purpose and/or special-purpose processing components with processing and computing capabilities. Some examples of computing units 701 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 executes the various methods and processes described above, such as data processing methods. For example, in some embodiments, a data processing method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 708 . In some embodiments, part or all of the computer program may be loaded and/or installed on device 700 via ROM 702 and/or communication unit 709 . When a computer program is loaded into RAM 703 and executed by computing unit 701, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the data processing method by any other suitable means (eg, by means of firmware).

Various implementations of the systems and techniques described herein above may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips system (SOC), complex programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that The processor, which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device an output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, performs the functions/functions specified in the flowcharts and/or block diagrams. Action is implemented. The program code may execute entirely on the machine, partly on the machine, partly on the machine and partly on a remote machine as a stand-alone software package or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user's computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a distributed system server, or a server combined with blockchain.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, no limitation is imposed herein.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure should be included within the protection scope of the present disclosure.

Claims (17)

1. A data processing method, comprising: From at least one query statement, determine at least one target query statement that meets a specified condition; wherein, the specified condition includes that the execution duration exceeds a predetermined duration threshold; Perform deprivation processing on the at least one target query statement respectively to obtain at least one statement to be utilized; Clustering the at least one sentence to be used according to the text content of the at least one sentence to be used to obtain at least one sentence cluster; For each sentence cluster in the at least one sentence cluster, a corresponding processing task is output. 2. The method according to claim 1, wherein, for each statement cluster in the at least one statement cluster, outputting a corresponding processing task, comprising: For each statement cluster, determine the processing object corresponding to the statement cluster, and output the corresponding processing task; Wherein, the processing object is a to-be-used statement in the statement cluster. 3. The method according to claim 1, wherein the determining from at least one query statement at least one target query statement that meets a specified condition comprises: From at least one query statement generated under the specified operating environment, determine at least one target query statement that meets the specified condition; For each statement cluster in the at least one statement cluster, output the corresponding processing task, including: For each statement cluster in the at least one statement cluster, the processing task corresponding to the statement cluster is output to the processing terminal corresponding to the specified operating environment. 4. The method according to claim 1, wherein the at least one sentence to be used is clustered according to the text content of the at least one sentence to be used to obtain at least one sentence cluster, comprising: Hash operation is performed on each to-be-utilized statement in the at least one to-be-utilized statement to obtain a signature corresponding to the to-be-utilized statement; At least one sentence cluster is obtained according to the signature corresponding to the to-be-utilized sentence. 5. The method according to any one of claims 1-4, after the at least one sentence to be used is clustered according to the text content of the at least one sentence to be used to obtain at least one sentence cluster, the Methods also include: Determine the query statement to be executed; Perform deprivation processing on the query statement to be executed to obtain the statement to be analyzed; If it is detected that the statement to be analyzed belongs to any one of the at least one statement cluster, intercept and/or alarm the query statement to be executed. 6. The method according to claim 5, wherein, if it is detected that the statement to be analyzed belongs to any one of the at least one statement cluster, the query statement to be executed is intercepted and/or alarmed, including: If it is detected that the to-be-analyzed statement belongs to any one of the at least one statement cluster, and the specified evaluation index corresponding to any one of the statement clusters meets a predetermined threshold condition, intercept and/or intercept the to-be-executed query statement Call the police. 7. The method according to claim 1, wherein, after outputting the corresponding processing task for each statement cluster in the at least one statement cluster, the method further comprises: When a designated regulation instruction for any processing task is received, the processing task responds to the designated regulation instruction. 8. A data processing device, comprising: a first determination module, configured to determine, from at least one query statement, at least one target query statement that meets a specified condition; wherein the specified condition includes that the execution duration exceeds a predetermined duration threshold; a first processing module, configured to perform deprivation processing on the at least one target query statement to obtain at least one to-be-utilized statement; a clustering module, configured to perform clustering on the at least one sentence to be used according to the text content of the at least one sentence to be used to obtain at least one sentence cluster; The output module is configured to output a corresponding processing task for each sentence cluster in the at least one sentence cluster. 9. The apparatus according to claim 8, wherein the output module is specifically used for: For each sentence cluster, a processing object corresponding to the sentence cluster is determined, and a corresponding processing task is output; wherein, the processing object is a to-be-used sentence in the sentence cluster. 10. The apparatus according to claim 8, wherein the first determining module is specifically configured to: From at least one query statement generated under the specified operating environment, determine at least one target query statement that meets the specified condition; The output module is specifically used for: For each statement cluster in the at least one statement cluster, the processing task corresponding to the statement cluster is output to the processing terminal corresponding to the specified operating environment. 11. The apparatus according to claim 8, wherein the clustering module is specifically used for: Hash operation is performed on each to-be-utilized statement in the at least one to-be-utilized statement to obtain a signature corresponding to the to-be-utilized statement; At least one sentence cluster is obtained according to the signature corresponding to the to-be-utilized sentence. 12. The apparatus of any one of claims 8-11, further comprising: The second determination module determines the query statement to be executed; a second processing module, configured to perform deprivation processing on the query statement to be executed to obtain the statement to be analyzed; An interception module, configured to intercept and/or alarm the query statement to be executed if it is detected that the statement to be analyzed belongs to any one of the at least one statement cluster. 13. The apparatus according to claim 12, wherein the interception module is specifically configured to: If it is detected that the to-be-analyzed statement belongs to any one of the at least one statement cluster, and the specified evaluation index corresponding to any one of the statement clusters meets a predetermined threshold condition, intercept and/or intercept the to-be-executed query statement Call the police. 14. The apparatus of claim 8, wherein the apparatus further comprises: The response module is configured to respond to the specified control instruction for any processing task when a specified control instruction is received for the processing task. 15. An electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the execution of any of claims 1-7 Methods. 16. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any of claims 1-7. 17. A computer program product comprising a computer program which, when executed by a processor, implements the method of any of claims 1-7.

Images