CN105630789B - A kind of inquiry plan method for transformation and device - Google Patents

Abstract

The embodiment of the present invention discloses a query plan conversion method and device, which relate to the field of computers and can reduce the number of physical query tasks constituting a physical query plan to a greater extent. The specific solution is: extract the first query operator and the second query operator from the logical query plan; if the partition attribute of the first query operator is the prefix of the partition attribute of the second query operator, rewrite the first query operator in the logical query plan. The partition attribute of the second query operator, so that the partition attribute of the second query operator is the same as the partition attribute of the first query operator; the partition operator of the second query operator is deleted from the logical query plan, and the second query operator The operator, the first query operator and the partition operator of the first query operator generate a physical query task to form a physical query plan, and the present invention is used in the process of converting a logical query plan into a physical query plan.

Description

A query plan conversion method and device

technical field

The present invention relates to the field of computers, in particular to a query plan conversion method and device.

Background technique

A logical query plan is a tree-like query structure that is parsed from a query statement and takes the query operators in the query statement as nodes. Each query operator in the logical query plan has a partition attribute and a partition operator and partition attribute. Is an attribute value common to all data that the query operator needs to operate on. The partition operator is located between this query operator and another query operator to separate two query operators. Logical query plans can be transformed into physical query plans.

The specific process of converting a logical query plan into a physical query plan is as follows: starting with a partition operator as an identifier to generate a physical query task, and generating a physical query task from the query operator between the partition operator and the next partition operator , the next partition operator is used as the identifier for generating the next physical query task, and the next physical query task is generated. The physical query plan is a tree-like query structure with the generated multiple physical query tasks as nodes.

When the number of physical query tasks required to form a physical query plan decreases, the read data operations and write data operations required to execute the physical query tasks will also decrease, and the time overhead introduced by the read data operations and write data operations will also be reduced. will be smaller, so the time taken to execute the physical query plan will be smaller.

In the prior art, in order to achieve the purpose of reducing the number of physical query tasks required to form a physical query plan, two query operators that are separated by one partition operator and have identical partition attributes in the logical query plan are searched, and the above two query operators are deleted. The partition operator between the two, so that in the process of converting the logical query plan into the physical query plan, the above two query operators are changed from two different physical query tasks to one physical query task, so as to Reduce the number of physical query tasks.

However, this technology requires two query operators to have exactly the same partition properties and a direct predecessor-successor relationship, which limits the application scenarios of this technology.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a query plan conversion method and device, which can reduce the number of physical query tasks constituting a physical query plan to a greater extent.

To achieve the above object, the embodiments of the present invention adopt the following technical solutions:

A first aspect of the embodiments of the present invention provides a query plan conversion method, including:

extracting a first query operator and a second query operator from the logical query plan, where the first query operator is a precursor operator of the second query operator;

If the partition attribute of the first query operator is a prefix of the partition attribute of the second query operator, rewrite the partition attribute of the second query operator in the logical query plan, so that the second query operator The partition attribute of the query operator is the same as the partition attribute of the first query operator;

The partition operator of the second query operator is deleted from the logical query plan, and is generated according to the second query operator, the first query operator, and the partition operator of the first query operator A physical query task to form the physical query plan.

With reference to the first aspect, in a first possible implementation manner, if the partition attribute of the first query operator is a prefix of the partition attribute of the second query operator, rewrite the partition attribute in the logical query plan The partition attribute of the second query operator, so that the partition attribute of the second query operator is the same as the partition attribute of the first query operator, including:

If the partition attribute of the first query operator is a prefix of the partition attribute of the second query operator, and a third query operator is spaced between the first query operator and the second query operator , the third query operator can be implemented using a broadcast query algorithm, then the third query operator is rewritten as a broadcast query operator, and the broadcast query operator does not have a partition operator;

rewriting the partition attribute of the second query operator in the logical query plan, so that the partition attribute of the second query operator is the same as the partition attribute of the first query operator;

The partition operator of the second query operator is deleted from the logical query plan, and is generated according to the second query operator, the first query operator, and the partition operator of the first query operator A physical query task to form the physical query plan, including:

The partition operator of the second query operator is removed from the logical query plan, and the second query operator, the broadcast query operator, the first query operator, and the first query The partition operator of the operator generates a physical query task to form the physical query plan;

The first query operator is a predecessor operator of the third query operator, and the third query operator is a predecessor operator of the second query operator.

With reference to the first aspect, in a second possible implementation manner, if the partition attribute of the first query operator is the same as the partition attribute of the second query operator, and the first query operator is the same as the A third query operator is spaced between the second query operators, and the third query operator can be implemented by using a broadcast query algorithm, then the third query operator is rewritten as a broadcast query operator, and the broadcast query operation operator has no partition operator;

The partition operator of the second query operator is removed from the logical query plan, and the second query operator, the broadcast query operator, the first query operator, and the first query The partition operator of the operator generates a physical query task to form the physical query plan;

The first query operator is a predecessor operator of the third query operator, and the third query operator is a predecessor operator of the second query operator.

With reference to the first aspect, the first possible implementation manner, and the second possible implementation manner, in a third possible implementation manner, in the deletion of the partition of the second query operator from the logical query plan before the operator, the method further includes:

rewriting the sorting attribute of the first query operator so that the sorting attribute of the first query operator is the same as the sorting attribute of the second query operator;

Wherein, the sorting attribute of the first query operator before rewriting is the same as the partition attribute of the first query operator, and the sorting attribute of the second query operator is the same as the partitioning attribute of the second query operator before the rewriting same;

The sorting attribute is used to perform partition sorting on the data in the data table operated by the query operator of the logical query plan.

With reference to the first aspect, the first possible implementation manner, and the second possible implementation manner, in a fourth possible implementation manner, the The partition operator of the first query operator generates a physical query task to form the physical query plan, including:

Using task flow associative optimization JFC technology, a physical query task is generated according to the second query operator, the first query operator and the partition operator of the first query operator to form the physical query plan .

The second aspect of the embodiment of the present invention also provides a query plan conversion device, including:

an extraction unit, configured to extract a first query operator and a second query operator from the logical query plan, where the first query operator is a precursor operator of the second query operator;

a first rewriting unit, configured to rewrite the logical query if the partition attribute of the first query operator extracted by the extracting unit is a prefix of the partition attribute of the second query operator extracted by the extracting unit the partition attribute of the second query operator in the plan, so that the partition attribute of the second query operator is the same as the partition attribute of the first query operator;

a deletion unit, configured to delete the partition operator of the second query operator from the logical query plan after the first rewriting unit rewrites the partition attribute of the second query operator in the logical query plan ;

a generating unit, configured to, after the deleting unit deletes the partition operator of the second query operator, generate the partition according to the second query operator, the first query operator and the partition of the first query operator The operator generates a physical query task to form the physical query plan.

With reference to the second aspect, in a first possible implementation manner, the first rewriting unit includes:

The first rewriting module is used for if the partition attribute of the first query operator is a prefix of the partition attribute of the second query operator, and the relationship between the first query operator and the second query operator is There is a third query operator in the interval, and the third query operator can be implemented using a broadcast query algorithm, then the third query operator is rewritten as a broadcast query operator, and the broadcast query operator does not have a partition operator;

a second rewriting module, configured to rewrite the partition attribute of the second query operator in the logical query plan, so that the partition attribute of the second query operator is the same as the partition attribute of the first query operator;

The generating unit is further configured to generate a physical query task according to the second query operator, the broadcast query operator, the first query operator and the partition operator of the first query operator, to constitute the physical query plan;

The first query operator is a predecessor operator of the third query operator, and the third query operator is a predecessor operator of the second query operator.

With reference to the second aspect, in a second possible implementation manner, a second rewriting unit is configured to, if the partition attribute of the first query operator extracted by the extracting unit is the same as the partition attribute of the second query operator extracted by the extracting unit The partition attributes of the query operators are the same, and a third query operator is spaced between the first query operator and the second query operator, and the third query operator can be implemented using a broadcast query algorithm, then the The third query operator is rewritten as a broadcast query operator, and the broadcast query operator has no partition operator;

The deleting unit is further configured to delete the partition of the second query operator from the logical query plan after the second rewriting unit rewrites the third query operator as the broadcast query operator operator;

The generating unit is further configured to, after the deleting unit deletes the partition operator of the second query operator, according to the second query operator, the broadcast query operator, the first query operator and the partition operator of the first query operator generates a physical query task to form the physical query plan;

The first query operator is a predecessor operator of the third query operator, and the third query operator is a predecessor operator of the second query operator.

In combination with the second aspect, the first possible implementation manner, and the second possible implementation manner, in a third possible implementation manner, a third rewriting unit is configured to extract data from the logical query plan in the deletion unit Before deleting the partition operator of the second query operator, rewrite the sorting attribute of the first query operator so that the sorting attribute of the first query operator is the same as the sorting attribute of the second query operator ;

Wherein, the sorting attribute of the first query operator before rewriting is the same as the partition attribute of the first query operator, and the sorting attribute of the second query operator is the same as the partitioning attribute of the second query operator before the rewriting same;

The sorting attribute is used to perform partition sorting on the data in the data table operated by the query operator of the logical query plan.

In combination with the second aspect, the first possible implementation manner, and the second possible implementation manner, in the fourth possible implementation manner, the generating unit is specifically configured to optimize the JFC technology by using the task flow correlation, according to the The second query operator, the first query operator and the partition operator of the first query operator generate a physical query task to form the physical query plan.

In the query plan conversion method and device provided by the embodiments of the present invention, as long as the partition attribute of the first query operator is the prefix of the partition attribute of the second query operator, the partition attribute of the second query operator in the logical query plan can be rewritten by rewriting the partition attribute of the second query operator. , so that when the partition attribute of the second query operator is the same as that of the first query operator, the partition operator of the second query operator can be deleted to generate A physical query task, reducing the number of physical query tasks that make up the physical query plan.

Compared with the prior art, the partition operator of the second query operator can be deleted only when the partition attribute of the first query operator is exactly the same as that of the second query operator. When the partition attribute of the second query operator is the prefix of the partition attribute of the second query operator, rewriting the partition attribute of the second query operator can obtain more query operator pairs that satisfy the predecessor-successor relationship (such as the first query operator and the second query operator), which can further reduce the number of physical query tasks that constitute the physical query plan.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

1 is a schematic flowchart of a query plan conversion method in an embodiment of the present invention;

2 is a schematic diagram of an example of a logical query plan in an embodiment of the present invention;

3 is a schematic flowchart of another query plan conversion method in an embodiment of the present invention;

4 is a schematic diagram of an example of another logical query plan in an embodiment of the present invention;

5 is a schematic diagram of an example of another logical query plan in an embodiment of the present invention;

6 is a schematic diagram of an example of another logical query plan in an embodiment of the present invention;

7 is a schematic flowchart of another query plan conversion method in an embodiment of the present invention;

8 is a schematic structural diagram of a query plan conversion device in an embodiment of the present invention;

9 is a schematic structural diagram of another query plan conversion device in an embodiment of the present invention;

10 is a schematic structural diagram of another query plan conversion device in an embodiment of the present invention;

FIG. 11 is a schematic structural diagram of another query plan conversion apparatus in an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Additionally, the terms "system" and "network" are often used interchangeably herein. The term "and/or" in this article is only an association relationship to describe the associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, A and B exist at the same time, and A and B exist independently B these three cases. In addition, the character "/" in this document generally indicates that the related objects are an "or" relationship.

Example 1

An embodiment of the present invention provides a query plan conversion method, which can be applied to the process of converting a logical query plan into a physical query plan. As shown in FIG. 1 , the query plan conversion method includes:

S101. The query plan conversion apparatus extracts a first query operator and a second query operator from the logical query plan, where the first query operator is a precursor operator of the second query operator.

The method for the query plan transformation device to extract the first query operator and the second query operator from the logical query plan may include: the query plan transformation device searches the logical query plan for two query operators that have a predecessor-successor relationship, and respectively determine the partition attributes of the two query operators with the predecessor-successor relationship; when the partition attribute of the predecessor query operator in the two query operators with the predecessor-successor relationship is the prefix of the partition attribute of the successor query operator, Or when the partition attribute of the predecessor query operator is the same as the partition attribute of the successor query operator, the two query operators with a predecessor-successor relationship can be determined as the first query operator and the second query operator. Specifically, among the two query operators that have a predecessor-successor relationship, the predecessor query operator is used as the first query operator, and the successor query operator is used as the second query operator.

It should be noted that, in this embodiment of the present invention, the existence of a predecessor-successor relationship between two query operators in the logical query plan may include: the two query operators have a direct predecessor-successor relationship.

Specifically, when there is a direct dependency between two query operators in the logical query plan, that is, the data required to be operated by one of the two query operators is the result of the data operated by the other query operator , it can be considered that there is a direct predecessor-successor relationship between these two query operators. For example, assuming that the two query operators are query operator 1 and query operator 2, when the data required to be operated by query operator 2 is the result of the operation data of query operator 1, it can be considered that query operator 1 and query operator Operator 2 has a direct predecessor-successor relationship, and query operator 1 is the direct predecessor operator of query operator 2, and query operator 2 is the direct successor operator of query operator 1.

Further, the existence of a predecessor-successor relationship between the two query operators in the logical query plan may further include: the two query operators have an indirect predecessor-successor relationship.

Specifically, when there is an indirect dependency between two query operators in the logical query plan, that is, when there is at least one other query operator between the two query operators, if one of the two query operators queries When the data to be operated by the operator is the result of the data operated by the other query operator, and the data operated by the other query operator is the result of the data operated by the other query operator among the two query operators, then These two query operators can be considered to have an indirect predecessor-successor relationship. For example, suppose the two query operators are query operator 3 and query operator 4. There is query operator 5 between query operator 3 and query operator 4. When query operator 4 needs to operate the data, it is a query operation. When the result of the operation of data operator 5, and the data required to operate by the query operator 5 is the result of the operation data of the query operator 3, it can be considered that the query operator 3 and the query operator 4 have an indirect predecessor-successor relationship, and the query operation Operator 3 is the indirect predecessor operator of query operator 4, and query operator 4 is the indirect successor operator of query operator 3.

Correspondingly, when there is a direct dependency between the first query operator and the second query operator, the first query operator is the direct predecessor operator of the second query operator; There is an indirect dependency between query operators, that is, when there is a third query operator between the first query operator and the second query operator, the first query operator is the indirect predecessor operator of the second query operator .

Exemplarily, in the logical query plan shown in FIG. 2 , the data to be operated by query operator 2 is the result of the data operated by query operator 1, so it can be determined that there is a direct relationship between query operator 1 and query operator 2. Predecessor-successor relationship, query operator 1 is the direct predecessor operator of query operator 2 (the partition attribute "partition attribute 1" of query operator 1 is the same as the partition attribute "partition attribute 1" of query operator 2, then it can be determined that Query operator 1 is used as the first query operator, and query operator 2 is used as the second query operator); the data to be operated by query operator 5 is the result of the operation data of query operator 4, then it can be determined that query operator 5 and There is a direct predecessor-successor relationship between query operator 4, and query operator 4 is the direct predecessor operator of query operator 5 (the partition attribute "partition attribute 3" of query operator 4 is the partition attribute "partition attribute" of query operator 5. attribute 3, partition attribute 5” prefix, query operator 4 can be determined as the first query operator, and query operator 5 as the second query operator).

Exemplarily, in the logical query plan shown in FIG. 2, the data required to operate by query operator 5 is the result of the operation data of query operator 4, and the data required to operate by query operator 4 is the operation data of query operator 3. , it can be determined that there is an indirect predecessor-successor relationship between query operator 3 and query operator 5, and query operator 3 is the indirect predecessor operator of query operator 5 (the partition attribute of query operator 3 "Partition attribute 3 " is the prefix of the partition attribute "partition attribute 3, partition attribute 5" of query operator 5, then query operator 3 can be determined as the first query operator, query operator 5 as the second query operator, and query operator 4 as the third query operator).

It should be noted that the query plan conversion method provided in the embodiment of the present invention can be applied to a device with a data query function (such as a communication system or a background server/storage device of a data processing system), and the query plan conversion method in the embodiment of the present invention The device may be a processor of the above-mentioned device with a data query function, such as a central processing unit (Central Processing Unit, CPU), or the query plan conversion device may also be a functional module in the processor of the above-mentioned device with a data query function, The embodiment of the present invention does not limit the specific form of the query plan conversion device.

S102. If the partition attribute of the first query operator is a prefix of the partition attribute of the second query operator, the query plan conversion device rewrites the partition attribute of the second query operator in the logical query plan, so that the second query operator's partition attribute The partition attribute is the same as that of the first query operator.

Exemplarily, assuming that the partition attribute of the first query operator is: "partition attribute 1", and the partition attribute of the second query operator is: "partition attribute 1, partition attribute 2", then the first query operator's partition attribute is obtained. The partition attribute "partition attribute 1" is the prefix of the partition attribute "partition attribute 1, partition attribute 2" of the second query operator.

Wherein, the partition attribute of the first query operator is the prefix of the partition attribute of the second query operator, which can indicate that partitioning according to the partition attribute of the first query operator is to put the operation required by the first query operator in the data table. All rows containing partition attribute 1 are divided into one area; partitioning according to the partition attribute of the second query operator is to divide all rows containing partition attribute 1 and partition attribute 2 in the data table to be operated by the second query operator It can be seen from this that when the extracted partition attribute of the first query operator is the prefix of the extracted partition attribute of the second query operator, the region divided by the partition attribute of the first query operator The data in contains data in regions partitioned by the partition attribute of the second query operator.

It should be noted that rewriting the partition attribute of the second query operator in the logical query plan means rewriting the partition attribute of the second query operator to the partition attribute of the first query operator, and keeping the partition attribute of the first query operator unchanged. , to ensure that the partition attribute of the rewritten second query operator is the same as that of the first query operator. In this way, the partition attribute of the first query operator is the same as the partition attribute of the second query operator, which satisfies the same condition as the partition attribute of the query operator pair required by the prior art, so that the first query operator Generate a physical query task with the second query operator.

S103, the query plan conversion device deletes the partition operator of the second query operator from the logical query plan, and generates a physical query task according to the second query operator, the first query operator, and the partition operator of the first query operator , to form the physical query plan.

Wherein, the partition attribute of the rewritten second query operator is the same as the partition attribute of the first query operator. At this time, the query operator pair for which the first query operator and the second query operator meet the requirements of the prior art have complete The same partition attribute conditions, after deleting the partition operator of the second query operator, because there is a predecessor-successor relationship between the first query operator and the second query operator, and the first query operator and the second query operator There is no partition operator between operators, and the partition operator of the first query operator is used as an identifier for generating a new physical query task. The first query operator and the second query operator can generate a physical query task. The query plan conversion device constructs a physical query plan for the nodes by using the generated multiple physical query tasks.

In the query plan conversion method provided by the embodiment of the present invention, through this solution, as long as the partition attribute of the first query operator is the prefix of the partition attribute of the second query operator, the second query operator in the logical query plan can be rewritten by rewriting the partition attribute of the second query operator. Partition attribute, so that when the partition attribute of the second query operator is the same as the partition attribute of the first query operator, the partition operator of the second query operator can be deleted, so as to operate according to the first query operator and the second query operator The operator generates one physical query task, reducing the number of physical query tasks that make up the physical query plan.

Compared with the prior art, the partition operator of the second query operator can be deleted only when the partition attribute of the first query operator is exactly the same as that of the second query operator. When the partition attribute of the second query operator is the prefix of the partition attribute of the second query operator, rewriting the partition attribute of the second query operator can obtain more query operator pairs that satisfy the predecessor-successor relationship (such as the first query operator and the second query operator), which can further reduce the number of physical query tasks that constitute the physical query plan.

Example 2

The embodiment of the present invention provides a query plan conversion method, which can be applied to the process of converting a logical query plan to a physical query plan. The query plan conversion method includes:

S201. The query plan conversion apparatus extracts a first query operator and a second query operator from the logical query plan, where the first query operator is a precursor operator of the second query operator.

The first query operator is the predecessor operator of the second query operator may specifically include: the first query operator is the direct predecessor operator of the second query operator, the interval between the first query operator and the second query operator There is a third query operator.

In the first application scenario of the embodiment of the present invention, as shown in FIG. 3 , after the query plan transformation device extracts the first query operator and the second query operator, the query plan transformation device may first determine the first query operation Whether there is a third query operator between the operator and the second query operator, that is, execute S202:

S202. The query plan conversion apparatus determines whether there is a third query operator spaced between the first query operator and the second query operator.

Wherein, a third query operator is spaced between the first query operator and the second query operator, which means that the first query operator is an indirect precursor operator of the second query operator.

Specifically, in the first application scenario of the embodiment of the present invention, if the first query operator is a direct predecessor operator of the second query operator, that is, there is no gap between the first query operator and the second query operator If the first query operator is an indirect precursor operator of the second query operator, that is, there is a third interval between the first query operator and the second query operator The query operator continues to execute S203 and subsequent processes.

S203. The query plan conversion device determines whether the third query operator can be implemented using a broadcast query algorithm.

Specifically, if the third query operator can be implemented using the broadcast query algorithm, the query plan conversion device continues to execute S204; if the third query operator cannot be implemented using the broadcast query algorithm, the query plan transformation device cannot convert the third query operation. The operator is rewritten as a broadcast query operator, so that the partition operator of the second query operator cannot be deleted to merge the first query operator and the second query operator.

S204, the query plan conversion apparatus rewrites the third query operator as a broadcast query operator, and the broadcast query operator does not have a partition operator.

Wherein, when there is a third query operator spaced between the first query operator and the second query operator, it means that if the query plan conversion device is in the process of rewriting the logical query plan into a physical query plan, The query operator and the second query operator only generate one physical query task. Before generating the physical query task, the third query operator at the interval between the first query operator and the second query operator needs to be rewritten, otherwise When generating a physical query task, since the third query operator spaced between the first query operator and the second query operator has a partition operator, it means that there is a partition operator between the first query operator and the second query operator. A partition operator, when the query plan conversion device generates a physical query task, at least two physical query tasks can be generated according to the first query operator and the second query operator, and the physical query that constitutes the physical query plan cannot be reduced. The purpose of the number of tasks.

Based on the above description, when there is a third query operator spaced between the first query operator and the second query operator, the query plan conversion device can rewrite the third query operator as a broadcast query operator, and the broadcast query operator The operator has no partition operator. Therefore, since the broadcast query operator between the first query operator and the second query operator has no partition operator, after deleting the partition operator of the second query operator , there is no partition operator between the first query operator and the second query operator, when the query plan conversion device generates a physical query task, it can only generate one physical query task according to the first query operator and the second query operator , the purpose of reducing the number of physical query tasks that constitute the physical query plan can be achieved.

Exemplarily, as shown in FIG. 4 or FIG. 5, the data to be operated by the query operator 2 is the result of the operation of the query operator 3, and the data to be operated by the query operator 3 is the result of the operation of the query operator 1. , then it is determined that there is an indirect predecessor-successor relationship between query operator 1 and query operator 2. Among them, the partition attribute of query operator 1 in the logical query plan before rewriting in FIG. 4 is the prefix of the partition attribute of query operator 2, and the partition attribute of query operator 1 in the logical query plan before rewriting in FIG. 5 is the same as the query The partition properties of operator 2 are the same.

S205. The query plan conversion apparatus determines the relationship between the partition attribute of the first query operator and the partition attribute of the second query operator.

The relationship between the partition attribute of the first query operator and the partition attribute of the second query operator includes: the partition attribute of the first query operator is the prefix of the partition attribute of the second query operator, the partition of the first query operator The properties are the same as the partition properties of the second query operator.

Specifically, if the partition attribute of the first query operator is the prefix of the partition attribute of the second query operator, then continue to execute S206 and subsequent processes; if the partition attribute of the first query operator is the same as the partition attribute of the second query operator If the same, then continue to execute S207 and subsequent processes.

S206. The query plan conversion device rewrites the partition attribute of the second query operator in the logical query plan, so that the partition attribute of the second query operator is the same as the partition attribute of the first query operator.

The query plan conversion device can rewrite the partition attribute of the second query operator in the logical query plan, that is, rewrite the partition attribute of the second query operator to the partition attribute of the first query operator, while maintaining the partition attribute of the first query operator. The partition attribute remains unchanged to ensure that the partition attribute of the rewritten second query operator is the same as the partition attribute of the first query operator, so that after the rewrite, the first query operator and the second query operator have exactly the same partition attribute , satisfies the condition that the pair of query operators required by the prior art to generate a physical query task has exactly the same partition attribute, so that when the logical query plan is converted into a physical query plan, the first query operator and the second query operator Generate a physical query task.

Exemplarily, as shown in FIG. 4 , the query plan conversion device may rewrite the partition attribute of the query operator 2 in the “logical query plan after rewriting the broadcast query operator”, and convert the partition attribute of the query operator 2 to the “partition attribute”. attribute 1, partition attribute 2" is rewritten as the partition attribute "partition attribute 1" of query operator 1, while keeping the partition attribute "partition attribute 1" of query operator 1 unchanged, so that the partition attribute of query operator 1 is the same as the query If the partition attributes of operator 2 are the same, the "logical query plan after rewriting the partition attributes" as shown in Figure 4 can be obtained.

S207. The query plan conversion apparatus deletes the partition operator of the second query operator from the logical query plan.

It should be noted that, at this time, the partition attribute of the second query operator is exactly the same as that of the first query operator, and the partition attribute of the query operator that meets the requirements of the prior art is exactly the same. After the partition operator of the second query operator is deleted, since there is a predecessor-successor relationship between the first query operator and the second query operator, and there is no more partition between the first query operator and the second query operator Therefore, the partition operator of the first query operator is used as an identifier for generating a new physical query task, and the first query operator and the second query operator can generate a physical query task.

Exemplarily, the query plan conversion device may use the “logical query plan after rewriting the partition attribute” as shown in FIG. 4 or the “logical query plan after rewriting” as shown in FIG. "Logical query plan before partition operator", and delete the partition operator of query operator 2 in "Logical query plan before delete partition operator" to obtain "Logical query after delete partition operator" as shown in Figure 6 plan".

S208, the query plan conversion device generates a physical query task according to the second query operator, the first query operator and the partition operator of the first query operator, so as to form a physical query plan.

Wherein, when there is a third query operator spaced between the first query operator and the second query operator, the query plan conversion device operates according to the partition of the second query operator, the first query operator and the first query operator The method for generating a physical query task by the operator may specifically be: generating a physical query task according to the second query operator, the broadcast query operator, the first query operator and the partition operator of the first query operator.

Exemplarily, the query plan conversion apparatus may use a job-flow correlation optimization (Job-flow Correlation, JFC) technology to generate a physical query according to the second query operator, the first query operator, and the partition operator of the first query operator. tasks to form the physical query plan. The query plan conversion device uses the JFC technology to generate a physical query task, and the specific method for forming a physical query plan is similar to the method for using the JFC technology to generate a physical query task in the prior art to form a physical query plan, and the embodiment of the present invention will not be repeated here. .

In the second application scenario of the embodiment of the present invention, as shown in FIG. 7 , after the query plan transformation device extracts the first query operator and the second query operator, the query plan transformation device may first determine the first query operation The relationship between the partition attribute of the operator and the partition attribute of the second query operator is executed, that is, S208 is executed.

In the second application scenario, as shown in FIG. 7 , after executing S205, if the query plan conversion apparatus determines that the partition attribute of the first query operator is the prefix of the partition attribute of the second query operator, it can continue to execute S206 , that is, the query plan transformation device rewrites the partition attribute of the second query operator in the logical query plan, so that the partition attribute of the second query operator is the same as the partition attribute of the first query operator; if the query plan transformation device determines that the first query If the partition attribute of the operator is the same as the partition attribute of the second query operator, then directly execute S207, that is, the query plan transformation device deletes the partition operator of the second query operator from the logical query plan; then the query plan transformation device determines the first query plan transformation device. Check whether there is a third query operator between the query operator and the second query operator, that is, go to S202; if the query plan conversion device determines that there is a third query operator between the first query operator and the second query operator , then continue to execute S204, and if the query plan conversion apparatus determines that there is no third query operator spaced between the first query operator and the second query operator, then directly execute S208.

In the embodiment of the present invention, the following formula is used to calculate the beneficial effect brought by the query plan conversion method provided by the embodiment of the present invention.

The time spent executing a physical query task consists of four parts: the time spent reading data, the time spent writing data, the time spent transmitting data over the network, and the time spent in CPU operations. The time used, and the size of the data input before the query operator operation in the current physical query task, to calculate the time spent reading data; detecting the time used to write data per unit data volume, and the current physical query task. The size of the output data volume after the operation of the query operator is used to calculate the time spent writing data; the time it takes for the CPU to perform operations on the data per unit of data is detected, and the query operator in the current physical query task uses the input data. The percentage and the amount of data input before the query operator operation in the current physical query task are used to calculate the time consumed by the CPU operation; the time it takes for the network to transmit data per unit of data is detected and compared with the current physical query task. The percentage of input data used by the query operator and the amount of data input before the query operator operation in the current physical query task are used to calculate the time spent on network transmission of data.

For the three query operators OP _i , OP _j and OP _k in a logical query plan, it is satisfied that OP _i is the immediate predecessor operator of OP _j , OP _j is the immediate predecessor operator of OP _k , and the partition of OP _i The attribute is the prefix of the partition attribute of OP _k , and OP _j can be done by broadcasting the query operator.

Before adopting the query plan conversion method provided by the embodiment of the present invention, the three query operators generate three physical query tasks, and the time spent in executing the three physical query tasks is the time spent in executing the three physical query tasks respectively. and can be calculated according to the following formula 1:

Cost _before =H _r ×I _i +v×I _i +μ×a _i ×I _i +v×a _i ×I _i +H _w ×M _i +H _r ×I _j +v×I _j +μ×a _j ×I _j +v×a _j ×I _j +H _w ×M _j +H _r ×I _k +v×I _k +μ×a _k ×I _k +v×a _k ×I _k +H _w ×M _k

Formula one

Wherein, Cost _before is used to represent the time taken to execute three physical query tasks before adopting the query plan conversion method provided by the embodiment of the present invention, H _r represents the time used to read data of a unit data volume, and H _w represents a unit data volume of writing The time it takes for the data to be generated, v represents the time it takes for the CPU to perform operations on the unit data volume, μ represents the time it takes for the unit data volume data to be transmitted in the network, I _i , I _j and I _k represent the current three The size of the data input before the query operators OP _i , OP _j and OP _k in the physical query task, M _i , M _j and M _k respectively represent the query operators OP _i , OP i , and OP k in the current three physical query tasks. The size of the data output after the operations of OP _j and OP _k , a _i , a _j and a _k respectively represent the query operators OP _i , OP _j and OP _k in the currently executed three physical query tasks used for the input data. percentage.

After using the query plan conversion method provided by the embodiment of the present invention, the query operator OP _k and the query operator OP _i have the same partition attribute, the query operator OP _j is rewritten as a broadcast query operator, and three query operators can be generated as one For a physical query task, since the three query operators are in the same physical query task, only one read data operation and one data write operation can be performed when the physical query task is executed. Although the broadcast query operator in this physical query task It is not necessary to perform read data operations on the output data of the query operator OP _i , but it is still necessary to perform read data operations on data from other input sources. Therefore, the time spent by the system to execute this physical query task can be calculated according to the following formula 2:

Cost _after =H _r ×I _i +v×I _i +μ×a _i ×I _i +v×a _i ×I _i +H _r ×(I _j -M _i )+v×r×I _j +v× a _j ×I _j +v×a _k ×I _k +H _w ×M _k

Formula two

Wherein, Cost _after is used to represent the time taken to execute a physical query task generated by the three query operators OP _i , OP _j and OP _k after using the query plan conversion method provided by the embodiment of the present invention, and r represents the value of M _i The number of processes, H _r represents the time it takes to read the unit data volume, H _w represents the time it takes to write the unit data volume data, v represents the CPU operation time for the unit data volume data, μ represents the unit data in the network The time it takes to transmit the amount of data, I _i , I _j and I _k respectively represent the amount of data input before the operation of the query operators OP _i , OP _j and OP _k in the current physical query task, M _i and M _k represents the size of the data output after the operation of the query operators OP _i and OP _k in the current physical query task, respectively, a _i , a _j and a _k respectively represent the query operator OP _i in the currently executed physical query task , OP _j , and OP _k are used as a percentage of the input data.

The time saved by using the query plan conversion method provided by the embodiment of the present invention is the time spent before using the query plan conversion method and device provided by the embodiment of the present invention and the time spent after using the query plan conversion method and device provided by the embodiment of the present invention. difference in time spent. The time that can be saved by adopting the query plan conversion method and device provided by the embodiment of the present invention can be calculated by the following formula 3:

Cost _saved =Cost _before -Cost _after =H _w ×M _i +H _r ×M _i -v×(r-1)×I _j +μ×a _j ×I _j +H _w ×M _j +H _r ×I _k +v×I _k +μ× _ak ×I _k =(H _w +H _r )×M _i +(μ×a _j -v×(r-1))×I _j +H _w ×M _j + (H _r +v+μ× _ak )×I _k

Formula three

Wherein, Cost _saved is used to represent the time that can be saved after adopting the query plan conversion method provided by the embodiment of the present invention, r represents the number of processes of _Mi , H _r represents the time used to read data of a unit data amount, and H _w represents the writing unit The time taken by the data of the data volume, v represents the time it takes for the CPU to perform operations on the data of the unit data volume, μ represents the time it takes for the data of the unit data volume to be transmitted in the network, I _j and I _k represent the query operator OP _j respectively and the size of the data input before the operation of OP _k , M _i and M _j represent the size of the data output after the operation of the query operators OP _i and OP _j respectively, a _j and a _k represent the query operators OP _j and OP respectively The percentage of _k used on the input data.

When read data operations and write data operations in the system dominate the time overhead, that is, when the proportion of read data operations and write data operations in the time overhead is greater than other time overheads, for the coefficient μ in formula 3 (the network to unit data volume The time used for transmission of the data) and the coefficient v (the time used by the CPU to calculate the data of the unit data volume) can be ignored, then the time that can be saved after adopting the query plan conversion method provided by the embodiment of the present invention is:

Cost _saved =(H _w +H _r )×M _i +H _w ×M _j +H _r ×I _k

Wherein, Cost _saved is used to represent the time that can be saved after adopting the query plan conversion method provided by the embodiment of the present invention, H _r represents the time used to read data of a unit data volume, H _w represents the time used to write data of a unit data volume, I _k represents the size of the input data before the operation of the query operator OP _k , and M _i and M _j respectively represent the size of the output data after the operation of the query operator OP _i and OP _j .

When the network transmission in the system dominates the time overhead, that is, the network transmission accounts for a larger proportion of the time overhead than other time overheads, for the coefficient H _r (the time it takes to read data of a unit amount of data) in formula 3, the coefficient H _w (the time it takes to write the data of the unit data volume) and the coefficient v (the time it takes for the CPU to perform the operation on the data of the unit data volume) can be ignored, then the time that can be saved by using the query plan conversion method provided by the embodiment of the present invention for:

Cost _saved = μ×a _j ×I _j +μ× _ak ×I _k

Wherein, Cost _saved is used to represent the time that can be saved after adopting the query plan conversion method provided by the embodiment of the present invention, μ represents the time taken for data transmission of unit data volume in the network, I _j and I _k represent the query operator OP respectively The size of the input data before the operation of _j and OP _k , a _j and a _k represent the percentage of the input data used by the query operators OP _j and OP _k , respectively.

When the proportion of the time overhead of read data operations and write data operations and network transmission in the system is equal, that is, when the proportion of read data operations and write data operations in the time overhead is equal to the proportion of network transmission in the time overhead, In formula 3, only the coefficient v (the time it takes for the CPU to perform operations on the data of the unit data volume) is ignored, then the time that can be saved after adopting the query plan conversion method provided by the embodiment of the present invention is:

Cost _saved =(H _w +H _r )×M _i +μ×a _j ×I _j +H _w ×M _j +(H _r +μ+ _ak )×I _k

Wherein, Cost _saved is used to represent the time that can be saved after adopting the query plan conversion method provided by the embodiment of the present invention, H _r represents the time used to read data of a unit data volume, H _w represents the time used to write data of a unit data volume, μ represents the time it takes to transmit a unit of data in the network, I _j and I _k represent the size of the data input before the operation of the query operators OP _j and OP _k , respectively, and M _i and M _j represent the query operators OP, respectively The size of the output data after the operations of _i and OP _j , a _j and a _k represent the percentage of the input data used by the query operators OP _j and OP _k , respectively.

Further optionally, in order to improve the query efficiency of the data query system, before deleting the partition operator of the second query operator from the logical query plan, the query plan conversion method may further include: rewriting the first query operator. ordering properties, so that the ordering properties of the first query operator are the same as the ordering properties of the second query operator.

Wherein, the sorting attribute of the first query operator before rewriting is the same as the partition attribute of the first query operator, and the sorting attribute of the second query operator is the same as the partitioning attribute of the second query operator before the rewriting The same; the sorting attribute is used to perform partition sorting on the data in the data table operated by the query operator of the logical query plan.

It should be noted that the partition operator sorts the data in the data table operated by the query operator of the logical query plan according to the sorting attribute of the query operator.

When the partition attribute of the first query operator is the prefix of the partition attribute of the second query operator before rewriting, that is, the sorting attribute of the first query operator is the prefix of the sorting attribute of the second query operator. Assuming that the sorting attribute of the first query operator is "sort attribute 1", and the sorting attribute of the second query operator is "sort attribute 1, sorting attribute 2", the partition operator according to the sorting attribute of the first query operator When sorting the data table operated by the first query operator and the second query operator, all the rows in the data table with the same sorting attribute 1 are sorted together, and the partition operator is based on the sorting attribute of the second query operator. When the data table operated by the first query operator and the second query operator is partitioned and sorted, all rows in the data table with the same sorting attribute 1 and the same sorting attribute 2 are sorted together.

In the scenario where the sorting attribute is not rewritten, when the logical query plan is converted into a physical query plan, since the partition operator of the second query operator is deleted, the first query operator and the second query operator will generate a physical query Task, the partition operator will sort the data table that the first query operator and the second query operator need to operate according to the sorting attribute of the first query operator, because the partitioning attribute and sorting attribute of the first query operator are the same , so the result of partition sorting according to the sorting attribute of the first query operator is the same as before the partition sorting. When the second query operator operates on the data table sorted according to the sorting attribute of the first query operator, it still needs to perform a comprehensive scan on the data table to find out all the rows containing the sorting attribute 2. In this way, Will be more time consuming.

Rewriting the sorting attribute of the first query operator is to rewrite the sorting attribute of the first query operator to the sorting attribute of the second query operator, so that the partition operator will be based on the sorting attribute of the second query operator. Partition and sort the data table that the first query operator and the second query operator need to operate, and sort all the rows in the data table that contain the same sorting attribute 1 and the same sorting attribute 2 together. The second query operator does not need to be After a comprehensive scan of the data table, the data that needs to be manipulated can be found, which saves the time required to scan the data compared to the scenario where the sorting attribute is not rewritten.

In the query plan conversion method provided by the embodiment of the present invention, as long as the partition attribute of the first query operator is the prefix of the partition attribute of the second query operator, the partition attribute of the second query operator in the logical query plan can be rewritten, so as to facilitate When the partition attribute of the second query operator is the same as the partition attribute of the first query operator, the partition operator of the second query operator can be deleted to generate a physical query according to the first query operator and the second query operator tasks, reducing the number of physical query tasks that make up the physical query plan.

Compared with the prior art, the partition operator of the second query operator can be deleted only when the partition attribute of the first query operator is exactly the same as that of the second query operator. When the partition attribute of the second query operator is the prefix of the partition attribute of the second query operator, rewriting the partition attribute of the second query operator can obtain more query operator pairs that satisfy the predecessor-successor relationship (such as the first query operator and the second query operator), which can further reduce the number of physical query tasks that constitute the physical query plan.

And through this solution, when the first query operator is an indirect precursor operator of the second query operator, that is, there is a third query operator between the first query operator and the second query operator, and the third query operator When the broadcast query algorithm can be used, the third query operator can be rewritten as a broadcast query operator, so that when the partition attribute of the second query operator is the same as the partition attribute of the first query operator, the second query can be deleted. The partition operator of the operator generates a physical query task from the first query operator, the second query operator and the broadcast query operator, thereby reducing the number of physical query tasks constituting the physical query plan.

As in the prior art, the partition operator of the second query operator can be deleted only when the first query operator is the direct predecessor operator of the second query operator, so that the first query operation with the same partition attributes Compared with the second query operator to generate a physical query task, when there is a third query operator between the first query operator and the second query operator, the third query operator is rewritten as a broadcast query operator , so that the broadcast query operator and the second query operator with the same partition attribute can generate one physical query task, thereby reducing the number of physical query tasks constituting the physical query plan to a greater extent.

It should be noted that the logical query plan instances provided in the embodiments of the present invention are merely exemplary, and serve for illustrative purposes, and do not mean that the embodiments of the present invention can only be applied to these logical query plan instances.

Example 3

An embodiment of the present invention provides a query plan conversion device, as shown in FIG. 8 , including: an extraction unit 31 , a first rewrite unit 32 , a deletion unit 33 and a generation unit 34 .

The extraction unit 31 is configured to extract a first query operator and a second query operator from the logical query plan, where the first query operator is a precursor operator of the second query operator.

The first rewriting unit 32 is configured to rewrite the partition attribute of the second query operator in the logical query plan if the partition attribute of the first query operator extracted by the extracting unit 31 is the prefix of the partition attribute of the second query operator extracted by the extracting unit. Partition attributes so that the partition attributes of the second query operator are the same as the partition attributes of the first query operator.

The deletion unit 33 is configured to delete the partition operator of the second query operator from the logical query plan after rewriting the partition attribute of the second query operator in the logical query plan described by the first rewriting unit 32 .

The generating unit 34 is configured to generate a physical query task according to the second query operator, the first query operator and the partition operator of the first query operator after the deletion unit 33 deletes the partition operator of the second query operator, to form a physical query plan.

Further, as shown in FIG. 9 , the first rewriting unit 32 may include: a first rewriting module 321 and a second rewriting module 322 .

The first rewriting module 321 is used for if the partition attribute of the first query operator is a prefix of the partition attribute of the second query operator, and a third query operator is spaced between the first query operator and the second query operator , the third query operator can be implemented using a broadcast query algorithm, then the third query operator is rewritten as a broadcast query operator, and the broadcast query operator does not have a partition operator.

The second rewriting module 322 is configured to rewrite the partition attribute of the second query operator in the logical query plan, so that the partition attribute of the second query operator is the same as the partition attribute of the first query operator.

The generating unit 34 is further configured to generate a physical query task according to the second query operator, the broadcast query operator, the first query operator and the partition operator of the first query operator to form a physical query plan.

The first query operator is the direct predecessor operator of the third query operator, and the third query operator is the direct predecessor operator of the second query operator.

Further, as shown in FIG. 10 , the query plan conversion apparatus provided by the embodiment of the present invention may further include: a second rewriting unit 35 .

The second rewriting unit 35 is used for if the partition attribute of the first query operator extracted by the extraction unit 31 is the same as the partition attribute of the second query operator extracted by the extraction unit 31, and the first query operator and the second query operator There is a third query operator in between, and the third query operator can be implemented using a broadcast query algorithm, then the third query operator is rewritten as a broadcast query operator, and the broadcast query operator has no partition operator.

The deleting unit 33 is further configured to delete the partition operator of the second query operator from the logical query plan after the second rewriting unit 35 rewrites the third query operator as a broadcast query operator.

The generating unit 34 is further configured to, after the deleting unit 33 deletes the partition operator of the second query operator, according to the second query operator, the broadcast query operator, the first query operator and the first query operator The partition operator generates a physical query task to form the physical query plan.

The first query operator is the direct predecessor operator of the third query operator, and the third query operator is the direct predecessor operator of the second query operator.

Further, the generating unit 34 is specifically configured to use the task flow correlation optimization JFC technology to generate a physical query task according to the second query operator, the first query operator and the partition operator of the first query operator, to form a physical query task. query plan.

In the query plan conversion device provided by the embodiment of the present invention, if the partition attribute of the first query operator is a prefix of the partition attribute of the second query operator, the partition attribute of the second query operator in the logical query plan can be rewritten so as to When the partition attribute of the second query operator is the same as the partition attribute of the first query operator, the partition operator of the second query operator can be deleted to generate a physical query operator according to the first query operator and the second query operator. Query tasks, reducing the number of physical query tasks that make up the physical query plan.

Compared with the prior art, the partition operator of the second query operator can be deleted only when the partition attribute of the first query operator is exactly the same as that of the second query operator. When the partition attribute of the second query operator is the prefix of the partition attribute of the second query operator, rewriting the partition attribute of the second query operator can obtain more query operator pairs that satisfy the predecessor-successor relationship (such as the first query operator and the second query operator), which can further reduce the number of physical query tasks that constitute the physical query plan.

And through this solution, when the first query operator is an indirect precursor operator of the second query operator, that is, there is a third query operator between the first query operator and the second query operator, and the third query operator When the broadcast query algorithm can be used, the third query operator can be rewritten as a broadcast query operator, so that when the partition attribute of the second query operator is the same as the partition attribute of the first query operator, the second query can be deleted. The partition operator of the operator generates a physical query task from the first query operator, the second query operator and the broadcast query operator, thereby reducing the number of physical query tasks constituting the physical query plan.

As in the prior art, the partition operator of the second query operator can be deleted only when the first query operator is the direct predecessor operator of the second query operator, so that the first query operation with the same partition attributes Compared with the second query operator to generate a physical query task, when there is a third query operator between the first query operator and the second query operator, the third query operator is rewritten as a broadcast query operator , so that the broadcast query operator and the second query operator with the same partition attribute can generate one physical query task, thereby reducing the number of physical query tasks constituting the physical query plan to a greater extent.

Example 4

An embodiment of the present invention provides a query plan conversion apparatus, as shown in FIG. 11 , including: a memory 41 and a processor 42 ; the memory 41 is connected to the processor 42 .

The memory 41 is used to store a set of program codes, and the memory 41 is a computer storage medium of the query plan transformation apparatus, and the computer storage medium includes: a non-volatile storage medium.

The processor 42 is configured to execute the program code stored in the memory 41, and is specifically configured to perform the following operations: extracting a first query operator and a second query operator from a logical query plan, the first query The operator is the predecessor operator of the second query operator; if the partition attribute of the first query operator is the prefix of the partition attribute of the second query operator, rewrite the partition attribute of the second query operator, so that the partition attribute of the second query operator is the same as the partition attribute of the first query operator; delete the partition attribute of the second query operator from the logical query plan A partition operator is used, and a physical query task is generated according to the second query operator, the first query operator, and the partition operator of the first query operator, so as to form a physical query plan.

The memory 41 and the processor 42 are connected through a bus and communicate with each other.

The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into address bus, data bus, control bus and so on. For ease of presentation, only one thick line is used in FIG. 11, but it does not mean that there is only one bus or one type of bus.

Further, the processor 42 is further configured to, if the partition attribute of the first query operator is a prefix of the partition attribute of the second query operator, and the first query operator and the second query operator A third query operator is spaced between query operators, and the third query operator can be implemented using a broadcast query algorithm, then the third query operator is rewritten as a broadcast query operator, and the broadcast query operator does not have partition operator; rewrite the partition attribute of the second query operator in the logical query plan, so that the partition attribute of the second query operator is the same as the partition attribute of the first query operator; from the The partition operator of the second query operator is deleted from the logical query plan, and according to the second query operator, the broadcast query operator, the first query operator and the first query operator The partition operator generates a physical query task to compose the physical query plan.

The first query operator is a predecessor operator of the third query operator, and the third query operator is a predecessor operator of the second query operator.

Further, the processor 42 is further configured to, if the partition attribute of the first query operator is the same as the partition attribute of the second query operator, and the first query operator and the second query There is a third query operator spaced between the operators, and the third query operator can be implemented using a broadcast query algorithm, then the third query operator is rewritten as a broadcast query operator, and the broadcast query operator has no partition operator; delete the partition operator of the second query operator from the logical query plan, and use the second query operator, the broadcast query operator, the first query operator, and the The partition operator of the first query operator generates a physical query task to form the physical query plan.

The first query operator is a predecessor operator of the third query operator, and the third query operator is a predecessor operator of the second query operator.

Further, the processor 42 is further configured to rewrite the sorting attribute of the first query operator before deleting the partition operator of the second query operator from the logical query plan, so that the The ordering property of the first query operator is the same as the ordering property of the second query operator.

Wherein, the sorting attribute of the first query operator before rewriting is the same as the partition attribute of the first query operator, and the sorting attribute of the second query operator is the same as the partitioning attribute of the second query operator before the rewriting The same; the sorting attribute is used to perform partition sorting on the data in the data table operated by the query operator of the logical query plan.

Further, the processor 42 is further configured to adopt the task flow dependency optimization JFC technology, and generate the generated data according to the second query operator, the first query operator and the partition operator of the first query operator. A physical query task to compose the physical query plan.

It should be noted that, for the specific description of some functional modules in the query plan conversion device provided in the embodiment of the present invention, reference may be made to the corresponding content in the method embodiment, which will not be described in detail in this embodiment.

In the query plan conversion device provided by the embodiment of the present invention, if the partition attribute of the first query operator is a prefix of the partition attribute of the second query operator, the partition attribute of the second query operator in the logical query plan can be rewritten so as to When the partition attribute of the second query operator is the same as the partition attribute of the first query operator, the partition operator of the second query operator can be deleted to generate a physical query operator according to the first query operator and the second query operator. Query tasks, reducing the number of physical query tasks that make up the physical query plan.

Compared with the prior art, the partition operator of the second query operator can be deleted only when the partition attribute of the first query operator is exactly the same as that of the second query operator. When the partition attribute of the second query operator is the prefix of the partition attribute of the second query operator, rewriting the partition attribute of the second query operator can obtain more query operator pairs that satisfy the predecessor-successor relationship (such as the first query operator and the second query operator), which can further reduce the number of physical query tasks that constitute the physical query plan.

And through this solution, when the first query operator is an indirect precursor operator of the second query operator, that is, there is a third query operator between the first query operator and the second query operator, and the third query operator When the broadcast query algorithm can be used, the third query operator can be rewritten as a broadcast query operator, so that when the partition attribute of the second query operator is the same as the partition attribute of the first query operator, the second query can be deleted. The partition operator of the operator generates a physical query task from the first query operator, the second query operator and the broadcast query operator, thereby reducing the number of physical query tasks constituting the physical query plan.

As in the prior art, the partition operator of the second query operator can be deleted only when the first query operator is the direct predecessor operator of the second query operator, so that the first query operation with the same partition attributes Compared with the second query operator to generate a physical query task, when there is a third query operator between the first query operator and the second query operator, the third query operator is rewritten as a broadcast query operator , so that the broadcast query operator and the second query operator with the same partition attribute can generate one physical query task, thereby reducing the number of physical query tasks constituting the physical query plan to a greater extent.

From the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of the above functional modules is used as an example for illustration. In practical applications, the above functions can be allocated as required. It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the functions described above. For the specific working process of the system, apparatus and unit described above, reference may be made to the corresponding process in the foregoing method embodiments, and details are not described herein again.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be Incorporation may either be integrated into another system, or some features may be omitted, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, removable hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention. should be included within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (10)

1. a query plan conversion method, is characterized in that, comprises: extracting a first query operator and a second query operator from the logical query plan, where the first query operator is a precursor operator of the second query operator; If the partition attribute of the first query operator is a prefix of the partition attribute of the second query operator, rewrite the partition attribute of the second query operator in the logical query plan, so that the second query operator The partition attribute of the query operator is the same as the partition attribute of the first query operator; The partition operator of the second query operator is deleted from the logical query plan, and is generated according to the second query operator, the first query operator, and the partition operator of the first query operator A physical query task to form the physical query plan. 2 . The method according to claim 1 , wherein, if the partition attribute of the first query operator is a prefix of the partition attribute of the second query operator, rewrite the partition attribute in the logical query plan. 3 . The partition attribute of the second query operator, so that the partition attribute of the second query operator is the same as the partition attribute of the first query operator, including: If the partition attribute of the first query operator is a prefix of the partition attribute of the second query operator, and a third query operator is spaced between the first query operator and the second query operator , the third query operator can be implemented using a broadcast query algorithm, then the third query operator is rewritten as a broadcast query operator, and the broadcast query operator does not have a partition operator; rewriting the partition attribute of the second query operator in the logical query plan, so that the partition attribute of the second query operator is the same as the partition attribute of the first query operator; The partition operator of the second query operator is deleted from the logical query plan, and the partition operation is performed according to the second query operator, the first query operator and the first query operator The operator generates a physical query task to form the physical query plan, including: The partition operator of the second query operator is removed from the logical query plan, and the second query operator, the broadcast query operator, the first query operator, and the first query The partition operator of the operator generates a physical query task to form the physical query plan; The first query operator is a predecessor operator of the third query operator, and the third query operator is a predecessor operator of the second query operator. 3. The method of claim 1, further comprising: If the partition attribute of the first query operator is the same as the partition attribute of the second query operator, and a third query operator is spaced between the first query operator and the second query operator, The third query operator can be implemented using a broadcast query algorithm, then the third query operator is rewritten as a broadcast query operator, and the broadcast query operator does not have a partition operator; The partition operator of the second query operator is removed from the logical query plan, and the second query operator, the broadcast query operator, the first query operator, and the first query The partition operator of the operator generates a physical query task to form the physical query plan; The first query operator is a predecessor operator of the third query operator, and the third query operator is a predecessor operator of the second query operator. 4. The method according to any one of claims 1-3, characterized in that before the partition operator of the second query operator is deleted from the logical query plan, the method further comprises: : rewriting the sorting attribute of the first query operator so that the sorting attribute of the first query operator is the same as the sorting attribute of the second query operator; Wherein, the sorting attribute of the first query operator before rewriting is the same as the partition attribute of the first query operator, and the sorting attribute of the second query operator is the same as the partitioning attribute of the second query operator before the rewriting same; The sorting attribute is used to perform partition sorting on the data in the data table operated by the query operator of the logical query plan. 5. The method according to any one of claims 1-3, wherein the partition operation according to the second query operator, the first query operator and the first query operator The operator generates a physical query task to form the physical query plan, including: Using task flow associative optimization JFC technology, a physical query task is generated according to the second query operator, the first query operator and the partition operator of the first query operator to form the physical query plan . 6. A query plan conversion device, characterized in that, comprising: an extraction unit, configured to extract a first query operator and a second query operator from the logical query plan, where the first query operator is a precursor operator of the second query operator; a first rewriting unit, configured to rewrite the logical query if the partition attribute of the first query operator extracted by the extracting unit is a prefix of the partition attribute of the second query operator extracted by the extracting unit the partition attribute of the second query operator in the plan, so that the partition attribute of the second query operator is the same as the partition attribute of the first query operator; a deletion unit, configured to delete the partition operator of the second query operator from the logical query plan after the first rewriting unit rewrites the partition attribute of the second query operator in the logical query plan ; a generating unit, configured to, after the deleting unit deletes the partition operator of the second query operator, generate the partition according to the second query operator, the first query operator and the partition of the first query operator The operator generates a physical query task to form the physical query plan. 7. The device according to claim 6, wherein the first rewriting unit comprises: The first rewriting module is used for if the partition attribute of the first query operator is a prefix of the partition attribute of the second query operator, and the relationship between the first query operator and the second query operator is There is a third query operator in the interval, and the third query operator can be implemented using a broadcast query algorithm, then the third query operator is rewritten as a broadcast query operator, and the broadcast query operator does not have a partition operator; a second rewriting module, configured to rewrite the partition attribute of the second query operator in the logical query plan, so that the partition attribute of the second query operator is the same as the partition attribute of the first query operator; The generating unit is further configured to generate a physical query task according to the second query operator, the broadcast query operator, the first query operator and the partition operator of the first query operator, to constitute the physical query plan; The first query operator is a predecessor operator of the third query operator, and the third query operator is a predecessor operator of the second query operator. 8. The apparatus of claim 6, further comprising: The second rewriting unit is configured to, if the partition attribute of the first query operator extracted by the extracting unit is the same as the partition attribute of the second query operator extracted by the extracting unit, and the first query operation There is a third query operator spaced between the operator and the second query operator, and the third query operator can be implemented using a broadcast query algorithm, then the third query operator is rewritten as a broadcast query operator, so The broadcast query operator described above does not have a partition operator; The deleting unit is further configured to delete the partition of the second query operator from the logical query plan after the second rewriting unit rewrites the third query operator as the broadcast query operator operator; The generating unit is further configured to, after the deleting unit deletes the partition operator of the second query operator, according to the second query operator, the broadcast query operator, the first query operator and the partition operator of the first query operator generates a physical query task to form the physical query plan; The first query operator is a predecessor operator of the third query operator, and the third query operator is a predecessor operator of the second query operator. 9. The device according to any one of claims 6-8, further comprising: a third rewriting unit, configured to rewrite the sorting attribute of the first query operator before the deletion unit deletes the partition operator of the second query operator from the logical query plan, so that the first query operator The sorting attribute of a query operator is the same as the sorting attribute of the second query operator; Wherein, the sorting attribute of the first query operator before rewriting is the same as the partition attribute of the first query operator, and the sorting attribute of the second query operator is the same as the partitioning attribute of the second query operator before the rewriting same; The sorting attribute is used to perform partition sorting on the data in the data table operated by the query operator of the logical query plan. 10. The apparatus according to any one of claims 6-8, wherein the generating unit is specifically configured to adopt the task flow correlation optimization JFC technology, according to the second query operator, the first A query operator and the partition operator of the first query operator generate a physical query task to form the physical query plan.