CN115757525A - Column operator blood relationship construction method, server and computer readable storage medium - Google Patents

Abstract

The present invention provides a column operator lineage construction method, which is applied in the field of computer technology and includes the following steps: S1) generating a parsing tree ParseTree through Antlr parsing SQL; S2) designing an abstract syntax tree AST for building lineage chains from input columns to output columns Road and intermediate processing logic; S3) recursively traverse the parse tree ParseTree obtained by S1) to construct the abstract syntax tree AST designed by S2); S4) traverse the abstract syntax tree AST constructed by S3) and extract the column operator kinship model; S5) Traverse the lineage model of column operators and build point-edge relationships, and store the lineage of column operators into the graph database. Provides the most fine-grained SQL analysis capability to help users understand the operator-level lineage of data assets, including the direct source of columns, functions used, processing caliber, and indirect impacts, and solves the problem that users do not understand data assets and cannot effectively empower them business problem.

Description

Column operator lineage construction method, server, and computer-readable storage medium

technical field

The present invention relates to the field of computer technology, in particular to a method for constructing lineage of column operators, a server, and a computer-readable storage medium.

Background technique

More and more enterprises need to carry out digital transformation and empower business development through data analysis. This is already an inevitable path for future enterprise development. However, in order to adapt to the development of the enterprise, the internal data processing methods and processing logic of the enterprise are becoming more and more complex. As time goes by, users often cannot quickly and easily understand the processing logic of historical data, and cannot quickly respond to the increasingly agile business need. In order not to affect the agile implementation of data analysis, users need to write comments for processing scripts or write a complete document to back up the thinking logic of data processing at that time, so as to avoid ignorance of data when data analysts analyze data. lead to errors in the analysis results. However, this depends on the self-consciousness of users, and the implementation of knowledge in enterprises cannot currently be scaled up.

Through the blood relationship of data, the upstream and downstream blood relationship of data assets can be queried. The existing technology can only detect the upstream and downstream paths of data assets, such as what is the upstream and downstream of the library, table, and column; the current solution is basically All of the above: analyze the processing logic script of data assets, form table-level or column-level data lineage, let users understand the source and application of data in a timely manner through visualization, and provide a basis for subsequent data analysis; the defects of the above method are as follows, Use the following code:

INSERT INTO T1(C1)

SELECT MAX(T2.C1+T3.C1)as C1 FROM T2,T3 WHERE T2.C2>1;

Among them, the table lineage can only show the processing relationship between tables. For example, the T1 table is derived from the upstream T2 and T3 tables, and the column-level lineage can show that T1.C1 is derived from T2.C1 and T3.C1, but it cannot be known. T1.C1 is obtained by adding T2.C1 and T3.C1 and then using the MAX function. At the same time, it is impossible to know that the data range of T1.C1 is affected by the screening of T2.C2.

Therefore, the processing logic of different data assets may be consistent in terms of usage sources, for example, they all use the same table or column, but there are a large number of conversion rules in the real business logic of data assets, which users cannot intuitively understand through existing methods How each conversion, calculation, and movement of data will affect the data will lead to incomplete understanding by users. It is still necessary to consult developers offline, which cannot meet the existing agile business needs, that is, only know where the columns come from, but not Know how.

On the other hand, the direct data source of data assets and the data source that indirect processing relies on represent different data meanings and business meanings. Through the existing methods, users will develop a large-scale expansion of upstream sources and downstream applications, which cannot be intuitively perceived The direct source of the data, which leads to the failure of data analysis, now requires manual query path to record the processing logic, where is the source of the query data;

Furthermore, the label diffusion based only on the column-level blood relationship is also inaccurate. Because the data is processed layer by layer, the meaning it carries will change, because the program cannot know how the column is processed. Diffusion layer by layer down through the label will greatly reduce the accuracy of the diffusion.

Contents of the invention

In order to solve the above technical problems, the present invention provides the most fine-grained SQL parsing ability, helps users understand the operator-level lineage of data assets, including the direct source of the column, the function used, the processing caliber and the indirect influence received, and solves the user's concerns about data assets. The problem of not being able to effectively empower the business due to lack of understanding.

In order to solve the above-mentioned main technical problems, the following technical solutions are adopted to realize:

The column operator lineage construction method includes the following steps,

S1) parse SQL through Antlr to generate a parsing tree ParseTree;

S2) Design the abstract syntax tree AST, which is used to construct the lineage link from the input column to the output column and the processing logic in the middle;

S3) recursively traverse the parse tree ParseTree that S1) obtains and build the abstract syntax tree AST of S2) design;

S4) Traversing the abstract syntax tree AST constructed in S3) and extracting the column operator consanguinity model;

S5) Traversing the lineage model of the column operator and constructing a point-edge relationship, and storing the lineage of the column operator into the graph database.

Preferably, S2) comprises the steps of:

S21) Based on relational algebra, a complete SQL is divided into at least one paragraph, each paragraph is a trunk, and a tree structure Scope is designed to abstract the corresponding trunk of SQL, and establish a hierarchical relationship between the trunk and the trunk;

S22) Scope is divided into two types: input type Scope and output type Scope;

Among them: the input type Scope is divided into ProjectScope, JoinScope, UnionScope and ScanScope, which are used to abstract the input part of the SQL statement;

The output type Scope is divided into CreateAsScope and InsertScope, which are used to abstract the output part of the SQL statement;

Among them: each Scope includes the type of the Scope, the alias of the Scope, and the fields exposed to the outside of the Scope. The parent-child relationship between different Scopes is recorded through the index, and the outer backbone set of the Scope is set to parentScopeList, which is the parent Scope set; set The inner backbone collection of Scope is childrenScopeList, which is the collection of child Scopes, so as to build the parent-child relationship between different Scopes;

S23) define the 6 kinds of Scopes divided by S22), which are used to correspond to the abstract syntax tree AST;

S24) Define the externally exposed fields of each Scope as the holding field set of the Scope, and the holding field set of each Scope can be referenced by the parent Scope in its parentScopeList;

S25) Attach a source attribute to each holding field in the holding field set

ExpressionOrigin is used to record the source information of the held field, which is used to build the lineage link between the input column and the output column;

S26) Establish an expression for recording the processing logic of the column and the source of the column;

S27) Each Scope uses the field information of the child Scope in its childrenScopeList to fill the field information of the current Scope, and then transfers the field information of the current Scope to the parent Scope, thereby constructing the processing chain between the output column and the source column road, and then extract the lineage information of the column operator.

Preferably, S3) comprises the steps of:

S31) recursively traverse the parse tree ParseTree generated by S1) from the root node;

S32) interpret the root node as CreateAsScope or InsertScope according to the type of SQL, and then traverse the child nodes;

S33) If the node type is the query node select xx from, it is interpreted as ProjectScope, and the expression after SELECT, WHERE, GROUP BY, HAVING, ORDER BY is constructed; the child node after its FROM is parsed as the corresponding Scope, and Add it to the childScopeList of the current ProjectScope, wait for the field information of the Scope in the childScopeList to fill in the source information of the expression of the current ProjectScope, and then use the expression information to fill the field of the current ProjectScope;

S34) If the child node after FROM is a physical table node, then create a ScanScope and set it as the current Scope, and fill in the holding field of the current Scope according to the metadata information;

If the sub-node after FROM is a sub-query node, then skip to S33) for recursive processing;

If the child node after FROM contains more than one JOIN node, then create a JoinScope and set it as the current Scope, construct all the expressions after the ON condition, and skip the JoinItem on both sides of each JOIN one by one to S33) for recursive processing; finally The holding field of JoinScope is constructed in the way of widening the holding field of each child Scope in childScopeList, and fills in the source information of all ON expressions;

If the child node after FROM contains more than one UNION node, create a UnionScope and set it as the current Scope, and jump the UnionItems on both sides of each UNION to S33) for recursive processing; finally, the holding field of the UnionScope is based on each child Scope in the childScopeList Constructed in the way of superposition of holding fields;

S35) Traverse the parse tree ParseTree through S32)-S34), and construct the abstract syntax tree AST designed by S2).

Preferably, S4) comprises the steps of:

S41) Extracting information from Scope to build a column operator lineage model, and dividing the column operator lineage model into operator Operator, physical column Column, and virtual column VirtualColumn;

Divide operators into six types, including:

SELECT operator: the content of each projection item, that is, the content before AS after SELECT, is extracted from ProjectScope;

WHERE operator: the content after WHERE is extracted from ProjectScope;

GROUP operator: the content after GROUP is extracted from ProjectScope;

HAVING operator: the content after HAVING is extracted from ProjectScope;

JOIN operator: the content after the ON condition is extracted from JoinScope;

UNION operator: used to aggregate multiple SELECT statements, extracted from UnionScope;

Among them, the SELECT and UNION operators belong to the direct lineage, and are used to trace the processing logic and processing link of the column, and the WHERE, GROUP, HAVING, and JOIN operators belong to the indirect lineage, and are used to analyze the influence surface of the column;

The physical column Column includes the real input physical column and output physical column, which belongs to metadata information. The input physical column is extracted from ScanScope, and the output physical column is extracted from CreateAsScope or InsertScope;

Virtual column VirtualColumn: the alias part in SELECT item AS alias in the subquery, extracted from ProjectScope.

The second aspect of the present invention further proposes a server, including a memory, a processor, and a computer program stored in the memory and operable on the processor, and the processor implements the steps of the above method when executing the program.

In a third aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the above method are implemented.

The beneficial effects of the present invention are: compared with the prior art, the method for constructing lineages of operators 1) judges all direct and indirect relationships between data assets in the complex data environment of an enterprise by deeply analyzing the processing logic scripts of data assets. The detailed logic of dependencies translates real data processing codes into user-friendly expression capabilities, avoiding inability to self-service data analysis due to incomprehension of data or data source application expansion during the data analysis process, affecting agile business demand response;

2) Column operator blood relationship is a more fine-grained blood relationship, which can be used to analyze the direct source, function used, processing caliber, and indirect impact of the column.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those skilled in the art Ordinary technicians can also obtain other drawings based on these drawings on the premise of not paying creative work.

Fig. 1 is the system flowchart of the present invention;

Fig. 2 is 6 kinds of Scope framework system diagrams of the present invention;

Fig. 3 is the relation diagram of 6 kinds of Scopes of the present invention;

Fig. 4 is step 3) recursive traversal flowchart of the present invention;

Fig. 5 is a relationship diagram between Scope construction and blood relationship filling in the present invention;

Fig. 6 is the flow chart of extracting operator in Scope in the present invention;

Fig. 7 is a schematic diagram of comparison between operators extracted in Scope and Scope structure in the present invention;

Fig. 8 is a schematic diagram of storing points and edges in a graph database.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Example 1

Please refer to Figure 1, the column operator consanguinity construction method, including the following steps:

S1) Analyzing SQL through Antlr to generate a parse tree ParseTree; ParseTree generated after parsing SQL through Antlr is only an abstract display of the information of each node in the original SQL, and cannot directly extract column operator blood relationship information from it. Therefore, it is necessary to redesign an abstract syntax tree AST according to the basic syntax of relational algebra and SQL. This abstract syntax tree AST decomposes SQL into several trunks and expressions, and records the relationship between the trunk and the trunk, and between expressions. blood relationship to achieve the purpose of extracting the blood relationship of column operators.

Extracting the lineage of column operators requires SQL to have an input part and an output part, that is, a query part and an output part, so there are two types of SQL for extracting lineage of column operators in this patent: INSERT INTO TABLE SELECT... and CREATE TABLE ASSELECT.... The input part is a DQL statement, and the output part is INSERT or CREATE.

S2) Design the abstract syntax tree AST, which is used to construct the lineage link from the input column to the output column and the processing logic in the middle;

The structure of the above abstract syntax tree AST is as follows:

1) Based on relational algebra, a complete SQL is divided into several paragraphs, and each paragraph is a trunk. This patent defines a new tree structure Scope to abstract the corresponding trunk of SQL, and can reflect the relationship between the trunk and the trunk. hierarchical relationship between them. Each Scope contains the type of the Scope, the alias of the Scope, and the fields exposed to the outside of the Scope. The parent-child relationship between different Scopes is recorded through the index, and the outer backbone collection of the Scope is set to parentScopeList, which is the parent Scope collection; The inner backbone collection is childrenScopeList, which is the collection of child scopes, so as to build the parent-child relationship between different scopes;

2) The above-mentioned SQL supporting the extraction of column operator lineage includes an input part and an output part. Therefore, Scope is also divided into two types: input type Scope and output type Scope; among them, input type Scope is divided into ProjectScope, JoinScope, UnionScope and ScanScope, which mainly abstract the input part in the SQL statement; output type Scope is divided into CreateAsScope and InsertScope, which mainly abstract There are two main output parts in the SQL statement;

3) Please refer to Figure 2 to define the 6 Scopes declared in 2). If the SQL is CREATE TBALEtable_name AS, it will be interpreted as CreateAsScope. Since CREATE TBALE table_name AS is at the beginning of a piece of SQL as the output part, Therefore, corresponding to the root node in the tree structure Scope, the part after AS is interpreted as other types of Scope, and the parent-child relationship is recorded with the index. If the SQL is INSERT INTO table_name (col1, col2…) SELECT, interpret the part before SELECT as InsertScope. Similarly, since INSERT INTO table_name (col1, col2…) is also at the beginning of a piece of SQL as the output part, so the corresponding In the tree structure Scope is the root node, the part after AS is interpreted as other types of Scope, and the parent-child relationship is recorded with the index. ProjectScope is used to abstract the structure of the SELECT statement, including the projection part, WHERE part, GROUP BY part, HAVING part and ORDER BY part of a SELECT statement. If the SELECT statement is followed by UNIONSELECT..., the first SELECT statement part to the last SELECT statement part are abstracted into a UnionScope, and each SELECT statement part is a separate ProjectScope, which is recorded into the childScopeList collection of the UnionScope through the index . In ProjectScope, the content after FROM and before WHERE is interpreted as a new Scope, and is recorded into the children Scope collection of this layer Scope through the index, that is, the inner backbone. If FROM is followed by a physical table name table_name, then the physical table name table_name is interpreted as ScanScope, and then the parent-child relationship is recorded through the index. In the tree structure Scope designed by the present invention, the leaf nodes are ScanScope; if the physical table name after FROM If there is a JOIN part after the table name, interpret the part after FROM and before WHERE as JoinScope, the alias of JoinScope is empty, and the content on both sides of each JOIN continues to be parsed into the corresponding Scope, and the ON condition content of each JOIN is recorded at the same time. If FROM is followed by a subquery (subquery) AS alias, then this subQuery is also interpreted as ProjectScope, and the alias is recorded as alias, and then the parent-child relationship is recorded through the index. If FROM is followed by multiple subqueries UNION together, that is (subQuery UNION subQuery ...) AS alias, then this part is interpreted as UnionScope, and so on.

Please refer to Figure 3. In the designed Scope abstract syntax tree AST, the parent-child relationship of different scopes is as follows:

CreateAsScope or InsertScope, their parentScopeList must be empty, childrenScopeList must not be empty and is one of ProjectScope or UnionScope.

The parentScopeList of ProjectScope is not empty and is one of CreateAsScope, InsertScope, JoinScope, ProjectScope, and UnionScope, and the childrenScopeList is not empty and is one of JoinScope, ProjectScope, UnionScope, and ScanScope.

The parentScopeList of JoinScope is not empty and must be ProjectScope, and the childrenScopeList is not empty and has at least two or more, which is any combination of ProjectScope, UnionScope, and ScanScope.

The parentScopeList of UnionScope is not empty and is one of CreateAsScope, InsertScope, JoinScope, ProjectScope, and UnionScope. The childrenScopeList is not empty and has at least two, which is any combination of ProjectScope and JionScope.

The parentScopeList of ScanScope is not empty, and it is one of ProjectScope and JoinScope, and the childrenScopeList must be empty.

4) Define each Scope’s exposed fields as the Scope’s held field set holdFields, and each Scope’s held field set can be referenced by its parentScopeList; the ScanScope’s held field set is metadata information, that is, the physical table The set of column names below is the set of held fields of the ScanScope. For example, the t1 table has three columns a, b, and c, then {"t1":["a", "b", "c"]} is the set of held fields of the ScanScope. The set of fields held by ProjectScope is the projected part in the SELECT statement, for example, SELECT a,b,c as c1 FROM, the alias is alias, if not, it is an empty string, then the set of fields held by the ProjectScope is {"alias ":["a", "b", "c1"]}. The UnionScope's holding field set is superimposed for each sub-Scope's holding field, such as (SELECT d,e,f)UNION(SELECT g,h,i), then the UnionScope's holding field set is {"alias": ["d", "e", "f"]}. The holding fields of JoinScope are stored after summarizing the collection of holding fields of each sub-scope. For example, t1 JOIN (SELECT d, e, f), where t1 has three columns a, b, and c, then the holding fields of this JoinScope The field collection is {"":[a,b,c,d,e,f]}, and the JoinScope alias must be empty, so the key holding the field collection is empty.

The set of fields held by CreateAsScope is the set of fields held by childrenScope, such as CRAETETABLE table_name AS SELECT a, b, c, then the set of fields held by this Scope is {"table_name":["a", "b", " c"]}. The set of fields held by InsertScope is the set of metadata column information of the target table. For example, INSERT INTOtable_name(a,b,c) SELECT d,e,f, then the set of fields held by the Scope is {“table_name”:[“a” , "b", "c"]}.

5) The Scope designed above has a parent-child relationship, that is, an origin relationship, so the holding fields of each Scope also have an origin relationship. Here, an origin attribute ExpressionOrigin is attached to the holding field in the holding field collection, that is, the holding field Which field is derived from which Scope in the childrenScopeList, it can be traced from the root Scope, that is, the field of CreateAsScope or InsertScope to the field of the physical layer ScanScope, so as to construct the blood relationship between the input column and the output column link.

6) The processing logic of the column cannot be seen intuitively only by holding the field. This patent defines Expression expression to record the expression form of the column. The expression can be a projection item, such as SELECT MAX(a+b) AS col1,c AScol2,1AS col3, where MAX(a+b) is an expression, and c and 1 are also an expression. Similarly, the content after WHERE is also an expression, such as WHERE c1>c2 AND c3=c4 OR c5+c6–MAX(c7), where c1>c2 AND c3=c4 OR c5+c6–MAX(c7) is an expression as a whole Mode.

Each subpart after GROUP BY and ORDER BY is also an expression. The storage structure of the expression is designed as a tree structure, each node has the type of the current expression, and the reference of each sub-expression, and the expression origin attribute ExpressionOrigin, which is used to store the origin information of the expression, and The source property ExpressionOrigin of the above holding field is the same property. Through the expression, you can clearly know what the processing logic of the column is, and at the same time know what the source of the column is through the source information of the expression, and finally pass it up layer by layer by holding the field, you can build the link between the output column and the source column Process the link, and then extract the lineage of column operators.

S3) Recursively traverse the parsing tree ParseTree obtained in S1) to construct the abstract syntax tree AST designed by S2); the process of constructing Scope and Expression is as follows: constructing the column operator lineage between the output column and the source column can be transformed into the construction of the Scope and Expression of this layer The dependency relationship between columns between the inner Scopes in childrenScopeList, further, each held field can only come from its "subquery", that is, from childrenScopeList, so "column operator lineage traceability construction" can be converted into "subquery Query holds padding issue for original field". The specific process is as follows:

Please focus on Figures 4 and 5,

S31) recursively traverse the parse tree ParseTree generated by S1) from the root node;

S32) interpret the root node as CreateAsScope or InsertScope according to the type of SQL, and then traverse the child nodes;

S33) If the node type is the query node select xx from, it is interpreted as ProjectScope, and the expression after SELECT, WHERE, GROUP BY, HAVING, ORDER BY is constructed; continue to explain the child node after its FROM as the corresponding Scope, And add it to the childScopeList of the current ProjectScope, wait for the field information of the Scope in the childScopeList to fill in the source information of the expression of the current ProjectScope, and then use the expression information to fill the field of the current ProjectScope;

S34) If the child node after FROM is a physical table node, then create a ScanScope and set it as the current Scope, and fill in the holding field of the current Scope according to the metadata information;

If the sub-node after FROM is a sub-query node, then skip to S33) for recursive processing;

If the child node after FROM contains more than one JOIN node, then create a JoinScope and set it as the current Scope, construct all the expressions after the ON condition, and skip the JoinItem on both sides of each JOIN one by one to S33) for recursive processing; finally The holding field of JoinScope is constructed in the way of widening the holding field of each child Scope in childScopeList, and fills in the source information of all ON expressions;

If the child node after FROM contains more than one UNION node, create a UnionScope and set it as the current Scope, and jump the UnionItems on both sides of each UNION to S33) for recursive processing; finally, the holding field of the UnionScope is based on each child Scope in the childScopeList Constructed in the way of superposition of holding fields;

Please refer to Figure 6 and Figure 7, S4) Traversing S3) to extract column operator blood relationship from the column blood relationship link model; after obtaining the above abstract syntax tree AST, it is necessary to traverse from the root Scope to extract the column operator blood relationship model The model contains three major contents: operator operator, physical column Column and virtual column VirtualColumn.

In this patent, the operators in the lineage of operators are named Operators, and there are 6 types in total:

Define SELECT operator: the content of each projection item, that is, the content before AS after SELECT, is extracted from ProjectScope; WHERE operator: the content after WHERE is extracted from ProjectScope; GROUP operator: the content after GROUP is extracted from ProjectScope HAVING operator: content after HAVING, extracted from ProjectScope; JOIN operator: content after ON condition, extracted from JoinScope; UNION operator: used to aggregate multiple SELECT statements, extracted from UnionScope.

Among them, SELECT and UNION operators belong to direct blood relationship and are used to trace the processing logic and processing links of columns. WHERE, GROUP, HAVING, and JOIN operators belong to indirect blood relationship and are used to analyze the impact of columns.

The physical column Column is the real input physical column and output physical column, which belongs to metadata information. Define the alias part in the SELECT item AS alias in the subquery as the virtual column VirtualColumn.

Please focus on Figure 7, for example: INSERT INTO t2(a,b)SELECT tt1.a AS a,tt1.b AS bFROM(SELECT a AS a,MAX(b+c)AS b FROM t1 WHERE d>1)AS tt1.

The column operator blood relationship model is to output a set of operators and virtual columns from the input physical columns every time a layer of subquery is passed, in which the operators are output to the virtual columns, and the operators of the outermost query are directly output to the output The physical column itself is no longer output to the virtual column. The link of the column operator lineage model is as follows:

Input physical column-->(operator-->virtual column)-->(...)-->operator-->output physical column.

That is, Column-->(Operator-->VirtualColumn)-->(...)-->Operator-->Column.

Please refer to Figure 8, S5) Traversing the lineage of operators to construct a point-edge relationship, and storing the lineage of column operators into the graph database.

In the graph database, create corresponding physical columns, operators, and virtual columns of three types of points, traverse the column-operator lineage model obtained in S4), construct corresponding point-edge relationships, and store them in the graph database:

INSERT INTO t2(a,b)SELECT tt1.a AS a,tt1.b AS b FROM(SELECT a AS a,MAX(b+c)AS b FROM t1 WHERE d>1)AS tt1.

Fig. 8 is a schematic diagram of the column operator lineage of the SQL stored in the graph database.

Example 2

The present invention provides a server, including a memory, a processor, and a computer program stored on the memory and operable on the processor. The processor implements the steps of the method in Embodiment 1 when executing the program.

Example 3

The present invention provides a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor as the steps of the method described in Embodiment 1.

Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, devices, or computer program products. Accordingly, embodiments of the invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The embodiments described above are only preferred embodiments, and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing examples, those skilled in the art can still use the technology described in the foregoing embodiments Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims (6)

1. The method for constituting the column operator consanguinity, comprising the following steps, S1) parse SQL through Antlr to generate a parsing tree ParseTree; S2) Design the abstract syntax tree AST, which is used to construct the lineage link from the input column to the output column and the processing logic in the middle; S3) recursively traverse the parse tree ParseTree that S1) obtains and build the abstract syntax tree AST of S2) design; S4) Traversing the abstract syntax tree AST constructed in S3) and extracting the column operator consanguinity model; S5) Traversing the lineage model of the column operator and constructing a point-edge relationship, and storing the lineage of the column operator into the graph database. 2. The column operator consanguinity construction method as claimed in claim 1, characterized in that, S2) comprises the steps of: S21) Based on relational algebra, a complete SQL is divided into at least one paragraph, each paragraph is a trunk, and a tree structure Scope is designed to abstract the corresponding trunk of SQL, and establish a hierarchical relationship between the trunk and the trunk; S22) Scope is divided into two types: input type Scope and output type Scope; Among them: the input type Scope is divided into ProjectScope, JoinScope, UnionScope and ScanScope, which are used to abstract the input part of the SQL statement; The output type Scope is divided into CreateAsScope and InsertScope, which are used to abstract the output part of the SQL statement; Among them: each Scope includes the type of the Scope, the alias of the Scope, and the fields exposed to the outside of the Scope. The parent-child relationship between different Scopes is recorded through the index, and the outer backbone set of the Scope is set to parentScopeList, which is the parent Scope set; set The inner backbone collection of Scope is childrenScopeList, which is the collection of child Scopes, so as to build the parent-child relationship between different Scopes; S23) define the 6 kinds of Scopes divided by S22), which are used to correspond to the abstract syntax tree AST; S24) Define the externally exposed fields of each Scope as the holding field set of the Scope, and the holding field set of each Scope can be referenced by the parent Scope in its parentScopeList; S25) Adding an origin attribute ExpressionOrigin to each holding field in the holding field set to record the source information of the holding field, which is used to build a lineage link between the input column and the output column; S26) Establish an expression for recording the processing logic of the column and the source of the column; S27) Each Scope uses the field information of the child Scope in its childrenScopeList to fill the field information of the current Scope, and then transfers the field information of the current Scope to the parent Scope, thereby constructing the processing chain between the output column and the source column road, and then extract the lineage information of the column operator. 3. The column operator consanguinity construction method as claimed in claim 1, characterized in that, S3) comprises the steps of: S31) recursively traverse the parse tree ParseTree generated by S1) from the root node; S32) interpret the root node as CreateAsScope or InsertScope according to the type of SQL, and then traverse the child nodes; S33) If the node type is the query node select xx from, it is interpreted as ProjectScope, and the expression after SELECT, WHERE, GROUP BY, HAVING, ORDER BY is constructed; the child node after its FROM is parsed as the corresponding Scope, and Add it to the childScopeList of the current ProjectScope, wait for the field information of the Scope in the childScopeList to fill in the source information of the expression of the current ProjectScope, and then use the expression information to fill the field of the current ProjectScope; S34) If the child node after FROM is a physical table node, then create a ScanScope and set it as the current Scope, and fill in the holding field of the current Scope according to the metadata information; If the sub-node after FROM is a sub-query node, then skip to S33) for recursive processing; If the child node after FROM contains more than one JOIN node, then create a JoinScope and set it as the current Scope, construct all the expressions after the ON condition, and skip the JoinItem on both sides of each JOIN one by one to S33) for recursive processing; finally The holding field of JoinScope is constructed in the way of widening the holding field of each child Scope in childScopeList, and fills in the source information of all ON expressions; If the child node after FROM contains more than one UNION node, create a UnionScope and set it as the current Scope, and jump the UnionItems on both sides of each UNION to S33) for recursive processing; finally, the holding field of the UnionScope is based on each child Scope in the childScopeList Constructed in the way of superposition of holding fields; S35) Traverse the parse tree ParseTree through S32)-S34), and construct the abstract syntax tree AST designed by S2). 4. The column operator consanguinity construction method as claimed in claim 1, characterized in that, S4) comprises the steps of: S41) Extracting information from Scope to build a column operator lineage model, and dividing the column operator lineage model into operator Operator, physical column Column, and virtual column VirtualColumn; Divide operators into six types, including: SELECT operator: the content of each projection item, that is, the content before AS after SELECT, is extracted from ProjectScope; WHERE operator: the content after WHERE is extracted from ProjectScope; GROUP operator: the content after GROUP BY is extracted from ProjectScope; HAVING operator: the content after HAVING is extracted from ProjectScope; JOIN operator: the content after the ON condition is extracted from JoinScope; UNION operator: used to aggregate multiple SELECT statements, extracted from UnionScope; Among them, the SELECT and UNION operators belong to the direct lineage, and are used to trace the processing logic and processing link of the column, and the WHERE, GROUP, HAVING, and JOIN operators belong to the indirect lineage, and are used to analyze the influence surface of the column; The physical column Column includes the real input physical column and output physical column, which belongs to metadata information. The input physical column is extracted from ScanScope, and the output physical column is extracted from CreateAsScope or InsertScope; Virtual column VirtualColumn: the alias part in SELECTitem AS alias in the subquery, extracted from ProjectScope. 5. A server, characterized in that it comprises a memory, a processor, and a computer program stored on the memory and operable on the processor, when the processor executes the program, it realizes any one of claims 1-4. steps of the method described above. 6. A computer-readable storage medium, wherein a computer program is stored thereon, and when the program is executed by a processor, the steps of the method according to any one of claims 1-4 are realized.

Images