CN116303853A - A data processing method, device, equipment and storage medium - Google Patents

Abstract

The invention discloses a data processing method, a device, equipment and a storage medium, wherein the method comprises the following steps: acquiring target map data; determining a plurality of operators to be executed based on the target map data; based on operator dependency relations of a plurality of operators to be executed, constructing a directed acyclic graph of the operators to be executed to obtain a target directed acyclic graph corresponding to the operators to be executed; each operator to be executed corresponds to one node in the target directed acyclic graph; determining the execution sequence corresponding to each of a plurality of operators to be executed based on the degree information of each node in the target directed acyclic graph; and processing the target map data based on the execution sequence corresponding to each of the plurality of operators to be executed to obtain a target processing result. The invention completes the construction of the target directed acyclic graph based on a plurality of operators to be executed, determines the execution sequence of the operators to be executed corresponding to each node in the directed acyclic graph, and can improve the efficiency of data processing.

Description

A data processing method, device, equipment and storage medium

technical field

The present invention relates to the field of vehicle intelligent driving, in particular to a data processing method, device, equipment and storage medium.

Background technique

High-precision maps are also known as car-oriented autonomous driving maps. At present, most of them use the crowdsourcing mode to realize the production and update of high-precision maps; Semi-automatic plus human-assisted mapping.

In the cloud preprocessing link, according to different data sources, formats, and types, corresponding operators are used to process the data. This process requires constant updating of parameters, and judging whether other algorithms need to be introduced to optimize the data according to the returned results; using multiple When an operator completes data processing, it is necessary to traverse multiple operators separately, and the execution order of multiple operators is not determined; parameter settings and combinations of operators in different orders affect the results of data preprocessing and Efficiency has a greater impact.

Contents of the invention

In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is to improve the efficiency of data processing.

In order to solve at least one technical problem raised above, the present invention discloses a data processing method, device, device and storage medium.

According to an aspect of the present disclosure, a data processing method is provided, including:

Obtain target map data;

determining a plurality of operators to be executed based on the target map data; the operators to be executed indicate various data processing operations corresponding to the target map data;

Based on the operator dependencies of the plurality of operators to be executed, constructing a directed acyclic graph of the plurality of operators to be executed to obtain a target directed acyclic graph corresponding to the plurality of operators to be executed ;Each operator to be executed corresponds to a node in the target DAG;

Based on the in-degree information of each node in the target DAG, determine the corresponding execution order of the plurality of operators to be executed;

The target map data is processed based on the execution order corresponding to each of the plurality of operators to be executed to obtain a target processing result.

In some possible embodiments, the determining a plurality of operators to be executed based on the target map data includes:

Performing data format detection on the target map data to obtain a format detection result;

When the format detection result indicates that the data format of the target map data is inconsistent with the preset data format, determine a data conversion operator corresponding to the data format of the target map data;

Performing data processing and analysis on the target map data to obtain a data processing and analysis result;

determining a data processing operator corresponding to the target map data based on the data processing analysis result;

determining a data output operator corresponding to the data format of the data conversion operator based on the data format of the data conversion operator;

The plurality of operators to be executed are determined based on the data conversion operator, the data processing operator, and the data output operator.

In some possible embodiments, based on the operator dependencies of the multiple operators to be executed, a directed acyclic graph is constructed for the multiple operators to be executed to obtain Before the target directed acyclic graph corresponding to the operator, the method further includes:

Obtain operator type parameters corresponding to each of the plurality of operators to be executed;

Based on the operator type parameters corresponding to the plurality of operators to be executed, the dependencies among the nodes are determined.

In some possible embodiments, based on the operator dependencies of the multiple operators to be executed, a directed acyclic graph is constructed for the multiple operators to be executed to obtain The target directed acyclic graph corresponding to the operator includes:

Based on the dependency relationship between the nodes, node verification is performed on any two nodes to obtain a node verification result; the node verification result includes a dependency verification result and a graph verification result;

Verifying the connection relationship between the arbitrary two nodes based on the node verification result to obtain the target directed acyclic graph.

In some possible embodiments, based on the operator dependencies of the multiple operators to be executed, a directed acyclic graph is constructed for the multiple operators to be executed to obtain After the target directed acyclic graph corresponding to the operator, the method includes:

Obtaining data operation information corresponding to each node in the target DAG;

Based on the data operation information, an information filling operation is performed on each node in the target DAG to obtain an updated target DAG.

In some possible embodiments, the determining the execution sequence corresponding to each of the plurality of operators to be executed based on the in-degree information of each node in the target DAG includes:

Determining a current node based on the updated target DAG; the current node is a node with a node in-degree value of 0 in the updated target DAG;

Add the current node to the node queue;

Deleting the current node and each outgoing edge of the current node from the current DAG to obtain an updated DAG;

Determining that in the updated directed acyclic graph, the node whose in-degree value is 0 is the current node;

Repeated execution: add the current node to the node queue until it is determined that in the updated directed acyclic graph, the node with a node in-degree value of 0 is the current node, until the current node is the updated In the directed acyclic graph of , the last node whose in-degree value is 0 and whose out-degree value is also 0;

Based on the node queue, an execution sequence corresponding to each of the plurality of operators to be executed is determined.

In some possible embodiments, the processing of the target map data based on the corresponding execution order of the plurality of operators to be executed to obtain a target processing result includes:

Traversing each node based on the execution sequence corresponding to each of the plurality of operators to be executed;

Based on the currently traversed node, performing a data processing operation corresponding to the currently traversed node on the target map data to obtain a data processing result corresponding to the currently traversed node;

The target processing result is generated based on the data processing result corresponding to each node.

According to a second aspect of the present disclosure, a data processing device is provided, including:

A data acquisition module, configured to acquire target map data;

An operator determination module, configured to determine a plurality of operators to be executed based on the target map data; the operators to be executed indicate various data processing operations corresponding to the target map data;

A graph construction module, configured to construct a directed acyclic graph for the plurality of operators to be executed based on the operator dependencies of the plurality of operators to be executed, to obtain the corresponding to the plurality of operators to be executed Target DAG; each operator to be executed corresponds to a node in the target DAG;

A sequence determination module, configured to determine the execution sequence corresponding to each of the plurality of operators to be executed based on the in-degree information of each node in the target DAG;

The data processing module is configured to process the target map data based on the respective execution sequences of the plurality of operators to be executed to obtain a target processing result.

According to a third aspect of the present disclosure, there is provided an electronic device, the device includes a processor and a memory, at least one instruction and at least one program are stored in the memory, and the at least one instruction and the at least one program are controlled by the The above-mentioned processor is loaded and executed to realize the above-mentioned data processing method.

According to a fourth aspect of the present disclosure, a computer storage medium is provided, at least one instruction and at least one program are stored in the computer storage medium, and the at least one instruction and the at least one program are loaded and executed by a processor to Implement the data processing method as described above.

Implement the present invention, have following beneficial effect:

The present invention determines a plurality of corresponding operators to be executed based on the target map data, and constructs a target directed acyclic graph corresponding to the plurality of operators to be executed, so as to determine the in-degree information corresponding to each node in the target directed acyclic graph ; Further, based on the in-degree information of each node in the target directed acyclic graph, the execution sequence of multiple operators to be executed is obtained, and the target map data is processed according to the execution sequence of multiple operators to be executed, which can avoid In the process of processing the target map data, multiple operators to be executed are traversed multiple times, thereby improving the efficiency of determining the operators to be executed; and then on the basis of reasonably determining the execution order of multiple operators based on the directed acyclic graph , which can improve the accuracy of target map data processing.

Description of drawings

In order to illustrate the technical solution of the present invention more clearly, the accompanying drawings that need to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention , for those skilled in the art, other drawings can also be obtained based on these drawings without creative work.

FIG. 1 is a schematic flowchart corresponding to a data processing method provided by an embodiment of the present invention;

FIG. 2 is a schematic flowchart of operator determination corresponding to an embodiment of the present invention;

FIG. 3 is a schematic flow diagram corresponding to determining node dependency provided by an embodiment of the present invention;

Fig. 4 is a schematic flow chart corresponding to the construction of the target directed acyclic graph provided by the embodiment of the present invention;

FIG. 5 is a schematic flow diagram corresponding to information filling provided by an embodiment of the present invention;

FIG. 6 is a schematic flow diagram corresponding to the determination of the execution sequence provided by the embodiment of the present invention;

FIG. 7 is a schematic flow chart corresponding to target map data processing provided by an embodiment of the present invention;

Fig. 8 is a schematic structural diagram of a data processing toolbox corresponding to an embodiment of the present invention;

FIG. 9 is a schematic diagram of the execution sequence of single-pass data cleaning corresponding to an embodiment of the present invention;

FIG. 10 is a schematic diagram of the execution sequence of multi-pass combination fusion corresponding to an embodiment of the present invention;

Fig. 11 is a visualized image corresponding to map data provided by an embodiment of the present invention;

FIG. 12 is a visualized image corresponding to denoised map data provided by an embodiment of the present invention;

Fig. 13 is a visualized image corresponding to the fitted map data provided by an embodiment of the present invention;

Fig. 14 is a visualized image corresponding to fused map data provided by an embodiment of the present invention;

Fig. 15 is a visualized image corresponding to a data processing result provided by an embodiment of the present invention;

FIG. 16 is a schematic structural diagram of a data processing device provided by an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present specification in combination with the drawings in the embodiments of the present specification. Obviously, the described embodiments are only some of the embodiments of the present specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts fall within the protection scope of the present invention.

It should be noted that the terms "first" and "second" in the description and claims of the present invention and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or server comprising a series of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures indicate functionally identical or similar elements. While various aspects of the embodiments are shown in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior or better than other embodiments.

The term "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and there exists alone B these three situations. In addition, the term "at least one" herein means any one of a variety or any combination of at least two of the more, for example, including at least one of A, B, and C, which may mean including from A, Any one or more elements selected from the set formed by B and C.

In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific implementation manners. It will be understood by those skilled in the art that the present disclosure may be practiced without some of the specific details. In some instances, methods, means, components and circuits that are well known to those skilled in the art have not been described in detail so as to obscure the gist of the present disclosure.

Fig. 1 shows a schematic flow chart corresponding to the data processing method provided by the embodiment of the present invention; the execution subject may be any terminal capable of running the data processing method; specifically, any terminal may be a vehicle body controller, a server terminal, and the like.

As shown in Figure 1, a data processing method includes:

Step S101: Obtain target map data;

In the embodiment of the present invention, the target map data may be map data obtained through a crowdsourcing collection mode and used for making or updating a high-precision map.

Step S102: Determine a plurality of operators to be executed based on the target map data; the operators to be executed indicate various data processing operations corresponding to the target map data;

In the embodiment of the present invention, an operator is an algorithm for data processing of the target map data, and different operators correspond to different data processing methods; multiple operators to be executed are determined according to the data format and processing requirements of the target map data , that is, multiple operators to be executed that are suitable for the current target map data can be selected from the operators.

Step S103: Based on the operator dependencies of the plurality of operators to be executed, construct a directed acyclic graph for the plurality of operators to be executed, and obtain target directed graphs corresponding to the plurality of operators to be executed Acyclic graph; each operator to be executed corresponds to a node in the target directed acyclic graph;

In the embodiment of the present invention, the operator dependency refers to the order in which operators are executed. Specifically, if operator A is used to process data, operator B is used to obtain data, and operator C is used to output data, the execution sequence among operator A, operator B, and operator C is operator B- Operator A-Operator C.

In a specific embodiment, each operator to be executed corresponds to a node in the target directed acyclic graph; when constructing a directed acyclic graph for multiple operators to be executed, in addition to considering operator dependencies , it is also necessary to consider whether a directed acyclic graph can be obtained after associating multiple operators to be executed. Specifically, for example, according to the acquired target map data, it is determined that operator 1, operator 2, operator 3, and operator 4 are multiple operators to be executed, and the operator dependency indicates that operator 1 can be connected to operator 2 , operator 2 can be connected to operator 3, and operator 3 can be connected to operator 2 and operator 4; in the directed acyclic graph generated according to operator dependencies, operator 2 and operator 3 form a closed loop, which does not conform to According to the requirement of directed acyclic graph, when constructing a directed acyclic graph, operator 2 can be connected to operator 3, and operator 3 cannot be connected to operator 2.

Step S104: Based on the in-degree information of each node in the target DAG, determine the execution order corresponding to each of the plurality of operators to be executed;

In the embodiment of the present invention, after the target DAG is obtained, the execution sequence of each node in the target DAG needs to be determined, and the execution sequence is calculated according to the in-degree information of each node.

Specifically, the in-degree information refers to the in-degree value corresponding to each node, and for each node in the target DAG, the in-degree value is the number of edges with the current node as the end point.

Step S105: Process the target map data based on the respective execution sequences of the plurality of operators to be executed to obtain a target processing result.

In the embodiment of the present invention, the execution order of each node is the execution order of each operator to be executed, and the target map data is processed according to the execution order of each operator to be executed.

Specifically, the corresponding operators to be executed are executed according to the first-in-first-out order of each node in the node queue, and the current operator to be executed is reflected and analyzed, and then combined with data operation information to realize task scheduling; reflection analysis is used to dynamically obtain or call each Node data information, such as operator type parameters, data operation information, etc. After each operator to be executed completes the task, the memory message monitoring is used to trigger the operator to be executed in the next stage to perform the corresponding data processing operation, and the data processing result of the previous stage is used as the input of the next stage, and finally the target processing result is obtained.

In the embodiment of the present invention, the corresponding multiple operators to be executed are determined based on the target map data, and the target directed acyclic graph corresponding to the multiple operators to be executed is constructed to determine the corresponding In-degree information; further, based on the in-degree information of each node in the target DAG, the execution sequence of multiple operators to be executed can be obtained, and the target map data is processed according to the execution sequence of multiple operators to be executed Data processing can avoid traversing multiple operators to be executed multiple times in the process of processing target map data, thereby improving the efficiency of operator determination to be executed; Based on the execution order, the accuracy of processing the target map data can be improved.

Fig. 2 shows a schematic flow diagram corresponding to operator determination provided by an embodiment of the present invention. As shown in Fig. 2, the determination of multiple operators to be executed based on the target map data includes:

Step S201: Perform data format detection on the target map data to obtain a format detection result;

In the embodiment of the present invention, the data processing toolbox includes a data source operator set, a data processing operator set, and a data output operator set. The data input operator set, the data processing operator set, and the data output operator set are respectively Corresponds to an operator type, and each operator type includes multiple operators. Therefore, the determination of the operator to be executed starts from data output and ends with data output.

Specifically, after the target map data is acquired, the data format detection is first performed on the data format of the target map data to determine whether the data format of the target map data is compatible with the data format provided by the data source operator set.

Step S202: If the format detection result indicates that the data format of the target map data is inconsistent with the preset data format, determine a data conversion operator corresponding to the data format of the target map data;

In the embodiment of the present invention, it is determined whether the target map data needs format conversion based on the format detection result, and if the data format of the target map data is inconsistent with the preset data format, a data conversion operator capable of converting the data format of the target map data is determined ; If the data format of the target map data is consistent with the preset data format, there is no need to determine the operator from the data source operator set.

In another specific embodiment, the data format of the target map data can be directly converted into a format compatible with the data processing toolbox. That is, for target map data in any format, based on the data source operator set in the data processing toolbox, convert the data format of the target map data into geojson format.

In another specific embodiment, if there is no data conversion operator corresponding to the data format of the target map data in the data source operator set, a data conversion operator corresponding to the data format can be developed according to the data format of the target map data, And save it to the data source operator set.

Step S203: Perform data processing and analysis on the target map data to obtain a data processing and analysis result;

In the embodiment of the present invention, the data processing and analysis results are obtained by analyzing the target map data, so as to determine the data processing operations required for the target map data.

Step S204: Based on the data processing and analysis results, determine a data processing operator corresponding to the target map data;

In the embodiment of the present invention, according to the data processing analysis result, the operator to be executed corresponding to the data processing operation required for the target map data is determined in the data processing operator set. According to different target map data, the operators to be executed determined from the data processing operator set may include one or more.

Step S205: Based on the data format of the data conversion operator, determine a data output operator corresponding to the data format of the data conversion operator;

In the embodiment of the present invention, the data format stipulated by the OGC (Open Geospatial Consortium, Open Geospatial Information Consortium) standard is used as the unified vector element data transmission format in the data processing toolbox, so as to shield the differences when the operators are connected. Therefore, according to the actual application of the target map data, it is necessary to convert the data format when outputting the processing results. The corresponding data format of the child is the same.

In another embodiment, the data output format may also be determined according to the data format corresponding to the storage layer storing the output data, that is, the data format corresponding to the data output operator is consistent with the data format corresponding to the storage layer.

Step S206: Based on the data conversion operator, the data processing operator, and the data output operator, determine the plurality of operators to be executed.

In the embodiment of the present invention, a data conversion operator, a data processing operator, and a data output operator determined according to the target map data, that is, a plurality of operators to be executed.

In the embodiment of the present invention, by unifying the data format of the target map data in the data processing toolbox, the difference when each operator is connected can be shielded; through the data format conversion, it can be compatible with target map data in different data formats, thereby improving The data processing toolbox is widely used; determining the corresponding multiple operators to be executed according to the target map data can improve the accuracy of the target map data processing.

Fig. 3 shows a schematic flow diagram corresponding to the node dependency determination provided by the embodiment of the present invention. As shown in Fig. 3 , based on the operator dependencies of the multiple operators to be executed, the Before constructing a directed acyclic graph and obtaining the target directed acyclic graph corresponding to the plurality of operators to be executed, the method also includes:

Step S301: Obtain operator type parameters corresponding to each of the plurality of operators to be executed;

In the embodiment of the present invention, a graph editing engine is used to construct a directed acyclic graph for multiple operators to be executed. Before constructing a target directed acyclic graph, each node in the target directed acyclic graph needs to be determined.

Specifically, the operator type parameter includes the name of the operator to be executed and the type of the operator to be executed; for example, operator m is used to perform multiple fitting operations on the target map data, and the operator type parameter corresponding to operator m is ( data processing, multi-pass fitting); further, you can set the logo to indicate the name of the operator to be executed and the type of the operator to be executed, for example, 01 represents data processing, and fitting represents multi-pass fitting, then the operator m corresponds to The operator type parameter is (01, fitting).

In the embodiment of the present invention, each node in the target directed acyclic graph corresponds to an operator to be executed, and the number of nodes in the target directed acyclic graph is determined according to the number of operators to be executed; after the number of nodes is determined, It is necessary to correspond the operator type parameter of the operator to be executed with each node, so as to determine the operator to be executed represented by each node in the target DAG.

Step S302: Based on the operator type parameters corresponding to the plurality of operators to be executed, determine the dependencies among the nodes.

In the embodiment of the present invention, the dependency relationship among the nodes is the operator dependency relationship, that is, the order in which operators are executed. Specifically, according to the type of operator parameters, the sequence of operator execution is judged, so as to determine the operator dependency. For example, operator A is used to process data, operator B is used to obtain data, and operator C is used to output data, then the execution sequence among operator A, operator B, and operator C is operator B-operator A -Operator C.

In another specific embodiment, operator A1 is a data processing operator 1, operator A2 is a data processing operator 2, operator B1 is a data conversion operator, operator C1 is a data output operator, and operators A1, The execution sequence among operator A2, operator B1, and operator C1 can be operator B1-operator A1-operator A2-operator C1, or operator B1-operator A2-operator A1-operator Sub-C1, which operator dependency relationship is applicable needs to be determined according to the specific target map data, that is, according to the specific target map data and combined with the construction requirements of the directed acyclic graph to determine the corresponding operator dependency relationship.

In another specific embodiment, the operator dependency can be preset by the user according to the data information contained in the specific target map data.

In the embodiment of the present invention, based on different target map data, corresponding operator dependencies are generated, which can improve the diversity of connections between operators.

Fig. 4 shows a schematic flow diagram corresponding to the construction of the target directed acyclic graph provided by the embodiment of the present invention. As shown in Fig. The operator to be executed carries out the directed acyclic graph construction, and the target directed acyclic graph corresponding to the plurality of operators to be executed is obtained, including:

Step S401: Based on the dependency relationship between the nodes, perform node verification on any two nodes to obtain a node verification result; the node verification result includes a dependency verification result and a graph verification result;

In the embodiment of the present invention, in the process of using the graph editing engine to construct the target directed acyclic graph, the nodes are randomly connected. In addition to the operator dependencies, multiple operator associations to be executed also need to be considered Whether it is possible to obtain a directed acyclic graph. After the nodes are connected, node verification needs to be performed on any two nodes to determine whether the node connection is correct and reasonable.

Further, node verification includes dependency verification and graph verification, and node verification results include dependency verification results and graph verification results. Dependency verification refers to verifying the connection relationship between each two nodes based on the dependency relationship between each node, and generates a dependency verification result; graphic verification refers to the connection relationship between two nodes conforming to each On the basis of the dependencies between nodes, check whether the initial acyclic graph meets the requirements of the directed acyclic graph, and generate a graph verification result.

Step S402: Verify the connection relationship between the arbitrary two nodes based on the node verification result, and obtain the target DAG.

In the embodiment of the present invention, according to the node verification result, if the dependency verification result indicates that the connection relationship between two nodes conforms to the dependency relationship between each node, a graph verification is performed to determine whether the connection between nodes conforms to the Directed acyclic graph; if the graph verification result indicates that the connection relationship between the two nodes conforms to the directed acyclic graph, the node connection is successful.

In another embodiment, if the result of the dependency verification indicates that the connection relationship between the two nodes does not conform to the dependency relationship between the nodes, a new node is selected for connection, and node verification is performed on it.

In another embodiment, if the result of dependency checking indicates that the connection relationship between two nodes conforms to the dependency between each node, but the graph checking result does not meet the requirements of directed acyclic graph, according to the specific The connection relationship of the target map data is corrected, and the node verification is performed again.

In a specific embodiment, specifically, according to the acquired target map data, it is determined that Operator 1, Operator 2, Operator 3, and Operator 4 are multiple operators to be executed, and the operator dependency indicates: Operator 1 can be connected to operator 2, operator 2 can be connected to operator 3, and operator 3 can be connected to operator 2 and operator 4; in the directed acyclic graph generated according to operator dependencies, operator 2 and operator Sub3 forms a closed loop, which does not meet the requirements of a directed acyclic graph. When constructing a directed acyclic graph, operator 2 can be connected to operator 3, but operator 3 cannot be connected to operator 2.

In the embodiment of the present invention, the accuracy of the construction of the target directed acyclic graph can be ensured through node verification, thereby improving: determining the execution order of operators based on the directed acyclic graph, so as to realize the process of processing the target map data. the efficiency of a process.

Fig. 5 is a schematic flow diagram corresponding to information filling provided by an embodiment of the present invention. As shown in Fig. 5, the multiple operators to be executed are effectively executed based on the operator dependencies of the plurality of operators to be executed. After constructing a directed acyclic graph and obtaining the target directed acyclic graph corresponding to the plurality of operators to be executed, the method includes:

Step S501: Obtain data operation information corresponding to each node in the target DAG;

In the embodiment of the present invention, the data operation information includes the entity attribute of the operator to be executed, and the entity attribute is determined based on the parameters of the data processing operation corresponding to the operator to be executed. For example, the data processing operation corresponding to the operator to be executed is distance thinning, and the entity attributes of the node corresponding to the operator to be executed include the set length, etc. The parameter corresponding to the sparse node is 1.

Step S502: Based on the data operation information, perform an information filling operation on each node in the target DAG to obtain an updated target DAG.

In the embodiment of the present invention, in the target directed acyclic graph, each node includes an operator type parameter of the operator to be executed, but does not include the entity attribute of the operator to be executed, so the operator to be executed needs to be The corresponding entity attributes are populated into the nodes.

In the embodiment of the present invention, the data operation information is filled into the target DAG to obtain the updated target DAG, and the operation corresponding to the operator to be executed can be executed directly according to the data operation information filled in the node. Data processing operations, thereby improving the convenience of data processing.

Fig. 6 shows a schematic flow diagram corresponding to the determination of the execution order provided by the embodiment of the present invention. As shown in Fig. 6, the determination of the multiple to-be-executed The corresponding execution order of operators, including:

Step S601: Determine a current node based on the updated target DAG; the current node is a node with a node in-degree value of 0 in the updated target DAG;

In the embodiment of the present invention, the order in which nodes join the node queue is determined according to the in-degree value of the nodes; the point with an in-degree value of 0, that is, the starting point of a directed acyclic graph, is the first step in the whole process. Therefore, in the embodiment of the present invention, after obtaining the updated target DAG, it is necessary to first determine the point whose in-degree value is 0 in the updated target DAG as the current node, so that the current node The node joins the node queue as the starting point for processing the target map data.

Step S602: adding the current node into the node queue;

In the embodiment of the present invention, each current node with an in-degree value of 0 will be added to the node queue. When there are multiple nodes with an in-degree value of 0 at the same time, multiple nodes with an in-degree value of 0 will be randomly added to node queue.

Step S603: deleting the current node and each outgoing edge of the current node from the current DAG to obtain an updated DAG;

Step S604: determining in the updated DAG that the node whose in-degree value is 0 is the current node;

In the embodiment of the present invention, according to the updated DAG, the in-degree value of each node in the graph is recalculated, and the node with an in-degree value of 0 is added to the node queue.

Step S605: judging whether the current node is a node in the updated DAG whose in-degree value is 0 and out-degree value is also 0;

Step S606: If yes, add the current node to the node queue to obtain an updated node queue; determine that the updated node queue is the corresponding execution order of the plurality of operators to be executed;

Step S607: If not, repeat step S602-step S605.

In the embodiment of the present invention, it is necessary to repeatedly execute the above step S602-step S605 to the updated directed acyclic graph, the last node whose in-degree value is 0 and the out-degree value is also 0, to generate: including update The node queue of each node in the subsequent directed acyclic graph, so as to determine the corresponding execution order of multiple operators to be executed according to the first-in-first-out order of the nodes in the node queue. Therefore, it is necessary to judge whether the current node is the last node in the updated DAG whose in-degree value is 0 and whose out-degree value is also 0, so as to determine the next execution step.

In the embodiment of the present invention, in the updated directed acyclic graph, the order in which each node joins the node queue is determined based on the in-degree value of each node, and the order in which each node joins the queue is determined according to the order in which each node joins the queue. Each operator to be executed corresponds to the execution sequence, and the corresponding execution sequence is generated according to different target map data, thereby improving the efficiency of determining the operator to be executed and improving the flexibility of data processing.

Fig. 7 shows a schematic flow chart corresponding to the processing of the target map data provided by the embodiment of the present invention. As shown in Fig. 7, the target map data is processed based on the respective execution sequences of the plurality of operators to be executed. , to get the target processing results, including:

Step S701: Traverse each node based on the corresponding execution order of the plurality of operators to be executed;

In the embodiment of the present invention, each node is traversed according to the first-in-first-out order of each node in the node queue, so as to obtain the currently to-be-executed operator.

Step S702: Based on the currently traversed node, perform a data processing operation corresponding to the currently traversed node on the target map data, and obtain a data processing result corresponding to the currently traversed node;

In the embodiment of the present invention, according to the currently obtained operator to be executed, the current operator to be executed is reflected and analyzed, and the data processing operation corresponding to the current operator to be executed is performed on the target map data according to the filled data operation information, to obtain Data processing results.

Step S703: Generate the target processing result based on the data processing result corresponding to each node.

In the embodiment of the present invention, after each operator to be executed completes the task, the memory message monitoring is used to trigger the operator to be executed in the next stage to perform the corresponding data processing operation, and the data processing result obtained in the previous stage is used as the input of the next stage, and finally Get the target processing result.

In the embodiment of the present invention, the target map data is processed according to the execution order of multiple operators to be executed, and the target processing result is obtained. It is not necessary to traverse multiple operators to be executed when performing each data processing operation, thereby Improve data processing efficiency.

In a specific embodiment, the data processing toolbox is shown in Figure 8, the data toolbox includes a data source operator set, a data processing operator set, and a data output operator set, and each operator set includes multiple The operator of the corresponding type. After obtaining the map data to be processed, determine the multiple operators to be used from the data processing toolbox, such as denoising, coordinate conversion, thinning sampling, distance clustering, and segmental fitting for the map data to be processed. During straight-line and data output operations, the target directed acyclic graph is constructed according to multiple operators to be executed to determine the execution sequence of the operators to be executed. Data processing of the map data to be processed can be triggered by real-time trigger task execution or scheduled task execution; real-time trigger uses Kafka (Apache Kafka, distributed publish and subscribe message system) to monitor and process task topics, and trigger tasks after receiving messages; Task triggering uses timers to set execution time rules.

Figure 9 shows the execution sequence of multiple operators to be executed corresponding to the single-pass data cleaning of the map data to be processed; as shown in Figure 9, the execution sequences of the multiple operators to be executed are: data input (PG format), Coordinate conversion, distance clustering, thinning sampling, denoising, distance clustering, piecewise fitting straight line, and data output (PG format), based on the order of execution, data processing is performed on the map data to be processed.

In another specific embodiment, FIG. 10 shows the execution sequence of multiple operators to be executed corresponding to the combination and fusion of the data to be processed; as shown in FIG. 10 , after the data input (PG format), the map to be processed The results of distance clustering, fitting, and aggregation of the data, and the intersection extraction with the data after the R-tree index of the data to be processed are performed, and smooth curves, topology verification, and data output (PG format) are performed on the extracted map data. Data processing is performed on the map data to be processed based on the execution order.

Fig. 11 is a visualized image corresponding to map data provided by an embodiment of the present invention; Fig. 12 is a visualized image corresponding to denoised map data provided by an embodiment of the present invention; Fig. 13 is provided by an embodiment of the present invention Figure 14 is a visualized image corresponding to the fused map data provided by an embodiment of the present invention; Figure 15 is a visualized image corresponding to the data processing result provided by an embodiment of the present invention .

In another specific embodiment, as shown in Figure 11, after the collected map data is input into the data processing toolbox, the data to be processed is denoised, as shown in Figure 12, the denoised map data is obtained; Perform data fitting processing on the denoised map data, as shown in Figure 13, to obtain the fitted map data; perform fusion processing on the fitted map data, as shown in Figure 14, to obtain the fused map data , the fused data is the final data processing result; the final data processing result is stored in the result database table, and the final data processing result is output, as shown in Figure 15, to obtain a high-speed base map corresponding to the map data.

The embodiment of the present invention also provides a data processing device, as shown in Figure 16, the device includes:

A data acquisition module 1610, configured to acquire target map data;

An operator determination module 1620, configured to determine a plurality of operators to be executed based on the target map data; the operators to be executed indicate various data processing operations corresponding to the target map data;

The graph construction module 1630 is configured to construct a directed acyclic graph for the multiple operators to be executed based on the operator dependencies of the multiple operators to be executed, and obtain the graph corresponding to the multiple operators to be executed. The target DAG; each operator to be executed corresponds to a node in the target DAG;

A sequence determination module 1640, configured to determine the execution sequence corresponding to each of the plurality of operators to be executed based on the in-degree information of each node in the target DAG;

The data processing module 1650 is configured to process the target map data based on the respective execution sequences of the plurality of operators to be executed to obtain a target processing result.

In other embodiments, the data acquisition module 1610 also includes:

A format detection module, configured to perform data format detection on the target map data to obtain a format detection result;

The first operator determination module is configured to determine a data conversion operator corresponding to the data format of the target map data when the format detection result indicates that the data format of the target map data is inconsistent with the preset data format. son;

A data analysis module, configured to perform data processing and analysis on the target map data to obtain data processing and analysis results;

A second operator determination module, configured to determine a data processing operator corresponding to the target map data based on the data processing analysis result;

A third operator determination module, configured to determine a data output operator corresponding to the data format of the data conversion operator based on the data format of the data conversion operator;

A total operator determination module, configured to determine the plurality of operators to be executed based on the data conversion operator, the data processing operator, and the data output operator.

In other embodiments, the device also includes:

A parameter determination module, configured to obtain operator type parameters corresponding to the plurality of operators to be executed;

The dependency determination module is configured to determine the dependency between the nodes based on the operator type parameters corresponding to the plurality of operators to be executed.

In some other embodiments, the graph construction module 1630 also includes:

A node verification module, configured to perform node verification on any two nodes based on the dependencies between the nodes, to obtain a node verification result; the node verification result includes a dependency verification result and a graphic verification result;

A graph generating module, configured to verify the connection relationship between any two nodes based on the node verification result, and obtain the target directed acyclic graph.

In other embodiments, the device also includes:

An information acquisition module, configured to acquire data operation information corresponding to each node in the target DAG;

An information filling module, configured to perform an information filling operation on each node in the target DAG based on the data operation information, to obtain an updated target DAG.

In some other embodiments, the order determining module 1640 also includes:

The first current node determination module: used to determine the current node based on the updated target DAG; the current node is a node with a node in-degree value of 0 in the updated target DAG ;

A node queue determination module, configured to add the current node to the node queue;

A graph update module, configured to delete the current node and each outgoing edge of the current node from the current DAG to obtain an updated DAG;

The second current node determination module: used to determine the updated directed acyclic graph, the node whose in-degree value is 0 is the current node;

The repeated execution module is used for repeated execution: add the current node to the node queue until it is determined that the node with a node in-degree value of 0 is the current node in the updated directed acyclic graph, until the current node The node is the node in the updated directed acyclic graph, the last node whose in-degree value is 0 and whose out-degree value is also 0;

An execution order generating module, configured to determine the execution order corresponding to each of the plurality of operators to be executed based on the node queue.

In some other embodiments, the data processing module 1160 also includes:

A node traversal module, configured to traverse each node based on the corresponding execution order of the plurality of operators to be executed;

A data processing execution module, configured to perform a data processing operation corresponding to the currently traversed node on the target map data based on the currently traversed node, and obtain a data processing result corresponding to the currently traversed node;

A target result generating module, configured to generate the target processing result based on the data processing results corresponding to each node.

The device and method embodiments in the device embodiments described above are based on the same inventive concept and are used to implement the above data processing method.

An embodiment of the present invention also provides a data processing device, the device includes: a processor and a memory, the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the The at least one program, the code set or the instruction set is loaded and executed by the processor to implement the data processing method in the method embodiment.

An embodiment of the present invention also provides a storage medium, which can be set in a server to store at least one instruction, at least one program, code set or instruction set for implementing a data management method in the method embodiment , the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the data processing method as described in any one of the method embodiments.

Optionally, in the embodiment of the present invention, the above-mentioned storage medium may be located in at least one network server among multiple network servers of the computer network. Optionally, in an embodiment of the present invention, the above-mentioned storage medium may include but not limited to: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random AccessMemory), mobile hard disk, Various media that can store program codes, such as magnetic disks or optical disks.

It can be seen from the embodiments provided by the present invention above that the present invention determines the corresponding multiple operators to be executed based on the target map data, and constructs a target directed acyclic graph corresponding to the multiple operators to be executed, so as to determine the target directed acyclic graph. The in-degree information corresponding to each node in the ring graph; further, based on the in-degree information of each node in the target directed acyclic graph, the execution sequence of multiple operators to be executed is obtained, and according to the execution sequence of multiple operators to be executed Data processing of the target map data can avoid traversing multiple operators to be executed multiple times in the process of processing the target map data, thereby improving the efficiency of determining the operator to be executed; Based on the execution sequence of multiple operators, the accuracy of processing target map data can be improved.

It should be noted that: the various embodiments of the present disclosure have been described above, and the above description is illustrative rather than exhaustive, and is not limited to the disclosed embodiments. Many modifications and alterations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principle of each embodiment, practical application or technical improvement in the market, or to enable other ordinary skilled in the art to understand each embodiment disclosed herein.

Claims (10)

1. A data processing method, characterized in that the method comprises: Obtain target map data; determining a plurality of operators to be executed based on the target map data; the operators to be executed indicate various data processing operations corresponding to the target map data; Based on the operator dependencies of the plurality of operators to be executed, constructing a directed acyclic graph of the plurality of operators to be executed to obtain a target directed acyclic graph corresponding to the plurality of operators to be executed ;Each operator to be executed corresponds to a node in the target DAG; Based on the in-degree information of each node in the target DAG, determine the corresponding execution order of the plurality of operators to be executed; The target map data is processed based on the execution order corresponding to each of the plurality of operators to be executed to obtain a target processing result. 2. A data processing method according to claim 1, wherein said determining a plurality of operators to be executed based on said target map data comprises: Performing data format detection on the target map data to obtain a format detection result; When the format detection result indicates that the data format of the target map data is inconsistent with the preset data format, determine a data conversion operator corresponding to the data format of the target map data; Performing data processing and analysis on the target map data to obtain a data processing and analysis result; determining a data processing operator corresponding to the target map data based on the data processing analysis result; determining a data output operator corresponding to the data format of the data conversion operator based on the data format of the data conversion operator; The plurality of operators to be executed are determined based on the data conversion operator, the data processing operator, and the data output operator. 3. A data processing method according to claim 1, characterized in that, based on the operator dependencies of the plurality of operators to be executed, performing directed acyclic processing on the plurality of operators to be executed Graph construction, before obtaining the target directed acyclic graph corresponding to the plurality of operators to be executed, the method also includes: Obtain operator type parameters corresponding to each of the plurality of operators to be executed; Based on the operator type parameters corresponding to the plurality of operators to be executed, the dependencies among the nodes are determined. 4. A data processing method according to claim 3, characterized in that, based on the operator dependencies of the plurality of operators to be executed, performing directed acyclic processing on the plurality of operators to be executed Graph construction, to obtain the target directed acyclic graph corresponding to the plurality of operators to be executed, including: Based on the dependency relationship between the nodes, node verification is performed on any two nodes to obtain a node verification result; the node verification result includes a dependency verification result and a graph verification result; Verifying the connection relationship between the arbitrary two nodes based on the node verification result to obtain the target directed acyclic graph. 5. A data processing method according to claim 1, characterized in that, based on the operator dependencies of the plurality of operators to be executed, performing directed acyclic processing on the plurality of operators to be executed Graph construction, after obtaining the target directed acyclic graph corresponding to the plurality of operators to be executed, the method includes: Obtaining data operation information corresponding to each node in the target DAG; Based on the data operation information, an information filling operation is performed on each node in the target DAG to obtain an updated target DAG. 6. A kind of data processing method according to claim 1, it is characterized in that, described based on the in-degree information of each node in the target directed acyclic graph, determine the number corresponding to each of the plurality of operators to be executed Execution sequence, including: Determining a current node based on the updated target DAG; the current node is a node with a node in-degree value of 0 in the updated target DAG; Add the current node to the node queue; Deleting the current node and each outgoing edge of the current node from the current DAG to obtain an updated DAG; Determining that in the updated directed acyclic graph, the node whose in-degree value is 0 is the current node; Repeated execution: add the current node to the node queue until it is determined that in the updated directed acyclic graph, the node with a node in-degree value of 0 is the current node, until the current node is the updated In the directed acyclic graph of , the last node has an in-degree value of 0 and an out-degree value of 0; Based on the node queue, an execution sequence corresponding to each of the plurality of operators to be executed is determined. 7. A data processing method according to claim 1, wherein said target map data is processed based on the corresponding execution order of said plurality of operators to be executed to obtain a target processing result, include: Traversing each node based on the execution sequence corresponding to each of the plurality of operators to be executed; Based on the currently traversed node, performing a data processing operation corresponding to the currently traversed node on the target map data to obtain a data processing result corresponding to the currently traversed node; The target processing result is generated based on the data processing result corresponding to each node. 8. A data processing device, characterized in that the device comprises: A data acquisition module, configured to acquire target map data; An operator determination module, configured to determine a plurality of operators to be executed based on the target map data; the operators to be executed indicate various data processing operations corresponding to the target map data; A graph construction module, configured to construct a directed acyclic graph for the plurality of operators to be executed based on the operator dependencies of the plurality of operators to be executed, to obtain the corresponding to the plurality of operators to be executed Target DAG; each operator to be executed corresponds to a node in the target DAG; A sequence determination module, configured to determine the execution sequence corresponding to each of the plurality of operators to be executed based on the in-degree information of each node in the target DAG; The data processing module is configured to process the target map data based on the respective execution sequences of the plurality of operators to be executed to obtain a target processing result. 9. An electronic device, the device comprising a processor and a memory, at least one instruction and at least one section of program are stored in the memory, the at least one instruction and the at least one section of program are loaded and executed by the processor to Realize the data processing method described in any one of claims 1-7. 10. A computer storage medium, at least one instruction and at least one section of program are stored in the computer storage medium, and the at least one instruction and the at least one section of program are loaded and executed by a processor to realize the Any one of the data processing methods.

Images