CN113515563A

CN113515563A - Data docking method, database, system and computer-readable storage medium

Info

Publication number: CN113515563A
Application number: CN202110352154.7A
Authority: CN
Inventors: 韩海豹; 周明伟
Original assignee: Zhejiang Dahua Technology Co Ltd
Current assignee: Zhejiang Dahua Technology Co Ltd
Priority date: 2021-03-31
Filing date: 2021-03-31
Publication date: 2021-10-19
Anticipated expiration: 2041-03-31
Also published as: CN113515563B

Abstract

The application discloses a data docking method, a database, a system and a computer readable storage medium, wherein the method is applied to a large-scale parallel database, and comprises the following steps: creating a plurality of transmission pipelines; leading the data export field in the large-scale parallel database into a corresponding transmission pipeline in parallel so as to lead the data docking field into a data docking platform through the corresponding transmission pipeline; or receiving the data export fields in the data docking platform in parallel through a plurality of transmission pipelines, wherein the number of the plurality of transmission pipelines is matched with the number of the data export fields. Through the mode, the data docking speed can be increased.

Description

Data docking method, database, system and computer-readable storage medium

Technical Field

The present application relates to the field of big data technologies, and in particular, to a data docking method, a database, a system, and a computer-readable storage medium.

Background

The current internet products have many requirements, different service requirements are different, and different service data are stored in different databases; when there is a service that needs to refer to data in other databases, data consumption needs to be performed, and different databases perform data query, entry or data import/export, but the currently adopted data docking scheme has a weak capability of processing data.

Disclosure of Invention

The application provides a data docking method, a database, a system and a computer-readable storage medium, which can improve the speed of data docking.

In order to solve the technical problem, the technical scheme adopted by the application is as follows: the data docking method based on the massively parallel database is applied to the massively parallel database and comprises the following steps: creating a plurality of transmission pipelines; leading the data export field in the large-scale parallel database into a corresponding transmission pipeline in parallel so as to lead the data docking field into a data docking platform through the corresponding transmission pipeline; or receiving the data export fields in the data docking platform in parallel through a plurality of transmission pipelines, wherein the number of the plurality of transmission pipelines is matched with the number of the data export fields.

In order to solve the above technical problem, another technical solution adopted by the present application is: there is provided a massively parallel database comprising a memory and a processor connected to each other, wherein the memory is used for storing a computer program, and the computer program is used for implementing the massively parallel database-based data interfacing method in the above technical solutions when being executed by the processor.

In order to solve the above technical problem, another technical solution adopted by the present application is: the data docking system comprises a large-scale parallel database and a data docking platform which are connected with each other, wherein the large-scale parallel database is used for sending data to the data docking platform or receiving the data sent by the data docking platform, and the large-scale parallel database is based on the large-scale parallel database in the technical scheme.

In order to solve the above technical problem, another technical solution adopted by the present application is: there is provided a computer readable storage medium for storing a computer program, which when executed by a processor is used to implement the massively parallel database-based data docking method in the above technical solution.

Through the scheme, the beneficial effects of the application are that: firstly, a plurality of transmission pipelines are constructed through an external table technology, each transmission pipeline is used as a data transmission channel for connecting a large-scale parallel database and a data docking platform, then the large-scale parallel database can lead the stored data into the data docking platform through each transmission pipeline, or the large-scale parallel database can receive the data led out by the data docking platform through each transmission pipeline; the data derived by parallel transmission through the plurality of transmission pipelines can be transmitted at the same time, so that the consumption capacity of data butt joint can be greatly improved, the speed of data butt joint is increased, and the time spent on data transmission is shortened.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts. Wherein:

FIG. 1 is a schematic flow chart diagram illustrating an embodiment of a massively parallel database-based data docking method provided by the present application;

FIG. 2 is a schematic flow chart diagram illustrating another embodiment of a massively parallel database-based data docking method provided by the present application;

FIG. 3 is a schematic diagram of a data export field transmitted to a transmission pipeline provided by the present application;

FIG. 4 is a schematic diagram of an export process and an import process provided herein;

FIG. 5 is a schematic diagram of an embodiment of a massively parallel database provided by the present application;

FIG. 6 is a schematic diagram of an embodiment of a data docking system provided herein;

FIG. 7 is a schematic structural diagram of another embodiment of a data docking system provided herein;

FIG. 8 is a schematic structural diagram of an embodiment of a computer-readable storage medium provided in the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

Referring to fig. 1, fig. 1 is a schematic flowchart of an embodiment of a data docking method based on a Massively Parallel database, where the method is applied to a Massively Parallel Processing (MPP) database, and the method includes:

step 11: a plurality of transport pipes are created.

For data docking, the MPP database may create a plurality of transmission pipelines based on user operations (such as receiving data docking instructions input by a user) so as to perform data interaction with the data docking platform; furthermore, the MPP database can be used as a data import terminal (i.e. one terminal for receiving data) for data consumption, and can also be used as a data export terminal (i.e. one terminal for sending data) for data production; similarly, a data docking platform (i.e., a third party platform) may perform data consumption as a data importing terminal and may also perform data production as a data exporting terminal to implement data transmission between the MPP database and the data docking platform, the data docking platform may be Message Middleware (MQ), KAFKA or Elastic Search (ES), and the MPP database may also be replaced by a greenplus database (GPDB), that is, data interaction between the GPDB database and the MQ, KAFKA or ES is implemented.

Step 12: leading the data export field in the MPP database into a corresponding transmission pipeline in parallel so as to lead the data docking field into a data docking platform through the corresponding transmission pipeline; or receiving the data export field in the data docking platform in parallel through a plurality of transmission pipelines.

When the MPP database is a data export end, the data docking platform is a data import end, the MPP database can send data to be transmitted (namely data export fields) by utilizing a plurality of constructed transmission pipelines, the number of the plurality of transmission pipelines is matched with the number of the data export fields, each transmission pipeline can transmit one data export field, the plurality of data export fields can be transmitted to the data docking platform in parallel through the corresponding transmission pipelines, and the data in the MPP database can be transmitted to the data docking platform; when the MPP database is a data input end, the data docking platform is a data output end and is responsible for outputting a plurality of data output fields, the data output fields output by the data docking platform enter the MPP database through corresponding transmission pipelines, and data in the data docking platform are transmitted to the MPP database.

In this embodiment, a pipeline technique and a multithreading technique are adopted, a plurality of transmission pipelines are constructed as a data transmission channel connecting a data import end and a data export end, and data exported by the data export end can be transmitted in parallel by using the plurality of transmission pipelines, so that the consumption capacity of data docking is greatly improved, and the data docking speed is increased.

Referring to fig. 2, fig. 2 is a schematic flowchart illustrating another embodiment of a data interfacing method based on an MPP database according to the present application, where the method is applied to the MPP database, and the method includes:

step 21: and acquiring first related information of the data export end and second related information of the data import end.

The MPP database acquires information (namely first related information) related to a data export end and information (namely second related information) related to the data import end, wherein the first related information comprises the type of the data export end, the connection information of the data export end, data export filtering conditions, a data export table or a data export format, and the second related information comprises the type of the data export end, the connection information of the data export end or the data import table.

Step 22: and determining the number of the plurality of transmission pipelines by using the first relevant information and the second relevant information.

After the first related information and the second related information are obtained, the first related information and the second related information can be analyzed to obtain the maximum concurrent number of the data export end and the maximum concurrent number of the data import end; then, determining a current maximum concurrency number by using the maximum concurrency number of the data export end, the maximum concurrency number of the data import end and a preset maximum concurrency number, wherein the preset maximum concurrency number can be the maximum concurrency number set in the working pool, and the current maximum concurrency number is the minimum value of the maximum concurrency number of the data export end, the maximum concurrency number of the data import end and the preset maximum concurrency number; and then taking the current maximum concurrency number as the number of the plurality of transmission pipelines.

Step 23: a plurality of transport pipes are created.

Step 23 is the same as step 11 and will not be described herein.

Step 24: leading the data export field in the MPP database into a corresponding transmission pipeline in parallel so as to lead the data docking field into a data docking platform through the corresponding transmission pipeline; or receiving the data export field in the data docking platform in parallel through a plurality of transmission pipelines.

For example, as shown in fig. 3, assuming that the total number of the data export fields is N, the data export fields are respectively recorded as Segment1-Segment N, and the transport pipes are respectively recorded as Pipe1-Pipe, the data export field Segment1 is transmitted through the Pipe1, the data export field Segment2 is transmitted through the Pipe2, and so on, the data export field Segment N is transmitted through the Pipe. Further, a plurality of threads may be set to process the data interfacing task, as shown in fig. 4, the number of the import process and the export process is N, and the export process 1 corresponds to the import process 1 to transmit the data export field Segment 1; export process 2 corresponds to import process 2 for transferring data export field Segment2, and so on, export process N corresponds to import process N for transferring data export field Segment N.

In order to selectively export data instead of only exporting the data in full, the MPP database can obtain preset data export filtering conditions, and then screens a data export table by using the data export filtering conditions to obtain a target export table, wherein the target export table comprises a plurality of data export fields, namely the fields in the target export table are partial fields in the data export table; for example, assuming that the MPP database is a data export terminal, the MPP database stores a data export table, which is as follows:

age (age)	Height of a person	Sex
			A1	B1	C1
A2	B2	C2
			A3	B3	C3

If the data derivation filtering condition is age and gender, the MPP database derives a target derivation table as follows:

age (age)	Sex
		A1	C1
A2	C2
		A3	C3

In a specific embodiment, the data export terminal is an MPP database, and the data import terminal is a data docking platform, and the first external table may be created based on the data export field and the data export field may be inserted into the first external table; and then, parallelly leading each data leading-out field in the first external table into a corresponding transmission pipeline, and transmitting the data leading-out field in the transmission pipeline to the data docking platform through each transmission pipeline.

In another specific embodiment, the data export terminal is a data docking platform, and the data import terminal is an MPP database, and can acquire a data import field and create a second external table by using the data import field; inserting the data export field in each transmission pipeline into a second external table; and then inserting the data export field in the second external table into the data import table to realize importing the data export field in the data docking platform into the MPP database.

In another specific embodiment, the MPP database is provided with a format conversion plug-in, and the MPP database can determine whether the format of the data export field is the same as a preset format, wherein the preset format is a format required by the data import end; if the format of the data export field is different from the preset format, the format conversion plug-in is called to convert the format of the data export field into the preset format, and the requirement of flexibly connecting different data formats is met.

In other specific embodiments, after the data docking task is completed, the plurality of transmission pipelines and the first external meter/the second external meter may be cleaned; whether the number of fields exported from the data export end is the same as the number of fields received by the data import end can also be judged; if the number of the fields exported from the data export end is different from the number of the fields received by the data import end, the abnormal docking is indicated, the data volume received by the data import end is different from the data volume sent by the data export end, and the abnormal docking can be displayed so as to remind a worker to overhaul in time; if the number of the fields exported from the data export end is the same as that of the fields received by the data import end, the data volume transmitted by the data docking is consistent, and the condition of executing the data docking task again is met.

The scheme adopted by the embodiment can carry out data processing on a plurality of platforms, and broadens the application scene of data docking; the import/export of distributed docking data is realized, the actual concurrency is controllable, the number of data fields transmitted at one time can be adaptively adjusted based on different data docking platforms, and the data transmission efficiency is improved; the data import and export format can be appointed, the data fields are converted by adopting an import and export data format processing technology, and different data formats are flexibly docked; in addition, the import and export data can be screened, and different application requirements can be met.

Referring to fig. 5, fig. 5 is a schematic structural diagram of an embodiment of an MPP database provided in the present application, in which the MPP database 50 includes a memory 51 and a processor 52 connected to each other, the memory 51 is used for storing a computer program, and the computer program is used for implementing the data interfacing method in the foregoing embodiment when being executed by the processor 52.

Referring to fig. 6, fig. 6 is a schematic structural diagram of an embodiment of a data docking system provided in the present application, the data docking system 60 includes an MPP database 61 and a data docking platform 62 that are connected to each other, the MPP database 61 is used for sending data to the data docking platform 62 or receiving data sent by the data docking platform 62, where the MPP database 61 is the MPP database in the above embodiment.

The embodiment provides a data processing system for docking multiple platforms based on an MPP architecture, and the consumption capability of data docking can be greatly improved based on an external table technology, a pipeline technology and a multithreading technology of a GPDB.

Referring to fig. 7, fig. 7 is a schematic structural diagram of another embodiment of the data docking system provided in the present application, the data docking system 70 includes an MPP database 71 and a data docking platform 72 that are connected to each other, the MPP database 71 includes a parameter configuration module 711, a management and control module 712, and an import/export data format module 713.

In order to determine the relevant information of the data export terminal and the data import terminal (not shown in the figure), the parameter configuration module 711 may be used to perform parameter configuration to generate corresponding configuration information, where the configuration of the data export terminal includes the type of the data export terminal, the connection information of the data export terminal, the added data export filter condition, the specified data export table, or the specified data export format; the configuration of the data import terminal includes the type of the data export terminal, the connection information of the data export terminal or a specific data import table, and the connection information of the data export terminal and the connection information of the data import terminal may be addresses, ports or information of connected databases.

The management and control module 712 is connected to the parameter configuration module 711, and is configured to analyze the parameter configuration, and analyze the maximum concurrency number that can be supported by the data export terminal, the maximum concurrency number that can be supported by the data import terminal, and the maximum concurrency number set in a working pool (not shown in the figure) to determine the current most reasonable maximum concurrency number (recorded as the current maximum concurrency number).

The management and control module 712 may further create a transmission pipeline 714 according to the current maximum number of concurrencies, start a parallel file service component (e.g., gpfdist) for use by the created external table, and create external tables based on different uses of gpfdist according to the configuration information.

The management and control module 712 may also issue an import-end concurrent task to the data import end, and issue an export-end concurrent task to the data export end, so as to notify the data import end and the data export end to prepare for data docking. After receiving the concurrent task of the importing end, the data importing end may send first state information to the management and control module 712, so as to notify the management and control module 712 that the data docking is ready; after receiving the export concurrent task, the data export terminal may send second state information to the management and control module 712, so as to notify the management and control module 712 that the data docking preparation is already made; it is to be understood that the status information (including the first status information and the second status information) may also be information such as exiting data docking or completion of docking.

Further, a transmission pipeline 714 is used for butting the data import end and the data export end, the control module 712 creates the butted transmission pipeline 714 while issuing a task, when the data import end is ready to consume data, the control module 712 informs the data export end to export the data, and the data is transmitted to the data import end through the transmission pipeline 714 for consuming the data.

After the data export end is started, the prepared state may be sent to the management and control module 712, and after the management and control module 712 sends the information that the data import end is prepared, the data export end may export data, and send the streaming data to the transmission pipeline 714 according to the configuration information, so as to be consumed by the data import end.

Further, if the MPP database 71 serves as a data export terminal, it may create a first external table according to a specified data export field, which is of the same type as the data export table; when the data docking task is started, the MPP database 71 executes a data export task, queries data in the data export table, inserts the data into the first external table, and then exports the data in the first external table to the created transmission pipeline 714; after the task is executed, an end state may be sent to the management and control module 712 to close the MPP database 71 connection; taking the GPDB database as an example, since the GPDB database is distributed, the process of querying the data export table to the first external table is performed on the distributed slave nodes, and the process of aggregation processing by the master node is not required, and the process can be directly performed on each slave node. It is understood that the data docking platform may consume the stream data in the transmission pipeline 714 according to the data import flow.

After the management and control module 712 issues the data docking task, the data import port docks the transmission pipeline 714, sends the prepared state to the management and control module 712, and informs that the data import port is ready to consume, the management and control module 712 can inform the data export port to start executing the data docking task, and when the streaming data is sent to the transmission pipeline 714, the data import port continuously consumes the data, so as to import the data to the data storage system.

Further, if the MPP database 71 serves as a data import, a second external table may be created according to the specified import field, where the specified import field and the data import table are of the same type; when the data docking task is started, the MPP database 71 queries the data of the transmission pipeline 714 corresponding to the second external table and inserts the data into the data import table. The same as data export, the operation is also executed in the slave nodes, data do not need to be gathered on the master node and then sent to the slave nodes, the data can be directly processed on each slave node, and the processing speed is improved. It can be understood that the data docking platform only needs to efficiently import data into the transmission pipeline 714 according to the data export process.

In another embodiment, the exported and imported data types may be specified, written in combination with source codes, and the format of data import and export is specified in the MPP database 71, and the import and export data format module 713 may directly import and export the corresponding format according to the specified data format when in use, and specifically may be directly installed in the GPDB database in a plug-in manner to perform format conversion.

In other embodiments, the management and control module 712 may further clear the transmission pipeline 714, the task, and the external table (including the first external table or the second external table) after the data docking task is ended; the management and control module 712 is also responsible for exception handling, and monitors whether an exception occurs, so as to handle the exception in time.

The MPP database provided by the embodiment can be docked with a multi-party platform, and platforms adapted to the whole system can be added into a data docking platform serving as the MPP database; the MPP database sinks data to the slave nodes through an external table technology for data import and export, and data aggregation does not need to be carried out through the master nodes; in addition, format conversion does not need to be carried out in a program, and a format conversion plug-in is added into the GPDB database through a database plug-in technology, so that the data import and export speed is improved; in addition, the export import data can be screened, screening conditions can be configured when the full export data is not needed, the required data can be exported, the operation is flexible, and different application requirements are met.

Referring to fig. 8, fig. 8 is a schematic structural diagram of an embodiment of a computer-readable storage medium 80 provided in the present application, where the computer-readable storage medium 80 is used to store a computer program 81, and when the computer program 81 is executed by a processor, the computer program is used to implement the MPP database-based data interfacing method in the foregoing embodiment.

The computer readable storage medium 80 may be a server, a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various media capable of storing program codes.

In the several embodiments provided in the present application, it should be understood that the disclosed method and apparatus may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of modules or units is merely a logical division, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed.

Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The above description is only an example of the present application and is not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings, or which are directly or indirectly applied to other related technical fields, are intended to be included within the scope of the present application.

Claims

1. A data docking method based on a massively parallel database is applied to the massively parallel database, and the method comprises the following steps:

creating a plurality of transmission pipelines;

parallelly leading the data export field in the massively parallel database into a corresponding transmission pipeline so as to lead the data docking field into a data docking platform through the corresponding transmission pipeline; or

Receiving data export fields in the data docking platform in parallel through the plurality of transmission pipelines, wherein the number of the plurality of transmission pipelines is matched with the number of the data export fields.

2. The massively parallel database-based data docking method according to claim 1, wherein said step of creating a plurality of transport pipes is preceded by the steps of:

acquiring first related information of a data export end and second related information of the data export end, wherein the first related information comprises the type of the data export end, the connection information of the data export end, a data export filtering condition, a data export table or a data export format, and the second related information comprises the type of the data export end, the connection information of the data export end or a data import table;

determining the number of the plurality of transmission pipelines by using the first related information and the second related information.

3. The massively parallel database based data docking method according to claim 2, wherein said step of determining the number of said plurality of transmission pipes using said first correlation information and said second correlation information comprises:

analyzing the first relevant information and the second relevant information to obtain the maximum concurrent number of the data export end and the maximum concurrent number of the data import end;

determining the current maximum concurrency number by using the maximum concurrency number of the data export end, the maximum concurrency number of the data import end and a preset maximum concurrency number;

and taking the current maximum concurrency number as the number of the plurality of transmission pipelines.

4. The massively parallel database-based data docking method according to claim 2, further comprising:

and acquiring the data export filtering condition, and screening the data export table by using the data export screening condition to obtain a target export table, wherein the target export table comprises a plurality of data export fields.

5. The massively parallel database-based data docking method according to claim 2, wherein the data export is the massively parallel database and the data import is the data docking platform, the method further comprising:

creating a first external table based on the data-export field and inserting the data-export field into the first external table;

and importing each data export field in the first external table into the corresponding transmission pipeline in parallel.

6. The massively parallel database-based data docking method according to claim 2, wherein said data export is said data docking platform and said data import is said massively parallel database, said method further comprising:

acquiring a data import field, and creating a second external table by using the data import field;

inserting a data export field in each of the transport pipes into the second external table;

inserting a data export field in the second external table into the data import table.

7. The massively parallel database-based data docking method according to claim 2, wherein said massively parallel database is provided with format conversion plug-ins, said method further comprising:

judging whether the format of the data export field is the same as a preset format or not;

if not, calling the format conversion plug-in to convert the format of the data export field into the preset format.

8. The massively parallel database-based data docking method according to claim 1, further comprising:

after the data docking task is completed, cleaning the plurality of transmission pipelines and the first external meter/the second external meter;

judging whether the number of fields exported from the data export end is the same as the number of fields received by the data import end or not;

if not, displaying the abnormal docking.

9. A massively parallel database, comprising a memory and a processor connected to each other, wherein the memory is configured to store a computer program, which when executed by the processor is configured to implement the data interfacing method of any one of claims 1-8.

10. A data docking system, comprising a massively parallel database and a data docking platform connected with each other, wherein the massively parallel database is used for sending data to the data docking platform or receiving data sent by the data docking platform, and the massively parallel database is the massively parallel database according to any one of claims 1 to 8.

11. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, is configured to implement the massively parallel database based data docking method as claimed in any one of claims 1 to 8.