CN115168307A - Data synchronization method, system, device and storage medium supporting resuming transmission from breakpoints - Google Patents

Abstract

The invention relates to a data synchronization method, a system, equipment and a storage medium for supporting breakpoint continuous transmission, wherein the technical scheme is as follows: acquiring log file data of a source database; establishing a table according to the log file data; writing corresponding data nodes in the data of the table according to the data characteristics of the data in the table; transmitting the data written with the data nodes to a target database, judging whether information fed back by the target database and receiving the data nodes is received or not under the condition that the data nodes are transmitted in the process of transmitting the data written with the data nodes to the target database, and storing the position information and the corresponding storage time of the data nodes under the condition that the information fed back by the target database is received; the method and the device have the advantages that after the breakpoint, the data which are not successfully transmitted can be rapidly transmitted again, the data which are transmitted before are prevented from being synchronized again, and the efficiency of data synchronization is greatly improved.

Description

Data synchronization method, system, device and storage medium supporting resuming transmission from breakpoints

technical field

The present invention relates to the technical field of data transmission, and more particularly, to a data synchronization method, system, device and storage medium supporting resumed transmission from a breakpoint.

Background technique

Existing big data platforms usually use offline data synchronization, and there are scenarios in which historical data recalculation is required in each business system and the amount of historical data is huge. It is difficult to define the scope of data synchronization, resulting in a certain error between the historical data stored in the data warehouse and the actual business data, and the accuracy of the data cannot support the business needs in certain scenarios. The higher requirements put forward the requirements for the real-time data of the platform.

SUMMARY OF THE INVENTION

Aiming at the deficiencies of the prior art, the purpose of the present invention is to provide a data synchronization method, system, device and storage medium that support resumed transmission from a breakpoint, which can quickly retransmit data that has not been successfully transmitted after a breakpoint. , the functional advantage of avoiding resynchronization of previously transmitted data.

The above-mentioned technical purpose of the present invention is achieved through the following technical solutions:

A data synchronization method that supports resuming transmission from a breakpoint, comprising:

Get the log file data of the source database;

Build a table according to the log file data;

Write the corresponding data node in the data of the table according to the data characteristics of the data in the table;

Transfer the data written in the data node to the target database, and in the case of transmitting the data node in the process of transmitting the data written in the data node to the target database, determine whether the data that is fed back by the target database is received. The information of the node, in the case of receiving the information fed back by the target database, store the location information of the data node and the corresponding storage time;

Receive a resuming request from a breakpoint, and determine the request time of the resuming request from a breakpoint;

Determine the target data node corresponding to the storage time closest to the time point according to the request time;

The data not transmitted to the target database is determined according to the target data node and the data written to the data node, and the data is continuously transmitted to the target database.

Optionally, the data features include: data peaks and data troughs; and writing corresponding data nodes in the data of the table according to the data features of the data in the table, including:

Determine whether the data in the table has a data peak, and if there is a data peak, write the corresponding data node at the data peak;

It is judged whether the data in the table has a data trough, and if there is a data trough, the corresponding data node is written at the data trough.

Optionally, the data characteristics further include: data consumption time; the data consumption time is determined according to the real-time network transmission speed; the data is written in the table data according to the data characteristics of the table corresponding data node, which also includes:

Determine whether the data consumption time required for data between adjacent data nodes in the table is greater than a preset time threshold, and in the case that the data consumption time required for data between adjacent data nodes is greater than the preset time threshold, Calculate the data length according to the preset time threshold and the real-time network transmission speed, add the data length to the previous data node in the adjacent data nodes to obtain the data writing point, and write the corresponding data at the data writing point node.

Optionally, the building a table according to the log file data includes:

Parse the log file data to obtain database name, table name, operation type, primary key and all field values;

Create a table according to the database name, table name, operation type, primary key and all field values.

Optionally, after creating the table according to the log file data, the method further includes:

dividing the data in the table into a plurality of data rows according to the primary key;

Each data row is divided into a plurality of data blocks according to the preset configuration.

Optionally, the transmitting the data written in the data node to the target database includes:

Read all data blocks in each of the data rows in sequence according to the timestamp, wherein the timestamp is obtained from log file data;

In the case of reading the front end of the data block, mark the first binlog position at the front end of the data block, wherein the first binlog position is the position information of the front end of the data block obtained from the log file data;

When the back end of the data block is read, a second binlog position is marked at the back end of the data block, wherein the second binlog position is the position information of the back end of the data block obtained from the log file data;

The data block marked with the first binlog position and the second binlog position is transmitted to the target data end.

Optionally, also include:

In the case of data changes in the source database during the process of transferring the data written to the data node to the target database, the data row corresponding to the changed data is determined according to the primary key corresponding to the changed data, and the position of the changed data is determined. The data block corresponding to the changed data is determined by comparing with all the first binlog positions and all the second binlog positions of the data row, and the data of the corresponding data block is changed according to the primary key corresponding to the changed data.

A data synchronization system supporting breakpoint resuming, including:

The data acquisition module is used to acquire the log file data of the source database;

a data table building module for building a table according to the log file data;

a node writing module, configured to write corresponding data nodes in the data of the table according to the data characteristics of the data in the table;

The transmission judgment module is used to transmit the data written in the data node to the target database, and in the case of transmitting the data node in the process of transmitting the data written in the data node to the target database, determine whether the target database is received The information of the data node is received in the feedback, and in the case of receiving the information fed back by the target database, the location information of the data node and the corresponding storage time are stored;

a receiving request module, configured to receive a breakpoint resume request, and determine the request time of the breakpoint resume request;

a node determination module, configured to determine the target data node corresponding to the storage time closest to its time point according to the request time;

A data retransmission module, configured to determine the data not transmitted to the target database according to the target data node and the data written in the data node, and continue to transmit the data to the target database.

A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when the processor executes the computer program.

A computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the above-mentioned method.

To sum up, the present invention has the following beneficial effects: after the breakpoint, the data that has not been successfully transmitted can be quickly retransmitted, avoiding the resynchronization of the previously transmitted data, and greatly improving the efficiency of data synchronization; and Compared with dividing the data into fixed segments, the data consumption time is determined by writing the corresponding data nodes at the data peaks and data valleys, and the real-time network transmission speed, and then according to the data consumption time and the preset time threshold. The time of data transmission between adjacent data nodes is controlled within the preset time threshold, so as to realize the division of data according to the characteristics of the data, and it can dynamically decide how much data to process as a unit, which is more suitable for data processing. Real-time synchronous transmission.

Description of drawings

Fig. 1 is the schematic flow chart of the data synchronization method supporting breakpoint resume provided by the present invention;

Fig. 2 is the structural block diagram of the data synchronization system supporting breakpoint resume provided by the present invention;

FIG. 3 is an internal structure diagram of a computer device in an embodiment of the present invention.

Detailed ways

In order to make the objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Several embodiments of the invention are presented in the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein.

In the present invention, unless otherwise expressly specified and limited, the terms "first" and "second" are only used for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. quantity. Thus, a feature defined as "first" or "second" may expressly or implicitly include one or more of that feature.

The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

The present invention provides a data synchronization method supporting breakpoint resuming, as shown in FIG. 1 , including:

Step 100: Obtain the log file data of the source database; specifically, the source database adopts a relational database, and obtains log file data by directly monitoring the binlog log of the relational database (that is, the source database), and binlog is a binary format file , which is used to record the SQL statement information that the user updates to the database;

Step 200, building a table according to the log file data;

Step 300, write a corresponding data node in the data of the table according to the data characteristic of the data in the table;

Step 400: Transmit the data written with the data node to the target database, and in the case of transmitting the data node in the process of transmitting the data written with the data node to the target database, determine whether to receive the feedback of the target database. With the information of the data node, in the case of receiving the information fed back by the target database, it means that the data before the data node has been transmitted to the target database, and the location information of the data node and the corresponding storage time are stored, The storage time is the time point when the location information of the data node is stored;

Step 500: Receive a breakpoint resume request, and determine the request time of the breakpoint resume request;

Step 600: Determine the target data node corresponding to the storage time closest to its time point according to the request time; in practical applications, before the breakpoint occurs, there may be location information corresponding to multiple data nodes that have been stored. Describe the request time of the breakpoint resuming request to determine the data node corresponding to the storage time closest to the request time, that is, to determine the location information of the data node that was last stored before the breakpoint, and use it as the target data node;

Step 700: Determine the data not transmitted to the target database according to the target data node and the data written to the data node, and continue to transmit it to the target database, wherein the data written to the data node specifically refers to the data from the target database. Write the data generated by the corresponding data node in the data of the table according to the data characteristics of the data in the table.

In practical applications, steps 100 to 400 are the normal transmission steps performed when the data transmission is normal, that is, the data transmission is not interrupted due to network interruption or power failure. is the breakpoint) and after restarting the data transmission task, that is, after a request for resuming the transmission from the breakpoint is issued, steps 500 to 700 are executed. After the breakpoint, the data synchronization method of the present application supporting resumed transmission can quickly retransmit the data that has not been successfully transmitted, thereby avoiding resynchronizing the previously transmitted data, and greatly improving the efficiency of data synchronization.

Further, the data characteristics include: data peaks and data troughs; the writing corresponding data nodes in the data of the table according to the data characteristics of the data in the table, including:

Determine whether the data in the table has a data peak, and if there is a data peak, write the corresponding data node at the data peak; specifically, in the case of multiple data peaks, at each data peak Write to its corresponding data node;

Determine whether the data in the table has a data trough, and in the case of a data trough, write the corresponding data node at the data trough; specifically, in the case of multiple data troughs, at each data trough Write to its corresponding data node.

In practical applications, compared to dividing data into fixed segments, by writing corresponding data nodes at data peaks and data troughs, the data can be divided according to the characteristics of the data, which is more suitable for data segmentation. Real-time synchronous transmission.

Further, the data characteristics further include: data consumption time; the data consumption time is determined according to the real-time network transmission speed; the data in the table is written according to the data characteristics of the data in the table corresponding data nodes ,Also includes:

Determine whether the data consumption time required for data between adjacent data nodes in the table is greater than a preset time threshold, and the preset time threshold represents a time period, the data required for data between adjacent data nodes When the consumption time is greater than the preset time threshold, it means that the data consumption time between adjacent data nodes in the data nodes determined according to the data peaks and data troughs is too long. Breakpoints are not conducive to efficient data synchronization. Therefore, the data length is calculated according to the preset time threshold and the real-time network transmission speed, and the data write point is obtained by adding the data length to the previous data node in the adjacent data nodes. , write the corresponding data node at the data write point.

Specifically, the data consumption time is determined according to the real-time network transmission speed, and then the data transmission time between adjacent data nodes is controlled within the preset time threshold according to the data consumption time and the preset time threshold, so that how much data to use can be dynamically determined. The processing is performed as a unit, so that after a data transmission breakpoint occurs, it does not take a long time to retransmit the previously transmitted data from the target data node, which further improves the efficiency of data synchronization.

Further, building a table according to the log file data includes:

Parse the log file data to obtain database name, table name, operation type, primary key and all field values;

Create a table according to the database name, table name, operation type, primary key and all field values.

Further, after creating the table according to the log file data, it also includes:

dividing the data in the table into a plurality of data rows according to the primary key;

Each data row is divided into a plurality of data blocks according to the preset configuration.

In practical applications, the data in the table is divided into multiple data rows according to the primary key, and in the process of data synchronization, the data in the multiple data rows can be transmitted at the same time, so as to realize the multiple concurrent transmission of the data, thereby It can further improve the effect of data synchronous transmission. The preset configuration can be a preset data length, and each data row is divided into multiple data blocks according to the preset configuration. When the data in the source database changes, it can directly The data blocks corresponding to the changed data are revised to realize real-time updating of the data.

Further, the transmitting the data written with the data node to the target database includes:

Read all data blocks in each of the data rows in sequence according to the timestamp, wherein the timestamp is obtained from log file data;

In the case of reading the front end of the data block, mark the first binlog position at the front end of the data block, wherein the first binlog position is the position information of the front end of the data block obtained from the log file data;

When the back end of the data block is read, a second binlog position is marked at the back end of the data block, wherein the second binlog position is the position information of the back end of the data block obtained from the log file data;

The data block marked with the first binlog position and the second binlog position is transmitted to the target data end.

Further, it also includes:

In the case of data changes in the source database in the process of transferring the data written into the data node to the target database, the data changes include: data addition, data deletion and data modification, and the data change is determined according to the primary key corresponding to the changed data. For the data row corresponding to the changed data, compare the position of the changed data with all the first binlog positions and all the second binlog positions of the data row to obtain a comparison result, and determine the data block corresponding to the changed data according to the comparison result, If the comparison result is that the position of the changed data is located between a first binlog position and a second binlog position corresponding to the first binlog position, it is determined that the data block corresponding to the changed data is the first binlog The position and the data block corresponding to the first binlog position are changed according to the primary key corresponding to the changed data. If the changed data is added, deleted or modified, the corresponding data in the database Data is added, deleted or modified accordingly.

Through the data block, the mark of the first binloug position and the mark of the second binlog position, the speed of searching for the position of the changed data in the table is accelerated, which can not only realize the real-time update of the data, but also improve the efficiency of data synchronization.

The data synchronization method of the present invention that supports resumed transmission from a breakpoint can quickly retransmit the data that has not been successfully transmitted after the breakpoint, avoids resynchronizing the previously transmitted data, and greatly improves the efficiency of data synchronization; And compared to dividing the data into fixed segments, the data consumption time is determined by writing the corresponding data nodes at the data peaks and data troughs, and the real-time network transmission speed, and then according to the data consumption time and preset time thresholds. The time of data transmission between adjacent data nodes is controlled within the preset time threshold, so as to realize the division of data according to the characteristics of the data, and it can dynamically decide how much data to process as a unit, which is more suitable for data processing. real-time synchronous transmission.

As shown in Figure 2, the present invention also provides a construction and distribution system for an application installation package, including:

The data acquisition module 10 is used for acquiring the log file data of the source database;

A data table building module 20, for building a table according to the log file data;

A node writing module 30, configured to write corresponding data nodes in the data of the table according to the data characteristics of the data in the table;

The transmission judgment module 40 is used to transmit the data written with the data node to the target database, and in the case of transmitting the data node in the process of transmitting the data written with the data node to the target database, judge whether to receive the target database The information of the data node is received in the database feedback, and in the case of receiving the information fed back by the target database, the location information of the data node and the corresponding storage time are stored;

The receiving request module 50 is used for receiving a request for resuming the transmission from a breakpoint, and determining the request time of the resuming request for the transmission from a breakpoint;

Node determination module 60, configured to determine the target data node corresponding to the storage time closest to its time point according to the request time;

The data retransmission module 70 is configured to determine the data not transmitted to the target database according to the target data node and the data written in the data node, and continue to transmit the data to the target database.

For the specific limitation of the construction and distribution system of the application installation package, reference may be made to the above definition of the construction and distribution method of the application installation package, which will not be repeated here. Each module of the construction and distribution system of the installation package of the above application may be implemented in whole or in part by software, hardware and combinations thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

In one embodiment, a computer device is provided, and the computer device may be a server, and its internal structure diagram may be as shown in FIG. 3 . The computer device includes a processor, memory, a network interface, and a database connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The nonvolatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for constructing and distributing an installation package of an application is implemented.

Those skilled in the art can understand that the structure shown in FIG. 3 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. Include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, comprising a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

Get the log file data of the source database;

Build a table according to the log file data;

Write the corresponding data node in the data of the table according to the data characteristics of the data in the table;

Transfer the data written in the data node to the target database, and in the case of transmitting the data node in the process of transmitting the data written in the data node to the target database, determine whether the data that is fed back by the target database is received. The information of the node, in the case of receiving the information fed back by the target database, store the location information of the data node and the corresponding storage time;

Receive a resuming request from a breakpoint, and determine the request time of the resuming request from a breakpoint;

Determine the target data node corresponding to the storage time closest to the time point according to the request time;

The data not transmitted to the target database is determined according to the target data node and the data written to the data node, and the data is continuously transmitted to the target database.

In one embodiment, the data features include: data peaks and data troughs; and writing corresponding data nodes in the data of the table according to the data features of the data in the table, including:

Determine whether the data in the table has a data peak, and if there is a data peak, write the corresponding data node at the data peak;

It is judged whether the data in the table has a data trough, and if there is a data trough, the corresponding data node is written at the data trough.

In one embodiment, the data characteristic further includes: data consumption time; the data consumption time is determined according to the real-time network transmission speed; the data corresponding to the data in the table is written according to the data characteristic of the data in the table The data node also includes:

Determine whether the data consumption time required for data between adjacent data nodes in the table is greater than a preset time threshold, and in the case that the data consumption time required for data between adjacent data nodes is greater than the preset time threshold, Calculate the data length according to the preset time threshold and the real-time network transmission speed, add the data length to the previous data node in the adjacent data nodes to obtain the data writing point, and write the corresponding data at the data writing point node.

In one embodiment, the building a table according to the log file data includes:

Parse the log file data to obtain database name, table name, operation type, primary key and all field values;

Create a table according to the database name, table name, operation type, primary key and all field values.

In one embodiment, after the creating the table according to the log file data, the method further includes:

dividing the data in the table into a plurality of data rows according to the primary key;

Each data row is divided into a plurality of data blocks according to the preset configuration.

In one embodiment, the transmitting the data written with the data node to the target database includes:

Read all data blocks in each of the data rows in sequence according to the timestamp, wherein the timestamp is obtained from log file data;

In the case of reading the front end of the data block, mark the first binlog position at the front end of the data block, wherein the first binlog position is the position information of the front end of the data block obtained from the log file data;

When the back end of the data block is read, a second binlog position is marked at the back end of the data block, wherein the second binlog position is the position information of the back end of the data block obtained from the log file data;

The data block marked with the first binlog position and the second binlog position is transmitted to the target data end.

In one embodiment, it also includes:

In the case of data changes in the source database during the process of transferring the data written to the data node to the target database, the data row corresponding to the changed data is determined according to the primary key corresponding to the changed data, and the position of the changed data is determined. The data block corresponding to the changed data is determined by comparing with all the first binlog positions and all the second binlog positions of the data row, and the data of the corresponding data block is changed according to the primary key corresponding to the changed data.

In one embodiment, a computer-readable storage medium is provided on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

Get the log file data of the source database;

Build a table according to the log file data;

Write the corresponding data node in the data of the table according to the data characteristics of the data in the table;

Transfer the data written in the data node to the target database, and in the case of transmitting the data node in the process of transmitting the data written in the data node to the target database, determine whether the data that is fed back by the target database is received. The information of the node, in the case of receiving the information fed back by the target database, store the location information of the data node and the corresponding storage time;

Receive a resuming request from a breakpoint, and determine the request time of the resuming request from a breakpoint;

Determine the target data node corresponding to the storage time closest to the time point according to the request time;

The data not transmitted to the target database is determined according to the target data node and the data written to the data node, and the data is continuously transmitted to the target database.

In one embodiment, the data features include: data peaks and data troughs; and writing corresponding data nodes in the data of the table according to the data features of the data in the table, including:

Determine whether the data in the table has a data peak, and if there is a data peak, write the corresponding data node at the data peak;

It is judged whether the data in the table has a data trough, and if there is a data trough, the corresponding data node is written at the data trough.

In one embodiment, the data characteristic further includes: data consumption time; the data consumption time is determined according to the real-time network transmission speed; the data corresponding to the data in the table is written according to the data characteristic of the data in the table The data node also includes:

Determine whether the data consumption time required for data between adjacent data nodes in the table is greater than a preset time threshold, and in the case that the data consumption time required for data between adjacent data nodes is greater than the preset time threshold, Calculate the data length according to the preset time threshold and the real-time network transmission speed, add the data length to the previous data node in the adjacent data nodes to obtain the data writing point, and write the corresponding data at the data writing point node.

In one embodiment, the building a table according to the log file data includes:

Parse the log file data to obtain database name, table name, operation type, primary key and all field values;

Create a table according to the database name, table name, operation type, primary key and all field values.

In one embodiment, after the creating the table according to the log file data, the method further includes:

dividing the data in the table into a plurality of data rows according to the primary key;

Each data row is divided into a plurality of data blocks according to the preset configuration.

In one embodiment, the transmitting the data written with the data node to the target database includes:

Read all data blocks in each of the data rows in sequence according to the timestamp, wherein the timestamp is obtained from log file data;

In the case of reading the front end of the data block, mark the first binlog position at the front end of the data block, wherein the first binlog position is the position information of the front end of the data block obtained from the log file data;

When the back end of the data block is read, a second binlog position is marked at the back end of the data block, wherein the second binlog position is the position information of the back end of the data block obtained from the log file data;

The data block marked with the first binlog position and the second binlog position is transmitted to the target data end.

In one embodiment, it also includes:

In the case of data changes in the source database during the process of transferring the data written to the data node to the target database, the data row corresponding to the changed data is determined according to the primary key corresponding to the changed data, and the position of the changed data is determined. The data block corresponding to the changed data is determined by comparing with all the first binlog positions and all the second binlog positions of the data row, and the data of the corresponding data block is changed according to the primary key corresponding to the changed data.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other medium used in the various embodiments provided in this application may include non-volatile and/or volatile memory. Nonvolatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Road (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features It is considered to be the range described in this specification.

Claims (10)

1. A data synchronization method supporting breakpoint continuous transmission is characterized by comprising the following steps:

acquiring log file data of a source database;

establishing a table according to the log file data;

writing corresponding data nodes in the data of the table according to the data characteristics of the data in the table;

transmitting data written with data nodes to a target database, judging whether information fed back by the target database and receiving the data nodes is received or not under the condition that the data nodes are transmitted in the process of transmitting the data written with the data nodes to the target database, and storing the position information and the corresponding storage time of the data nodes under the condition that the information fed back by the target database is received;

receiving a breakpoint continuous transmission request, and determining the request time of the breakpoint continuous transmission request;

determining a target data node corresponding to the storage time closest to the time point according to the request time;

and determining data which is not transmitted to the target database according to the target data node and the data written with the data node, and continuously transmitting the data to the target database.

2. The data synchronization method supporting breakpoint resume according to claim 1, wherein the data characteristics include: data peaks and data troughs; the writing of the corresponding data node in the data of the table according to the data characteristics of the data in the table includes:

judging whether the data in the table has a data wave crest or not, and writing a corresponding data node at the data wave crest under the condition that the data wave crest exists;

and judging whether the data in the table has data troughs or not, and writing corresponding data nodes in the data troughs under the condition that the data troughs exist.

3. The data synchronization method supporting breakpoint resume according to claim 2, wherein the data characteristics further include: data consumption time; the data consumption time is determined according to the real-time network transmission speed; the writing of the corresponding data node in the data of the table according to the data characteristics of the data in the table further comprises:

judging whether the data consumption time required by the data between the adjacent data nodes in the table is larger than a preset time threshold, calculating to obtain the data length according to the preset time threshold and the real-time network transmission speed under the condition that the data consumption time required by the data between the adjacent data nodes is larger than the preset time threshold, adding the data length to the previous data node in the adjacent data nodes to obtain a data writing point, and writing the data writing point into the corresponding data node.

4. The method for data synchronization supporting breakpoint resume according to claim 1, wherein the creating a table according to the log file data includes:

analyzing the log file data to obtain a database name, a table name, an operation type, a main key and all field values;

and building a table according to the database name, the table name, the operation type, the main key and all field values.

5. The method for data synchronization supporting breakpoint resume according to claim 4, further comprising, after said creating a table according to the log file data:

dividing data in the table into a plurality of data rows according to the primary key;

and dividing each data line into a plurality of data blocks according to the preset configuration.

6. The method for synchronizing data supporting breakpoint resuming according to claim 4, wherein the transmitting the data written with the data node to the target database includes:

sequentially reading all data blocks in each data row according to a time stamp, wherein the time stamp is acquired from log file data;

under the condition that the front end of the data block is read, marking a first binlog position at the front end of the data block, wherein the first binlog position is position information of the front end of the data block acquired from log file data;

under the condition that the rear end of the data block is read, marking a second binlog position at the rear end of the data block, wherein the second binlog position is position information of the rear end of the data block acquired from log file data;

and transmitting the data block marked with the first binlog position and the second binlog position to a target data end.

7. The data synchronization method supporting breakpoint resume according to claim 6, further comprising:

when the data of the source database is changed in the process of transferring the data written into the data node into the target database, determining a data row corresponding to the changed data according to a primary key corresponding to the changed data, comparing the positions of the changed data with all first binlog positions and all second binlog positions of the data row to determine a data block corresponding to the changed data, and changing the data of the data block corresponding to the changed data according to the primary key corresponding to the changed data.

8. A data synchronization system supporting breakpoint resuming, comprising:

the data acquisition module is used for acquiring log file data of a source database;

the data tabulation module is used for tabulating according to the log file data;

the node writing module is used for writing corresponding data nodes in the data of the table according to the data characteristics of the data in the table;

the transmission judging module is used for transmitting the data written with the data nodes to a target database, judging whether information fed back by the target database and receiving the data nodes is received or not under the condition that the data nodes are transmitted in the process of transmitting the data written with the data nodes to the target database, and storing the position information of the data nodes and the corresponding storage time under the condition that the information fed back by the target database is received;

the receiving request module is used for receiving a breakpoint resume request and determining the request time of the breakpoint resume request;

the node determining module is used for determining a target data node corresponding to the storage time closest to the time point according to the request time;

and the data continuous transmission module is used for determining the data which is not transmitted to the target database according to the target data node and the data written with the data node, and continuously transmitting the data to the target database.

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 7.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.

Images