CN115357661A - Database data synchronization method and device, storage medium and processor - Google Patents

Abstract

The application discloses a database data synchronization method and device, a storage medium, and a processor. The method includes: obtaining the data to be synchronized of the source database that needs to be synchronized, the data to be synchronized includes the data generated by the target event; splitting the target event of the data to be synchronized to obtain the first data generated by the first event, the second event Generated second data; perform data synchronization according to the first event and the second event, and synchronize the data to be synchronized to the corresponding target database. The target database is different from the source database, and the target event is executed in a different type of database. The structures are different, and the data structures of the first event and the second event executed in different types of databases are the same. It solves the problem that data synchronization involving distributed databases in related technologies needs to be realized by merging, sorting and aggregation of logs, resulting in poor operability and failure to synchronize in some cases.

Description

Database data synchronization method and device, storage medium, processor

technical field

The present application relates to the field of data synchronization, in particular, to a data synchronization method and device, a storage medium, and a processor for a database.

Background technique

The overall data synchronization architecture of the distributed database consists of the following parts: the source database, which currently supports distributed relational databases, distributed file systems, and unstructured databases. The target database currently supports distributed relational databases, distributed file systems, and unstructured databases. The management node cluster is used for data verification configuration, and the data verification configuration is pushed to the verification node. At the same time, it receives the data synchronization status, progress and other information fed back by the check node. A module that synchronizes the node cluster and executes the specific data verification process. The coordinator cluster is used to coordinate the modules of data verification.

When a distributed database is used as the source to synchronize to a non-distributed database, due to the difference in the synchronization progress of the underlying shard database, when the source data is updated across shards, log aggregation and sorting are required to ensure the final consistency of the data, which consumes a lot of performance and can be Operability is poor. When a non-distributed database is used as the source to synchronize to a distributed database, a certain field is a shard key on the target distributed database. After the source updates the value of this field, the target cannot directly to sync. The data synchronization process cannot be redone. A record at the source has undergone continuous primary key change operations. If the adjustment point is adjusted to a previous point in time and the entire synchronization process is redone, a primary key conflict error will occur.

For data synchronization involving distributed databases in related technologies, logs need to be merged, sorted and aggregated to achieve this, resulting in poor operability and inability to synchronize in some cases. No effective solution has been proposed yet.

Contents of the invention

The main purpose of this application is to provide a database data synchronization method and system to solve the problem of data synchronization involving distributed databases in related technologies. Logs need to be merged, sorted and aggregated to achieve this, resulting in poor operability and in some cases unable to synchronize The problem.

In order to achieve the above object, according to one aspect of the present application, a data synchronization method for a database is provided, including: acquiring data to be synchronized of a source database that needs to be synchronized, wherein the data to be synchronized includes data generated by a target event; The target event of the data to be synchronized is split to obtain first data and second data, wherein the first data is the data generated by the first event, and the second data is the data generated by the second event, so The superposition of the first event and the second event is the same as the target event; data synchronization is performed according to the first event and the second event, and the data to be synchronized is synchronized to the corresponding target database, wherein , the types of the target database and the source database are different, the data structures of the target events executed in different types of databases are different, and the data structures of the first event and the second event executed in different types of databases The structure is the same.

Optionally, performing data synchronization according to the first event and the second event, and synchronizing the data to be synchronized to the corresponding target database includes: according to the types of the source database and the target database, determining the Whether the first data or the second data is modified; if the first data or the second data is modified, according to the modified first data or second data, and the unmodified The second data or the first data, inputting the first event and the second event into the target database, so as to synchronize the data to be synchronized to the target database; and/or, in the first data or if the second data has not been modified, input the first event and the second event into the target database according to the execution attribute of the first data or the second data, so as to input the to-be-synchronized Data is synchronized to the target database.

Optionally, when the source database is a distributed database and the target database is a non-distributed database, it is determined that the first data needs to be modified; modify the first data according to a preset method; The modified first data, and the unmodified second data, input the first event and the second event into the target database, so as to synchronize the data to be synchronized to the target database.

Optionally, when the source database is a non-distributed database and the target database is a distributed database, it is determined that the first data or the second data does not need to be modified; according to the and the first execution attribute of the second event, input the first event and the second event into the target database, so as to synchronize the data to be synchronized to the target database.

Optionally, the method further includes: setting an attribute parameter of the second data, determining a second execution attribute of the second event; combining the first event and the The second event is input to the target database, so as to synchronize the data to be synchronized to the target database.

Optionally, the target event is an update event, the first event is a delete event, and the second event is an insert event.

Optionally, modifying the first data according to a preset method includes: adding a shard key to the search condition in the statement data of the delete event, wherein the shard key is used to identify the storage location of the data and In a modified version, when the first event is a delete event, the first data is the statement data of the delete event; and/or, the first execution attribute is the delete delete event and the insert insert event and/or, the second execution attribute is an attribute of an execution strategy that adopts an execution strategy of inserting if it does not exist and updating if it exists when the insert event is stored into the target database.

In order to achieve the above object, according to another aspect of the present application, a data synchronization device for a database is provided. The device includes: an acquisition module, configured to acquire data to be synchronized from a source database that needs to be synchronized, wherein the data to be synchronized includes data generated by a target event; a splitting module, configured to process the target event of the data to be synchronized Split to obtain first data and second data, wherein, the first data is the data generated by the first event, the second data is the data generated by the second event, the first event and the second The superposition of the event is the same as the target event; the synchronization module is configured to perform data synchronization according to the first event and the second event, and synchronize the data to be synchronized to the corresponding target database, wherein the target The types of the database and the source database are different, the data structures of the target events executed in different types of databases are different, and the data structures of the first event and the second event executed in different types of databases are the same.

According to another aspect of the present application, there is also provided a computer-readable storage medium, where the storage medium is used to store a program, wherein the program executes the data synchronization method for a database described in any one of the above.

According to another aspect of the present application, there is also provided an electronic device, including one or more processors and a memory, the memory is used to store one or more programs, wherein, when the one or more programs are executed When the one or more processors are executed, the one or more processors realize the data synchronization method for the database described in any one of the above.

Through this application, the data to be synchronized of the source database that needs to be synchronized is obtained, wherein the data to be synchronized includes the data generated by the target event; the target event of the data to be synchronized is split to obtain the first data and the second data, wherein, The first data is the data generated by the first event, the second data is the data generated by the second event, the superposition of the first event and the second event is the same as the target event; according to the first event and the second event, data synchronization is performed, and the The data to be synchronized is synchronized to the corresponding target database, wherein the target database is of different types from the source database, the data structure of the target event executed in different types of databases is different, and the first event and the second event executed in different types of databases The data structure is the same.

By acquiring the data to be synchronized generated by the target event, the target event is split into a first event and a second event that can be directly synchronized to the target database, thereby splitting the data to be synchronized into first data and second data, It achieves the purpose of directly splitting and synchronizing the data to be synchronized to the target database of the target event that relies on log merging and aggregation to achieve data synchronization, and reduces the operational difficulty of synchronizing the data to be synchronized at the target time, while avoiding log In the case of merging and aggregation, the technical effect of effectively and accurately realizing data synchronization, and then solving the data synchronization involving distributed databases in related technologies, needs to be realized by merging, sorting and aggregation of logs, resulting in poor operability and in some cases unable to synchronize The problem.

Description of drawings

The drawings constituting a part of the application are used to provide further understanding of the application, and the schematic embodiments and descriptions of the application are used to explain the application, and do not constitute an improper limitation to the application. In the attached picture:

Fig. 1 is a flow chart of a data synchronization method for a database provided according to an embodiment of the present application;

FIG. 2 is a flow chart of data synchronization performed by a distributed database provided by the related technology according to the embodiment of the present application;

FIG. 3 is a flow chart of data synchronization performed by a distributed database provided according to an embodiment of the present application;

FIG. 4 is a schematic diagram of an actual implementation example of data synchronization performed by a distributed database provided according to an embodiment of the present application;

FIG. 5 is a flow chart of data synchronization provided by a non-distributed database as a source according to an embodiment of the present application;

Fig. 6 is a flow chart of repeatable warehousing during data synchronization according to the database provided by the embodiment of the present application;

FIG. 7 is a schematic diagram of a data synchronization device for a database provided according to an embodiment of the present application;

Fig. 8 is a schematic diagram of an electronic device provided according to an embodiment of the present application.

Detailed ways

It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and embodiments.

In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiment of the application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the application. Obviously, the described embodiment is only It is an embodiment of a part of the application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

It should be noted that the terms "first" and "second" in the description and claims of the present application and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It should be understood that the data so used may be interchanged under appropriate circumstances for the embodiments of the application described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

It should be noted that the relevant information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are authorized by the user. Or information and data fully authorized by the parties. For example, there is an interface between this system and relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution. information.

Example

The present invention is described below in conjunction with preferred implementation steps. Fig. 1 is a flow chart of a data synchronization method for a database provided according to an embodiment of the present application. As shown in Fig. 1, the method includes the following steps:

Step S101, obtaining the data to be synchronized of the source database that needs to be synchronized, wherein the data to be synchronized includes the data generated by the target event;

Step S102, splitting the target event of the data to be synchronized to obtain first data and second data, wherein the first data is the data generated by the first event, the second data is the data generated by the second event, the first event and The superposition of the second event is the same as the target event;

Step S103, perform data synchronization according to the first event and the second event, and synchronize the data to be synchronized to the corresponding target database, wherein the target database is of different types from the source database, and the data structure of the target event executed in different types of databases Differently, the data structures executed by the first event and the second event in different types of databases are the same.

Through the above steps, by acquiring the data to be synchronized generated by the target event, the target event is split into a first event and a second event that can be directly synchronized to the target database, so that the data to be synchronized is split into first data and The second data achieves the purpose of directly splitting and synchronizing the data to be synchronized for the target event that relies on log merging and aggregation to achieve data synchronization, and achieves the goal of reducing the operational difficulty of synchronizing data to be synchronized at the target time , in the case of avoiding log merging and aggregation, effectively and accurately realize the technical effect of data synchronization, and then solve the data synchronization involving distributed databases in related technologies, which needs to be realized by merging, sorting and aggregation of logs, resulting in poor operability, and some The problem of not being able to synchronize in the case.

The execution subject of the above steps may be the above-mentioned database, or a functional module for managing the database. The functional module may be set in the above-mentioned source database or the target database, or it may be set separately outside the above-mentioned source database and the target database, or it may be set in Data synchronization between the source database and the target database is managed by establishing a data link with the source database and the target database on a third-party device outside the source database and the target database.

The above data to be synchronized includes when the target event is an update modification event, and when a modification event occurs in the database, its data has changed, and the changed data can be recorded for incremental data synchronization. In distributed databases and non-distributed databases, their data structures are different. For the data synchronization of update events, non-distributed databases cannot clarify the order of their data, while distributed databases can be determined according to the shard key, which leads to There are certain obstacles in the data synchronization of update events between distributed databases and non-distributed databases.

In this embodiment, the update event is split into a first event and a second event, and the superposition of the first event and the second event is the same as the target event, that is, the update event. The above-mentioned first event may be a delete event, and the second event may be an insert event. In distributed databases and non-distributed databases, delete events and insert events can be directly synchronized, and when update events are split, delete events and insert events are in sequence. Therefore, through the sequence of delete Events and insert events can directly synchronize the data of the update event to the target database.

Optionally, performing data synchronization according to the first event and the second event, and synchronizing the data to be synchronized to the corresponding target database includes: determining whether to modify the first data or the second data according to the types of the source database and the target database ; In the case of modifying the first data or the second data, according to the modified first data or second data, and the unmodified second data or first data, input the first event and the second event into The target database, to synchronize the data to be synchronized to the target database; and/or, in the case of no modification to the first data or the second data, according to the execution attribute of the first data or the second data, the first event and The second event is input to the target database to synchronize the data to be synchronized to the target database.

Specifically, during implementation, there are usually two cases. In the first case, the source database is a distributed database and the target database is a non-distributed database. In the second case, the source database is a non-distributed database and the target database is a distributed database. In this regard, in different situations, the delete event and insert event are handled differently.

Specifically, in the first case, considering the order of the delete event and the insert event, when the delete event is executed, due to the different data versions before and after the execution of the insert event, accidental deletion may occur. Therefore, it is necessary to add the target data version identifier, that is, the shard key, to the delete event to ensure that the data targeted by the delete event is accurate.

Optionally, when the source database is a distributed database and the target database is a non-distributed database, it is determined that the first data needs to be modified; the first data is modified according to a preset method; according to the modified first data, and For unmodified second data, input the first event and the second event into the target database, so as to synchronize the data to be synchronized to the target database.

In the second case, considering the performance and security considerations of distributed databases, the client is usually prohibited from performing cross-shard update actions, which will also cause the source to be non-distributed databases when related fields are involved. Unable to sync data. Therefore, analyzing the update event and constructing the delete event and insert event according to the information before and after the change of the update event is equivalent to equivalently converting the statement that cannot be directly executed, so as to store the target distributed database and achieve the final consistency of data synchronization sex. That is to say, there is no need to modify the delete event and insert event, and only need to split the update event into delete event and insert event to perform data synchronization.

Optionally, when the source database is a non-distributed database and the target database is a distributed database, it is determined that the first data or the second data does not need to be modified; according to the first execution attribute of the first event and the second event, Input the first event and the second event into the target database, so as to synchronize the data to be synchronized to the target database.

It should be noted that, regardless of the first case or the second case, the execution of the insert event can adopt the strategy of inserting if it does not exist, and updating if it exists. To ensure data security before and after the execution of the insert event.

Optionally, the method further includes: setting the attribute parameters of the second data, and determining the second execution attribute of the second event; inputting the first event and the second event into the target database according to the second execution attribute, so as to synchronize the Data is synchronized to the target database.

Optionally, modifying the first data according to a preset method includes: adding a shard key to the search condition in the statement data of the delete event, wherein the shard key is used to identify the storage location and modified version of the data. In the case that the first event is a delete event, the first data is the statement data of the delete event; and/or, the first execution attribute is the attribute of deleting the delete event and inserting the insert event into the target database; and/or, the second The second execution attribute is an attribute of an execution strategy that adopts an execution strategy of inserting if it does not exist and updating if it exists when inserting an insert event into the target database.

In addition, during implementation, there is usually data within a period of time for continuous primary key change operations. The data synchronization component replays the record change process through log parsing. Sometimes due to data repair or other reasons, it is necessary to adjust the synchronization point to a certain historical time point to perform synchronization again. At this time, the target table is not cleaned up. There will be situations where primary key conflicts cannot be stored at the target end. The data synchronization and repeatable storage method divides the update event into a delete+insert statement and executes the strategy of inserting if it does not exist and updating if it exists when the insert is stored, so as to realize the redoability of the data synchronization process without affecting the final data. consistency.

It should be noted that the steps shown in the flowcharts of the accompanying drawings may be performed in a computer system, such as a set of computer-executable instructions, and that although a logical order is shown in the flowcharts, in some cases, The steps shown or described may be performed in an order different than here.

It should be noted that the present application also provides an optional implementation manner, which will be described in detail below.

This implementation mode provides a redoable cross-shard synchronization method between non-distributed and distributed databases, which mainly includes a method for synchronizing to non-distributed databases when cross-slice update data appears on the distributed database as the source, and non-distributed databases. When the database is used as the source, a certain field is the sharding key storage method on the target side, and the data can be repeatedly synchronized into the storage method. The method proposed in this implementation mode cleverly adopts technologies such as data equivalent splitting, warehousing restrictions, and reasonable invocation of database native primary key merging to merge, sort, and aggregate logs that must be executed when the original distributed database is synchronized to the non-distributed database. Avoiding the link, greatly improving the performance and operability of data synchronization; breaking through the limitation that the non-distributed database cannot perform cross-shard update when the non-distributed database is synchronized to the distributed database; at the same time, it realizes the repeatability of data synchronization In the data storage process, there is no need to clean up the existing data on the target side in the data restoration process. When the synchronization points are equal, the final consistency of the data can be achieved, which has great application significance in actual operation and maintenance.

1. The method of synchronizing to a non-distributed database when the distributed database is used as the source when cross-slice update data appears:

Fig. 2 is a flow chart of the data synchronization of the distributed database provided by the related technology according to the embodiment of the present application. When there is a difference in the synchronization progress of the library, the data can still achieve final consistency. The traditional method consumes a lot of performance and has poor operability.

Fig. 3 is a flow chart of data synchronization of the distributed database provided according to the embodiment of the present application. As shown in Fig. 3, when the method of this embodiment handles a cross-shard update action at the source end, the proxy layer splits delete+insert After the log is parsed, the logs of all underlying libraries are read separately for independent synchronization. Additional processing is done on the SQL statement when storing in the database, and a limited condition is added when the delete statement is stored in the database, that is, an additional shard key is added in addition to the primary key field in the original where condition; the strategy of inserting when it does not exist and updating when it exists is adopted when inserting . After the above processing, no matter whether the target executes the delete statement first or the insert statement first, it will not affect the final consistency of data synchronization, and there is no need for complex log sorting and merging links.

Fig. 4 is a schematic diagram of an actual execution example of data synchronization provided by a distributed database according to an embodiment of the present application. As shown in Fig. 4, taking an actual execution process described in Fig. 4 as an example, the method of this embodiment is The delete and insert...on duplicate key update statements will be executed on different shards of the target-side distributed database during warehousing. No matter which statement is executed first, it will not affect the final consistency of data synchronization:

1) Execute the insert...on duplicate key update statement first, and then execute the delete statement. Since the where condition in the delete statement carries the condition of the shard key, the data inserted or updated by the insert statement executed first will not be deleted Broadcast delete.

2) Execute the delete statement first, and then execute the insert...on duplicate key update statement, which is consistent with the expected order, and data synchronization can achieve consistency.

2. When a non-distributed database is used as the source, a certain field is the storage method of the shard key at the target:

Fig. 5 is a flow chart of data synchronization provided by a non-distributed database as a source according to an embodiment of the present application. As shown in Fig. 5, usually a distributed database prohibits the client from performing cross-sharding due to performance and security considerations Update action, which will also lead to the problem that data synchronization cannot be performed when related fields are involved when the source is a non-distributed database. The method proposed in this embodiment analyzes the update event at the level of the data synchronization component, and constructs delete and insert events according to the information before and after the update, which is equivalent to equivalently converting the statement that cannot be directly executed, so as to store in the target distributed database , to achieve the final consistency of data synchronization.

3. Data synchronization and repeatable storage method:

In business, there is usually a continuous change operation of the primary key of a certain record within a period of time. The data synchronization component replays the record change process through log parsing. Sometimes due to data repair or other reasons, it is necessary to adjust the synchronization point to a certain historical time point to perform synchronization again. At this time, the target table is not cleaned up. There will be situations where primary key conflicts cannot be stored at the target end. The data synchronization and repeatable storage method divides the update event into a delete+insert statement and executes the strategy of inserting if it does not exist and updating if it exists when the insert is stored, so as to realize the redoability of the data synchronization process without affecting the final data. consistency.

Fig. 6 is a flow chart of repeatable warehousing during data synchronization according to the database provided by the embodiment of the present application. As shown in Fig. 6, Fig. 6 compares the difference between the traditional method and the method of this embodiment when processing repeated data synchronization , using this method can realize the replay of the data synchronization process under the condition of adjusting the position at will without cleaning the target table, and has strong operability and practicability.

Compared with the prior art, this implementation method cleverly uses the storage limit conditions to avoid the link of log merging, sorting and aggregation in the synchronization of distributed databases to non-distributed databases, reduces the overall complexity of the synchronization system, and greatly improves the synchronization speed. Using data equivalent splitting technology to break through the limitation that cross-shard update cannot be performed when the non-distributed database is synchronized to the distributed database under the premise of ensuring the final consistency of data synchronization. Reasonable use of the database's native primary key conflict merging technology, combined with the data equivalent split method to achieve repeatable data synchronization, has strong operability and operation and maintenance significance in practice.

For example, in the production system, a distributed database based on mySQL and an oracle database are used at the same time according to the division of different responsibilities. At the beginning, a full data cutover is required to import all the business data of the distributed data into the oracle database, and then transfer the distributed database to the oracle database every day. About 110G business-related data generated in the database is synchronized to the oracle database, and at the same time, the relevant configuration data in the oracle database is synchronized to the distributed database.

The specific implementation steps are as follows:

1. Use the data synchronization system developed by the method of this embodiment to synchronize table data, which can perform full synchronization and real-time incremental synchronization;

2. In this embodiment, mutual data synchronization between distributed and non-distributed databases is required, so two separate synchronization links are established. One of the links is from the distributed database to the oracle database, and the other link is from the oracle database to the distributed database. Fill in the corresponding source library information and target server information to automatically generate a synchronous mapping relationship with one click;

3. Open the synchronization link from the distributed database to the oracle database to perform a full data synchronization, and record the time point when the full data starts;

4. Before adjusting the incremental synchronization point to the full time point, start the incremental synchronization from the distributed database to the oracle database. When a cross-shard update action occurs, it will be processed by the method described in this implementation mode, and no additional operation by the user is required;

5. Open the synchronization link from the oracle database to the distributed data to perform incremental synchronization of configuration data. When a certain field is used as a shard key in the distributed database, the update statement cannot be executed directly in the distributed database due to the prohibition of cross-shard actions. The synchronization system will use the method described in this embodiment to split data equivalently and synchronize to the target end.

6. Occasionally, during the daily synchronization process, the application accidentally deletes some incremental data that is synchronized from the distributed database to the oracle database. The synchronization point can be adjusted according to the time. Using the method described in this embodiment, the data synchronization process can be redone to achieve The effect of data restoration.

Barrier-free and efficient synchronization between distributed and non-distributed databases. The data synchronization process can be redone. The incremental synchronization location can be adjusted at will according to the time point. When the incremental synchronization is equalized, the data can reach final consistency.

The embodiment of the present application also provides a data synchronization device for a database. It should be noted that the data synchronization device for a database in the embodiment of the present application can be used to implement the data synchronization method for a database provided in the embodiment of the present application. The following introduces the data synchronization device for the database provided by the embodiment of the present application.

Fig. 7 is a schematic diagram of a data synchronization device for a database provided according to an embodiment of the present application. As shown in Fig. 7, the device includes: an acquisition module 72, a splitting module 74, and a synchronization module 76. The device will be described in detail below illustrate.

The obtaining module 72 is used to obtain the data to be synchronized of the source database that needs to be synchronized, wherein the data to be synchronized includes the data generated by the target event; the splitting module 74 is connected to the above-mentioned obtaining module 72 and is used to target the data to be synchronized The event is split to obtain the first data and the second data, wherein the first data is the data generated by the first event, the second data is the data generated by the second event, the superposition of the first event and the second event and the target event Same; the synchronization module 76 is connected with the above-mentioned splitting module 74, and is used for performing data synchronization according to the first event and the second event, and synchronizing the data to be synchronized to the corresponding target database, wherein the target database is different from the source database , the target event has different data structures executed in different types of databases, and the data structures of the first event and the second event executed in different types of databases are the same.

The database data synchronization device provided by the embodiment of the present application obtains the data to be synchronized generated by the target event, and splits the target event into a first event and a second event that can be directly synchronized to the target database, thereby synchronizing the data to be synchronized The data is split into the first data and the second data, which achieves the purpose of directly splitting and synchronizing the data to be synchronized to the target database for the target event that relies on log merging and aggregation to achieve data synchronization, and reduces the waiting time of the target time. It is difficult to synchronize data for data synchronization. In the case of avoiding log merging and aggregation, the technical effect of data synchronization can be effectively and accurately realized, thereby solving the problem of data synchronization involving distributed databases in related technologies. Logs need to be merged, sorted and aggregated To achieve, resulting in poor operability, in some cases can not be synchronized.

The data synchronization device of the database includes a processor and a memory, and the above-mentioned acquisition module 72, splitting module 74, synchronization module 76, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding Function.

The processor includes a kernel, and the kernel fetches corresponding program units from the memory. One or more kernels can be set. By adjusting the kernel parameters, data synchronization involving distributed databases in related technologies can be solved. Logs need to be merged, sorted and aggregated to achieve this, resulting in poor operability and in some cases the problem of inability to synchronize.

Memory may include non-permanent memory in computer-readable media, random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM), memory includes at least one memory chip.

An embodiment of the present invention provides a computer-readable storage medium on which a program is stored, and when the program is executed by a processor, a data synchronization method for a database is implemented.

An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the data synchronization method of the database is executed when the program is running.

Fig. 8 is a schematic diagram of an electronic device provided according to an embodiment of the present application. As shown in Fig. 8, the embodiment of the present application provides an electronic device 80, which includes a processor, a memory, and storage on the memory and can process The program running on the processor implements the steps of any one of the above methods when the processor executes the program.

The devices in this application can be servers, PCs, PADs, mobile phones, etc.

The present application also provides a computer program product, which, when executed on a data synchronization device of a database, is suitable for executing a program initialized with any of the above method steps.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, a special purpose computer, an embedded processor, or a processor of other programmable database data synchronization equipment to produce a machine that is executed by a processor of a computer or other programmable database data synchronization equipment The instructions generate means for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable database data synchronization device to operate in a specific manner such that the instructions stored in the computer readable memory produce an article of manufacture comprising instruction means, The instruction means implements the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable database data synchronization device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The executed instructions provide steps for implementing the functions specified in the procedure or procedures of the flowchart and/or the block or blocks of the block diagrams.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include non-permanent storage in computer readable media, in the form of random access memory (RAM) and/or nonvolatile memory such as read only memory (ROM) or flash RAM. The memory is an example of a computer readable medium.

Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can be implemented by any method or technology for storage of information. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridge, tape disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media excludes transitory computer-readable media, such as modulated data signals and carrier waves.

It should also be noted that the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes Other elements not expressly listed, or elements inherent in the process, method, commodity, or apparatus are also included. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The above are only examples of the present application, and are not intended to limit the present application. For those skilled in the art, various modifications and changes may occur in this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims (10)

1. A data synchronization method of a database, characterized in that, comprising: Obtain the data to be synchronized of the source database that needs to be synchronized, where the data to be synchronized includes the data generated by the target event; Splitting the target event of the data to be synchronized to obtain first data and second data, wherein the first data is the data generated by the first event, and the second data is the data generated by the second event, The superposition of the first event and the second event is the same as the target event; According to the first event and the second event, data synchronization is performed, and the data to be synchronized is synchronized to a corresponding target database, wherein the type of the target database is different from that of the source database, and the target event is in The data structures implemented in different types of databases are different, and the data structures implemented in the first event and the second event in different types of databases are the same. 2. The method according to claim 1, wherein performing data synchronization according to the first event and the second event, and synchronizing the data to be synchronized to the corresponding target database comprises: determining whether to modify the first data or the second data according to the types of the source database and the target database; In the case of modifying the first data or the second data, according to the modified first data or second data and unmodified second data or first data, the first event and the inputting a second event into the target database, so as to synchronize the data to be synchronized to the target database; and / or, inputting the first event and the second event into the target database according to the execution attribute of the first data or the second data without modifying the first data or the second data , to synchronize the data to be synchronized to the target database. 3. The method according to claim 2, wherein, when the source database is a distributed database and the target database is a non-distributed database, it is determined that the first data needs to be modified; Modifying the first data according to a preset method; Inputting the first event and the second event into the target database according to the modified first data and the unmodified second data, so as to synchronize the data to be synchronized to the target database. 4. The method according to claim 2, wherein, when the source database is a non-distributed database and the target database is a distributed database, determining the first data or the second data no need for correction; inputting the first event and the second event into the target database according to the first execution attribute of the first event and the second event, so as to synchronize the data to be synchronized to the target database. 5. according to the described method of claim 3 or 4, it is characterized in that, described method also comprises: Setting an attribute parameter of the second data to determine a second execution attribute of the second event; Inputting the first event and the second event into the target database according to the second execution attribute, so as to synchronize the data to be synchronized to the target database. 6. The method according to claim 5, wherein the target event is an update event, the first event is a delete event, and the second event is an insert event. 7. The method according to claim 6, wherein modifying the first data according to a preset method comprises: A fragmentation key is added to the search condition in the statement data of the delete event, wherein the fragmentation key is used to identify the storage location and the modified version of the data, and when the first event is a delete event, the The first data is statement data of the deletion event; and / or, The first execution attribute is an attribute of executing the delete event and the insert event into the target database; and / or, The second execution attribute is an attribute of the execution strategy of inserting if it does not exist and updating if it exists when the insert event is stored in the target database. 8. A data synchronization device for a database, comprising: An acquisition module, configured to acquire data to be synchronized from a source database that requires data synchronization, wherein the data to be synchronized includes data generated by a target event; A splitting module, configured to split the target event of the data to be synchronized to obtain first data and second data, wherein the first data is the data generated by the first event, and the second data is the first The data generated by two events, the superposition of the first event and the second event is the same as the target event; A synchronization module, configured to perform data synchronization according to the first event and the second event, and synchronize the data to be synchronized to a corresponding target database, wherein the target database is of a different type from the source database, The target events have different data structures executed in different types of databases, and the data structures of the first event and the second event executed in different types of databases are the same. 9. A computer-readable storage medium, wherein the storage medium is used to store a program, wherein the program executes the data synchronization method for a database according to any one of claims 1-7. 10. An electronic device, characterized in that it includes one or more processors and memory, and the memory is used to store one or more programs, wherein, when the one or more programs are used by the one or more When the processors are executed, the one or more processors are made to realize the data synchronization method of the database according to any one of claims 1-7.

Images