CN119597836A - Database data synchronization method, equipment, storage medium and product - Google Patents

Abstract

The invention provides a data synchronization method, equipment, a storage medium and a product of a database. The data synchronization method of the database comprises the steps of obtaining a table mapping log event of a source database, determining whether a data table recorded by the table mapping log event needs to be filtered according to preset configuration parameters, if yes, updating data table information recorded by the table mapping log event in a filtering buffer to selectively read and analyze a row record log event of the source database according to the information recorded by the filtering buffer, wherein the configuration parameters are used for identifying the database to which the data table and the table need to be synchronized or the database to which the data table and the table need to be filtered belong. By storing the data table information to be filtered in the filtering buffer, the line record log event of the source database can be selectively read and analyzed according to the information recorded by the filtering buffer, so that the resource consumption of the database system is reduced.

Description

Database data synchronization method, equipment, storage medium and product

Technical Field

The present invention relates to the field of database technologies, and in particular, to a method, an apparatus, a storage medium, and a product for synchronizing data in a database.

Background

In the operation of a database system, it is a common work requirement to synchronize two databases, wherein the database providing data is often referred to as a source database and the database receiving data is often referred to as a target database. In the process of data synchronization of the two databases, initializing and loading the stock data, namely synchronizing the stock data of the source end database to the target end database as a base point of subsequent data synchronization, and synchronizing the incremental data by taking the base point as a reference. For incremental data synchronization, the incremental data needs to be acquired by analyzing the log of the source database.

In the actual use process of the database system, part of data of the source end database does not need to be synchronized to the target end database according to the service requirement of a user, so that filtering processing is carried out on the data, specifically, after the logs of the source end database are analyzed to be synchronous data which can be synchronized to the target end database, deleting the data which does not need to be synchronized in the synchronous data according to the filtering condition, and synchronizing the rest data to the target end database. However, since the analysis of the log of the source database also requires consuming a large amount of system resources, the analysis operation is performed on the data that does not need synchronization, which results in a waste of system resources.

Disclosure of Invention

It is an object of the present invention to provide a data synchronization method, apparatus, storage medium and product of a database capable of solving any of the above problems.

In particular, the present invention provides a data synchronization method for a database, including:

Acquiring a table mapping log event of a source database;

Determining whether a data table recorded by the table mapping log event needs to be filtered according to preset configuration parameters, if so, updating the data table information recorded by the table mapping log event in a filtering buffer memory so as to selectively read and analyze the log event of the source database according to the information recorded by the filtering buffer memory;

The configuration parameters are used for identifying the data table to be synchronized and the database to which the table belongs, or the data table to be filtered and the database to which the table belongs.

Optionally, after determining that the data table of the table mapping log event record needs to be filtered, stopping reading the table mapping log event and generating an event file to be parsed with a filtering mark;

Reading the table mapping log event and generating a complete event file to be analyzed, and updating the data table information recorded by the table mapping log event in a white list buffer, wherein the white list buffer is used for storing the information of the data table to be synchronized.

Optionally, the step of determining whether the data table of the table mapping log event record needs to be filtered according to a pre-configured configuration parameter includes:

and detecting whether a data table recorded by the table mapping log event exists in the filtering buffer, if so, executing the step of stopping reading the table mapping log event and generating an event file to be analyzed with a filtering mark.

Optionally, the step of detecting whether the data table of the table mapping log event record exists in the filtering buffer includes:

If not, detecting whether the data table recorded by the table mapping log event exists in the white list buffer, if so, executing the steps of reading the table mapping log event and generating a complete event file to be analyzed, and if not, executing the step of determining whether the data table recorded by the table mapping log event needs to be filtered according to the pre-configured configuration parameters.

Optionally, the data synchronization method of the database includes:

acquiring a line record log event of a source database;

and detecting whether a data table recorded by the line record log event exists in the filtering buffer, if so, stopping reading the line record log event and generating an event file to be analyzed with a filtering mark, and if not, reading the line record log event and generating a complete event file to be analyzed.

Optionally, the data synchronization method of the database includes:

acquiring a generated event file to be analyzed;

And detecting whether the event file to be analyzed has a filtering mark, if so, skipping the event file to be analyzed, and if not, analyzing the event file to be analyzed.

Optionally, the step of updating the data table information of the table mapping log event record in the filtering buffer includes:

And storing the table numbers, table names and database names of the data tables recorded by the table mapping log event in the filtering buffer memory in the form of key value pairs.

According to another aspect of the present invention, there is also provided a computer device comprising a memory, a processor and a computer executable program stored on the memory and running on the processor, and the processor implementing a data synchronization method of a database according to any one of the above when executing the computer executable program.

According to another aspect of the present invention, there is also provided a computer-readable storage medium having stored thereon a computer-executable program which, when executed by a processor, implements a data synchronization method of a database according to any one of the above.

According to another aspect of the present invention, there is also provided a computer program product comprising a computer executable program which when executed by a processor implements a method of data synchronization of a database according to any of the preceding claims.

The data synchronization method, the device, the storage medium and the product of the database of the invention determine whether a data table recorded by the table mapping log event needs to be filtered according to the preset configuration parameters after the table mapping log event of the source database is acquired by providing external configuration parameters and configuring the filtering buffer, and if the data table needs to be filtered, the data table information recorded by the table mapping log event is updated in the filtering buffer. Because each row must have a corresponding table mapped log event before it is logged, its corresponding table mapped log event must have been detected when a subsequent log event is acquired. In this way, in the subsequent synchronous work, after the log event of either the table mapping or the line logging is obtained, whether the line logging is required to be filtered or not can be judged according to the information recorded by the filtering buffer, if so, the log event can not be read and parsed, that is, the log event of the source database can be selectively read and parsed. Therefore, the method does not need to completely read and analyze all log events, reduces the resource consumption of the database system, and is beneficial to improving the synchronization efficiency of the database system. In addition, the configuration parameters can be set by setting the configuration parameters to be synchronous or synchronous, so that a user can flexibly set the configuration parameters according to actual needs, and the occurrence of subsequent configuration modification is reduced.

The above, as well as additional objectives, advantages, and features of the present invention will become apparent to those skilled in the art from the following detailed description of a specific embodiment of the present invention when read in conjunction with the accompanying drawings.

Drawings

Some specific embodiments of the invention will be described in detail hereinafter by way of example and not by way of limitation with reference to the accompanying drawings. The same reference numbers will be used throughout the drawings to refer to the same or like parts or portions. It will be appreciated by those skilled in the art that the drawings are not necessarily drawn to scale. In the accompanying drawings:

FIG. 1 is a schematic flow chart of a method of data synchronization of a database according to one embodiment of the invention;

FIG. 2 is a schematic flow chart of a method of data synchronization of a database according to another embodiment of the invention;

FIG. 3 is a schematic flow chart of a method of data synchronization of a database according to yet another embodiment of the invention;

FIG. 4 is a schematic flow chart of a method of data synchronization of a database according to yet another embodiment of the invention;

FIG. 5 is a schematic flow chart diagram of a method of data synchronization of a database according to yet another embodiment of the invention;

FIG. 6 is a schematic diagram of a computer device according to one embodiment of the invention;

FIG. 7 is a schematic diagram of a computer-readable storage medium according to one embodiment of the invention;

FIG. 8 is a schematic diagram of a computer program product according to one embodiment of the invention.

Detailed Description

It should be understood by those skilled in the art that the embodiments described below are only some embodiments of the present invention, but not all embodiments of the present invention, and the some embodiments are intended to explain the technical principles of the present invention and are not intended to limit the scope of the present invention. All other embodiments, which can be obtained by a person skilled in the art without any inventive effort, based on the embodiments provided by the present invention, shall still fall within the scope of protection of the present invention.

It should be noted that the logic and/or steps represented in the flowcharts or otherwise described herein, for example, may be considered as a ordered listing of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions.

The flowcharts provided by the present invention are not intended to indicate that the operations of the methods are to be performed in any particular order, or that all of the operations of the methods are included in all of each case. Furthermore, the method may include additional operations. Additional variations may be made to the above-described methods within the scope of the technical ideas provided by the methods of the present embodiments.

As shown in fig. 1, in one embodiment, a method of data synchronization of a database generally includes:

Step S101, a table mapping log event of a source database is obtained. Specifically, for binlog (Binary Log) of a database, history information is recorded in the form of events (in the database field, may be referred to as events), so that each Log can be said to record one event. The TABLE mapping LOG EVENT (in the database field, may be called as table_map_log_event) and the line logging LOG EVENT (in the database field, may be called as ROWS _log_event) are two LOG EVENT types, and the line logging LOG EVENT is a LOG EVENT for performing a specific operation on the data TABLE, including an insert operation LOG EVENT (in the database field, may be called as write_ ROWS _log_event), an UPDATE operation LOG EVENT (in the database field, may be called as update_ ROWS _log_event) and a DELETE operation LOG EVENT (in the database field, may be called as delete_ ROWS _log_event).

Before logging the log event of the specific operation, the database writes a table mapping log event in the binlog, where the table mapping log event records definition information of the data table to be modified by the specific operation, including a table name, a database name to which the specific operation belongs, and a table number (i.e., table_id) converted from the definition information. The table mapping log event is used for providing definition information of a table for the subsequent line logging event, and the subsequent line logging event can be analyzed without recording the table information only by referring to the table number, so that the storage occupation is reduced.

Thus, for a transaction, a table map log event is generated in binlog for each table corresponding to all specific operations in the transaction.

Further, in binlog, the recorded event type is identified by type coding (which may be referred to as type_code in the database field). Thus, for this step, a log file of the source database is obtained, the event type of the log file record is determined according to the type code, and the table-mapped log event is determined to be recorded in the log file, that is, the table-mapped log event of the source database is obtained.

Step S102, determining whether the data table of the table mapping log event record needs to be filtered according to the preset configuration parameters, if so, executing step S103, and if not, executing step S104. The configuration parameters are used to identify the data table and the database to which the table belongs, or the data table and the database to which the table belongs, which need to be filtered.

Specifically, the data synchronization software provides external configuration parameters, and the configuration parameters have two configuration modes, one is used for a user to fill in a data table to be synchronized when the configuration is needed, and the other is used for the user to fill in a data table not to be synchronized when the configuration is needed, namely, a data table to be filtered. The user can set the configuration parameters according to the needs, for example, if the number of data tables needing to be synchronized is small, the configuration parameters needing to be synchronized are filled, and if the number of data tables needing to be synchronized is small, the configuration parameters needing not to be synchronized are filled.

Further, the configuration parameters are used to identify the data table and the database to which the table belongs, for example, the parameter configuration may be directly performed in a format of "database name, table name", or may also be identified by a preset common configuration symbol, for example, by "x" indicating that all the data tables in the database need to be synchronized, or all the data tables in the database need not be synchronized.

After a table mapping log event is obtained, table identification information of the table mapping log event is read first, wherein the table identification information at least comprises a table name and a database name to which the table identification information belongs. Then, using the table name and the database name to match the configuration parameters, it can be determined whether the data table of the current table map log event record needs to be filtered.

For example, the configuration parameters set up a data table and a database to which the table belongs that do not need to be synchronized, if the table identification information of the current table mapping log event record can be successfully matched with the configuration parameters, the data table indicating the current table mapping log event record needs to be filtered, and if the table identification information cannot be successfully matched with the configuration parameters, the data table indicating the current table mapping log event record needs to be synchronized.

For example, the configuration parameters set a data table to be synchronized and a database to which the table belongs, if the table identification information of the current table mapping log event record can be successfully matched with the configuration parameters, the data table of the current table mapping log event record needs to be synchronized, and if the table identification information cannot be successfully matched with the configuration parameters, the data table of the current table mapping log event record needs to be filtered.

Step S103, updating the data table information recorded by the table mapping log event in the filtering buffer. Specifically, the data table information includes at least a data table name and a database name to which the data table information belongs. The filtering buffer is a preset storage space and is used for storing information of the data table to be filtered. If the data table of the table mapped log event record needs to be filtered, the data table information of the table mapped log event record is added to the filtering buffer. Therefore, the log event of the source database can be selectively read and parsed according to the information recorded by the filtering buffer, namely, the log event corresponding to the data table in the filtering buffer is not read and parsed.

Further, the method includes storing the table number and table name of the data table of the table mapping log event record, and the database name in the filter buffer in the form of key value pairs. Specifically, the key value pair is a group of data (may be referred to as key-value in the database field) corresponding one to one, that is, the table number is one, and the table name and the database name are key value pairs formed by the other. Through the arrangement, the searching efficiency of the information stored in the filtering buffer can be improved, so that the synchronization efficiency is improved.

In addition, when the table identification information of the read table mapping log event is read, the table identification information also includes a table number.

Step S104, the data table information recorded by the table mapping log event is updated in the white list buffer. If the data tables of the table mapping log event records need to be synchronized, the data table information of the table mapping log event records is added to the white list buffer. In the white list buffer, only table names and database names are stored and stored in the form of a linked list, that is, a corresponding set of table names and database names can be regarded as one node of the linked list.

In the scheme of the embodiment, by providing external configuration parameters and configuring a filtering buffer, after the table mapping log event of the source database is acquired, whether the data table recorded by the table mapping log event needs to be filtered or not is determined according to the preset configuration parameters, and if the data table needs to be filtered, the data table information recorded by the table mapping log event is updated in the filtering buffer. Because each row logging event must be preceded by a corresponding table mapped logging event, when a row logging event is acquired, its corresponding table mapped logging event must be detected. In this way, in the subsequent synchronous work, after the log event is obtained, whether the log event is a table mapping log event or a line logging log event is obtained, whether the log event needs to be filtered or not can be judged according to the information recorded by the filtering buffer, and if the log event needs to be filtered, the log event can not be read and parsed, that is, the log event of the source database can be selectively read and parsed. Therefore, the method does not need to completely read and analyze all log events, reduces the resource consumption of the database system, and is beneficial to improving the synchronization efficiency of the database system.

In addition, the configuration parameters can be set by setting the configuration parameters to be synchronous or synchronous, so that a user can flexibly set the configuration parameters according to actual needs, and the occurrence of subsequent configuration modification is reduced.

Further, by setting configuration parameters, the data table information is added to the filtering buffer after the table mapping log event is obtained later, so that the table number can be obtained and added to the filtering buffer together. And the table numbers, table names and database names of the data table recorded by the table mapping log event are stored in the filtering buffer memory in the form of key value pairs, so that when the filtering buffer memory is used for judging whether the log event needs to be filtered, the information stored in the filtering buffer memory is searched without traversing all contents of the filtering memory, thereby improving the searching efficiency of the information stored in the filtering buffer memory and further being beneficial to improving the data synchronization efficiency.

Further, by setting the white list buffer for storing the information of the data table to be synchronized, when judging whether filtering is needed or not in the subsequent table mapping log event, the filtering buffer and the white list buffer can be used for judging first, and compared with the filtering buffer only, the frequency of connection with external configuration parameters can be reduced, so that resource consumption is reduced.

As shown in fig. 2, in one embodiment, a method of data synchronization of a database generally includes:

Step S201, a table mapping log event of a source database is obtained.

Step S202, determining whether the data table of the table mapping log event record needs to be filtered according to the preset configuration parameters, if so, executing step S203, and if not, executing step S205.

In step S203, the data table information recorded by the table mapping log event is updated in the filtering buffer.

Step S204, stopping reading the table mapping log event and generating an event file to be resolved with a filtering mark. Specifically, after determining that the data table recorded by the table mapping log event needs to be filtered, only the table identification information of the table mapping log event is read, subsequent contents are not read any more, and then an event file to be parsed with a filtering mark is generated. That is, the content in the event file to be parsed with the filter tag is incomplete.

Note that, the sequence of this step may be not limited to step S203.

In step S205, the data table information recorded by the table mapping log event is updated in the white list buffer.

Step S206, reading the table mapping log event and generating a complete event file to be resolved. Specifically, after determining that the data table recorded by the table mapping log event does not need to be filtered, continuing to read the remaining content of the table mapping log event until all reading is completed, and generating a complete event file to be analyzed.

In the scheme of the embodiment, after the fact that the data table recorded by the table mapping log event needs to be filtered is determined, reading of the table mapping log event is stopped and a to-be-parsed event file with a filtering mark is generated, that is, even if the table mapping log event needs to be filtered, the to-be-parsed event file is still formed, and because the to-be-parsed event file is provided with the filtering mark, the to-be-parsed event file can be determined not to be parsed according to the filtering mark, the integrity of the number of events in a transaction is guaranteed on the basis of reducing the consumption of system resources, and the occurrence of synchronous faults is reduced.

As shown in fig. 3, in one embodiment, a method of data synchronization of a database generally includes:

step S301, a table mapping log event of a source database is obtained.

Step S302, whether the data table recorded by the table mapping log event exists in the filtering buffer is detected, if yes, step S306 is executed, and if not, step S303 is executed. In particular, because the same table may be operated on in different transactions, different transactions may generate table map log events for the same table. Therefore, after the table mapping log event of the source database is acquired, whether the data table recorded by the table mapping log event exists in the filtering buffer memory is detected, that is, whether the table number of the current table mapping log event exists is searched in the information recorded by the filtering buffer memory.

Step S303, detecting whether the data table recorded by the table mapping log event exists in the white list buffer, if yes, executing step S308, and if not, executing step S304. If the data table is not in the filtering buffer, detecting whether the data table recorded by the table mapping log event is in the white list buffer, namely searching whether the table name and the database name of the current table mapping log event are in the information recorded by the white list buffer.

Step S304, determining whether the data table of the table mapping log event record needs to be filtered according to the preset configuration parameters, if so, executing step S305, and if not, executing step S307. If the data table of the table mapping log event record is not stored in the filtering buffer memory or the white list buffer memory, indicating that the data table of the table mapping log event record is read for the first time, determining whether the data table of the table mapping log event record needs to be filtered according to the configuration parameters.

In step S305, the data table information recorded by the table mapping log event is updated in the filtering buffer. If it is determined that the data table of the table mapping log event record needs to be filtered, the data table information is added to the filtering buffer.

Step S306, stopping reading the table mapping log event and generating an event file to be resolved with a filtering mark. If the data table of the table mapping log event record exists in the filtering buffer, or the data table of the table mapping log event record is determined to be filtered through the configuration parameter, the reading of the table mapping log event is stopped and an event file to be resolved with a filtering mark is generated.

In step S307, the data table information recorded by the table mapping log event is updated in the white list buffer. If it is determined that the data table of the table map log event record does not need to be filtered, the data table information is added to the white list buffer.

Step S308, reading the table mapping log event and generating a complete event file to be resolved. Specifically, if the data table of the table mapping log event record exists in the white list buffer, or after the configuration parameter determines that the data table of the table mapping log event record does not need to be filtered, the remaining content of the table mapping log event is continuously read until all the reading is completed, and a complete event file to be analyzed is generated.

In the scheme of this embodiment, by setting the filtering buffer and the white list buffer, after the table mapping log event of the source database is acquired, the filtering buffer and the white list buffer may be used to determine whether the data table needs to be filtered, if the filtering buffer and the white list buffer have no corresponding information, the filtering buffer and the white list buffer are compared with the configuration parameters, that is, the internal buffer is compared first, and if the internal buffer has no internal buffer and the external configuration parameters are compared, so as to avoid frequent connection of the external configuration parameters, thereby being helpful to reduce resource consumption.

In addition, because the white list buffer exists, even if the white list buffer cannot be found in the filtering buffer, the white list buffer can be found instead of directly connecting the external configuration parameters, and compared with the filtering buffer only, the frequency of connecting the external configuration parameters can be further reduced, so that the resource consumption is reduced.

Further, because the detection of the table mapping log event and the line logging log event both use the filtering buffer, and the table mapping log event searches the filtering buffer first, the filtering buffer stores information in the form of key value pairs, and the information stored in the filtering buffer is searched without traversing all contents of the filtering buffer, so that the searching efficiency of the information stored in the filtering buffer can be improved, that is, the detecting efficiency of whether the log event needs to be filtered or not is improved, and further the data synchronization efficiency is improved.

As shown in fig. 4, in one embodiment, a method of data synchronization of a database generally includes:

Step S401, acquiring a line record log event of a source database. Specifically, a log file of a source database is obtained, the event type of the log file record is determined according to the type code, and the log file record is determined to be a line record log event, that is, the line record log event of the source database is obtained.

Step S402, detecting whether the data table recorded by the line record log event exists in the filtering buffer, if yes, executing step S403, otherwise, executing step S404. Because each line log event must be preceded by a corresponding table-mapped log event, when a line log event is acquired, the corresponding table-mapped log event must be detected, and whether the line log event needs to be filtered can be determined only by detecting whether the data table recorded by the line log event exists in the filtering buffer. Specifically, the table identification information of the line record log event, namely the table number, the table name and the database name, is read to be compared with the information stored in the filtering buffer.

Step S403, stopping reading the line record log event and generating an event file to be resolved with a filter mark. Specifically, after determining that the data table recorded by the line log event needs to be filtered, only reading the table identification information of the line log event, and then generating an event file to be parsed with a filtering mark, wherein subsequent contents are not read any more. That is, the content in the event file to be parsed with the filter tag is incomplete.

Step S404, reading the line record log event and generating a complete event file to be analyzed. Specifically, after determining that the data table recorded by the line record log event does not need to be filtered, continuing to read the remaining content of the line record log event until all reading is completed, and generating a complete event file to be analyzed.

In the scheme of the embodiment, after the fact that the data table recorded by the line record log event needs to be filtered is determined, reading of the line record log event is stopped, and a to-be-parsed event file with a filtering mark is generated, that is, even if the line record log event needs to be filtered, a to-be-parsed event file is still formed, and because the line record log event is provided with the filtering mark, the to-be-parsed event file can be determined not to be parsed according to the filtering mark, on the basis of reducing system resource consumption, the integrity of the number of events in a transaction is guaranteed, and the occurrence of synchronous faults is reduced.

As shown in fig. 5, in one embodiment, a method of data synchronization of a database generally includes:

step S501, the generated event file to be parsed is obtained. Specifically, the event file to be parsed is obtained, which is read from the log of the source database.

Step S502, detecting whether the event file to be analyzed has a filtering mark, if yes, executing step S503, and if not, executing step S504. Specifically, the event file to be resolved with the filtering mark is not needed to be resolved, and the content is not complete and cannot be resolved, so that whether the event file to be resolved has the filtering mark is detected.

Step S503, skipping the event file to be parsed. Specifically, the analysis of the event file to be analyzed is stopped, and the next event file to be analyzed is acquired.

Step S504, analyzing the event file to be analyzed. Specifically, the event file to be analyzed is analyzed, so that data which can be synchronized to the target end database is obtained.

The present embodiment also provides a computer device and a computer-readable storage medium. Fig. 6 is a schematic diagram of a computer device 10 according to one embodiment of the invention. Fig. 7 is a schematic diagram of a computer-readable storage medium 20 according to one embodiment of the invention.

The computer device 10 may include a memory 110, a processor 120, and a computer executable program 11 stored on the memory 110 and running on the processor 120, and the processor 120 implements the database data synchronization method of any of the embodiments described above when executing the computer executable program 11.

The computer-readable storage medium 20 has stored thereon a computer-executable program 11, which when executed by a processor, implements the data synchronization method of the database of any of the above embodiments.

The present embodiment also provides a computer program product. Fig. 8 is a schematic diagram of a computer program product 30 according to one embodiment of the invention. The computer program product 30 comprises a computer executable program 11 which when executed by the processor 120 implements a method of data synchronization of any of the databases described hereinbefore.

In particular, the computer-executable program 11 for performing the operations of the present invention may be assembly instructions, instruction Set Architecture (ISA) instructions, computer-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages.

For the purposes of this description of the embodiments, computer-readable storage medium 20 can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include an electrical connection (an electronic device) having one or more wires, a portable computer diskette (a magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable storage medium 20 may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system.

The computer device 10 may be, for example, a server, a desktop computer, a notebook computer, a tablet computer, or a smartphone. In some examples, computer device 10 may be a cloud computing node. Computer device 10 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer device 10 may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computing system storage media including memory storage devices.

Computer device 10 may include a processor 120 adapted to execute stored instructions, a memory 110 providing temporary storage for the operation of the instructions during operation. Processor 120 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. Memory 110 may include Random Access Memory (RAM), read only memory, flash memory, or any other suitable storage system.

Processor 120 may be connected through a system interconnect (e.g., PCI-Express, etc.) to an I/O interface (input/output interface) adapted to connect computer device 10 to one or more I/O devices (input/output devices). The I/O devices may include, for example, a keyboard and a pointing device, which may include a touch pad or touch screen, among others. The I/O device may be a built-in component of the computer device 10 or may be a device externally connected to the computing device.

The processor 120 may also be linked through a system interconnect to a display interface suitable for connecting the computer device 10 to a display device. The display device may include a display screen as a built-in component of the computer device 10. The display device may also include a computer monitor, television or projector, etc. that is externally connected to the computer device 10. In addition, a network interface controller (network interface controller, NIC) may be adapted to connect the computer device 10 to a network through a system interconnect. In some embodiments, the NIC may use any suitable interface or protocol (such as an internet small computer system interface, etc.) to transfer data. The network may be a cellular network, a radio network, a Wide Area Network (WAN), a Local Area Network (LAN), or the internet, among others. The remote device may be connected to the computing device through a network.

By now it should be appreciated by those skilled in the art that while a number of exemplary embodiments of the invention have been shown and described herein in detail, many other variations or modifications of the invention consistent with the principles of the invention may be directly ascertained or inferred from the present disclosure without departing from the spirit and scope of the invention. Accordingly, the scope of the present invention should be understood and deemed to cover all such other variations or modifications.

Claims (10)

1. A method of data synchronization for a database, comprising:

Acquiring a table mapping log event of a source database;

Determining whether a data table recorded by the table mapping log event needs to be filtered according to preset configuration parameters, if so, updating the data table information recorded by the table mapping log event in a filtering buffer memory so as to selectively read and analyze the log event of the source database according to the information recorded by the filtering buffer memory;

The configuration parameters are used for identifying the data table to be synchronized and the database to which the table belongs, or the data table to be filtered and the database to which the table belongs.

2. The data synchronization method of a database according to claim 1, wherein,

Stopping reading the table mapping log event and generating an event file to be resolved with a filtering mark after determining that the data table of the table mapping log event record needs to be filtered;

Reading the table mapping log event and generating a complete event file to be analyzed, and updating the data table information recorded by the table mapping log event in a white list buffer, wherein the white list buffer is used for storing the information of the data table to be synchronized.

3. The data synchronization method of a database according to claim 2, wherein,

The step of determining whether the data table of the table mapping log event record needs to be filtered according to the pre-configured configuration parameters comprises the following steps:

and detecting whether a data table recorded by the table mapping log event exists in the filtering buffer, if so, executing the step of stopping reading the table mapping log event and generating an event file to be analyzed with a filtering mark.

4. The method for synchronizing data of a database according to claim 3, wherein,

The step of detecting whether the data table of the table mapping log event record exists in the filtering buffer comprises:

If not, detecting whether the data table recorded by the table mapping log event exists in the white list buffer, if so, executing the steps of reading the table mapping log event and generating a complete event file to be analyzed, and if not, executing the step of determining whether the data table recorded by the table mapping log event needs to be filtered according to the pre-configured configuration parameters.

5. The data synchronization method of a database according to claim 2, wherein,

The data synchronization method of the database comprises the following steps:

acquiring a line record log event of a source database;

and detecting whether a data table recorded by the line record log event exists in the filtering buffer, if so, stopping reading the line record log event and generating an event file to be analyzed with a filtering mark, and if not, reading the line record log event and generating a complete event file to be analyzed.

6. The method for synchronizing data of a database according to claim 5, wherein,

The data synchronization method of the database comprises the following steps:

acquiring a generated event file to be analyzed;

And detecting whether the event file to be analyzed has a filtering mark, if so, skipping the event file to be analyzed, and if not, analyzing the event file to be analyzed.

7. The data synchronization method of a database according to claim 1, wherein,

The step of updating the data table information of the table mapping log event record in the filtering buffer comprises the following steps:

And storing the table numbers, table names and database names of the data tables recorded by the table mapping log event in the filtering buffer memory in the form of key value pairs.

8. A computer device comprising a memory, a processor and a computer executable program stored on the memory and running on the processor, and the processor implementing a method of data synchronization of a database according to any one of claims 1 to 7 when executing the computer executable program.

9. A computer-readable storage medium having stored thereon a computer-executable program which, when executed by a processor, implements the data synchronization method of a database according to any one of claims 1 to 7.

10. A computer program product comprising a computer executable program which when executed by a processor implements a method of data synchronization of a database according to any one of claims 1 to 7.