CN117520306B - Data verification method and device and electronic equipment - Google Patents

Abstract

The application provides a data verification method, a data verification device and electronic equipment. The method comprises the following steps: determining N first data tables to be checked and second data tables to be checked corresponding to each first data table to be checked; wherein N is a positive integer; dividing the N first data tables to be checked into M data table groups based on sampling differences among the N first data tables to be checked; wherein M is a positive integer greater than 1; sequentially distributing the verification tasks corresponding to the M data table groups to M thread groups; and based on the M thread groups, according to the verification task, carrying out data verification on the data in the N first data tables to be verified and the corresponding second data tables to be verified. According to the scheme, the first data tables to be verified are grouped based on sampling differences among the N first data tables to be verified, so that verification tasks corresponding to different concurrent thread groups are distributed more uniformly, and the efficiency of data verification can be improved as a whole.

Description

Data verification method, device and electronic equipment

Technical Field

The present application relates to the field of data processing technology, and in particular to a data verification method, device and electronic equipment.

Background Art

Usually when data between two databases is synchronized or migrated, there may be data differences between different data sources. In order to achieve data consistency, the data table needs to be verified. However, the data verification method in the related art has the problems of low processing efficiency and low resource utilization.

Summary of the invention

In order to solve the above problems, the present application provides a data verification method, device and electronic device.

According to a first aspect of the present application, a data verification method is provided, comprising:

Determine N first data tables to be verified, and a second data table to be verified corresponding to each of the first data tables to be verified; wherein N is a positive integer;

Based on the sampling differences between the N first data tables to be checked, the N first data tables to be checked are divided into M data table groups; wherein M is a positive integer greater than 1;

Allocate the verification tasks corresponding to the M data table groups to the M thread groups in sequence;

Based on the M thread groups and according to the verification task, data verification is performed on the data in the N first data tables to be verified and their respective corresponding second data tables to be verified.

According to a second aspect of the present application, a data verification device is provided, comprising:

A determination module, used to determine N first data tables to be checked, and a second data table to be checked corresponding to each of the first data tables to be checked; wherein N is a positive integer;

A grouping module, configured to divide the N first data tables to be checked into M data table groups based on sampling differences between the N first data tables to be checked; wherein M is a positive integer greater than 1;

An allocation module, used for allocating the verification tasks corresponding to the M data table groups to the M thread groups in sequence;

A verification module is used to perform data verification on the data in the N first data tables to be verified and their respective corresponding second data tables to be verified based on the M thread groups and according to the verification task.

According to a third aspect of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the instructions to implement the method described in the first aspect above.

According to a fourth aspect of the present application, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in the first aspect above.

According to the technical solution of the present application, by dividing N first data tables to be verified into M data table groups, and based on the sampling differences between the N first data tables to be verified, the verification tasks corresponding to the M data table groups are sequentially allocated to the M thread groups, and based on the M thread groups, according to the verification tasks, data verification is performed on the data in the N first data tables to be verified and their respective corresponding second data tables to be verified. This solution groups the first data tables to be verified based on the sampling differences between the first data tables to be verified to reduce the sampling differences of the verification tasks corresponding to each data table group, thereby making the verification tasks corresponding to different concurrent thread groups more evenly distributed, and thus improving the efficiency of data verification as a whole.

Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

FIG1 is a flow chart of a data verification method provided in an embodiment of the present application;

FIG2 is a flow chart of another data verification method provided in an embodiment of the present application;

FIG3 is a flow chart of another data verification method provided in an embodiment of the present application;

FIG4 is an example diagram of dividing data table groups in an embodiment of the present application;

FIG5 is a flow chart of another data verification method provided in an embodiment of the present application;

FIG6 is an example diagram of data verification in an embodiment of the present application;

FIG7 is a structural block diagram of a data verification device provided in an embodiment of the present application;

FIG8 is a structural block diagram of an electronic device provided in an embodiment of the present application.

DETAILED DESCRIPTION

Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

It should be noted that, usually when data between two databases is synchronized or migrated, there may sometimes be data differences between different data sources. In order to achieve data consistency, the data table needs to be verified.

However, the data verification method of the related art uses a serial method to verify the data table. The entire verification process is time-consuming, the processing efficiency is low, and the resources on the machine are not fully utilized, resulting in a low resource utilization rate.

In order to solve the above problems, the present application provides a data verification method, device and electronic device.

FIG1 is a flow chart of a data verification method provided in an embodiment of the present application. It should be noted that a data verification method provided in an embodiment of the present application can be applied to a data verification device in an embodiment of the present application, and the device can be configured in an electronic device. Among them, the electronic device can be a server, a server cluster, etc. As shown in FIG1, the method can include the following steps:

Step 101, determining N first data tables to be checked and a second data table to be checked corresponding to each first data table to be checked; wherein N is a positive integer.

In some embodiments of the present application, the first data table to be verified and the second data table to be verified corresponding to each first data table to be verified can be determined by a verification task. The verification task includes a source data table and a target data table to be verified, as well as a data range to be verified. As an example, the type of the verification task can be at least one of the following: a data verification type for a specified row, a data verification type for a specified time period, an incremental data verification type, and a full data verification type.

Among them, the verification task of the data verification type of the specified row is suitable for the case of a small amount of data verification, and the scenario of verifying the data in the range of known inconsistent data rows. The verification task of the data verification type of the specified time period can be used for the scenario where the system data fails for a period of time, so as to verify the data within a certain period of time. The verification task of the incremental data verification type is to perform incremental data verification on the normally running system to ensure the real-time consistency of data between different databases, which is suitable for real-time data verification scenarios. The verification task of the full data verification type is to perform data verification on the newly created database or the fully migrated database.

In some embodiments of the present application, the first data table to be verified may be a source data table in a verification task, and the second data table to be verified may be a target data table in the verification task. In the verification task, one first data table to be verified may correspond to one second data table to be verified, or one first data table to be verified may correspond to multiple second data tables to be verified. Usually, the first data table to be verified and the second data table to be verified belong to different databases.

Step 102 : Based on the sampling differences between the N first verification data tables, divide the N first to-be-verified data tables into M data table groups; wherein M is a positive integer greater than 1.

In some embodiments of the present application, M may be a pre-set value or a value determined based on current hardware conditions, which is not limited here. For example, based on hardware resource conditions, the resources currently available for data verification are determined to determine the number of thread groups currently available for data verification. It should be noted that the number of data table groups is consistent with the number of thread groups used to perform verification tasks.

The data table group is not a storage tool for accommodating multiple first data tables to be verified. The data table group can only be an intermediate process for implementing grouping of N first data tables to be verified, which can be based on the division of identification information of the first data tables to be verified. For example, each data table group contains at least identification information of the first data table to be verified.

In some embodiments of the present application, the sampling difference between N first data tables to be checked refers to the difference in sampling difficulty of each first data table to be checked. For example, the sampling difference between N first data tables to be checked can be the difference in time corresponding to taking out the same number of samples from each first data table to be checked, or it can be the difference between the data structures of each first data table to be checked, or it can be the difference between the data amounts of each first data table to be checked, or it can be the difference in the number of samples taken out from each first data table to be checked in the same duration.

As a possible implementation method, based on the sampling differences between the N first data tables to be verified, the implementation method of dividing the N first data tables to be verified into M data table groups may include: determining the amount of data to be verified of each first data table to be verified; according to the amount of data to be verified of each first data table to be verified, dividing the N first data tables to be verified into M data table groups, so that the difference in the total amount of data to be verified corresponding to each data table group is as small as possible, so that the time consumption of the verification tasks corresponding to each data table group can be made closer.

As another possible implementation method, due to differences in structure, data volume, etc. of different data tables, the time consumed to extract a certain amount of data samples from different data tables varies greatly. Based on the sampling differences between the N first verification data tables, an implementation method for dividing the N first data tables to be verified into M data table groups may include: determining the sampling time of each first data table to be verified; and dividing the N first data tables to be verified into M data table groups according to the sampling time of each first data table to be verified, so as to minimize the difference in total sampling time of at least one first data table to be verified corresponding to each data table group.

Step 103: distribute the verification tasks corresponding to the M data table groups to the M thread groups in sequence.

That is, the verification tasks corresponding to the M data table groups are assigned to the M thread groups one by one, so that each thread group performs the data verification task corresponding to a specific data table group. The M thread groups may be pre-configured based on hardware conditions.

Step 104 , based on the M thread groups and according to the verification task, data verification is performed on the data in the N first data tables to be verified and their corresponding second data tables to be verified.

That is, the M thread groups complete the verification task in parallel, so as to implement data verification on the data in the N first to-be-verified data tables and their respective corresponding second to-be-verified data tables.

As an example, if the data verification task corresponding to data table group 1 is assigned to thread group 1, and data table group 1 includes a first data table A to be verified, a first data table B to be verified, and a first data table C to be verified, then thread group 1, according to the verification task and according to the row range, batches takes out the first batch of first data to be verified from the first data table A to be verified, and takes out the first batch of second data to be verified from the second data table to be verified corresponding to the first data table A to be verified; compares the first batch of first data to be verified with the first batch of second data to be verified row by row to obtain the verification results of the first batch of data; continues to take out the second batch of data to be verified from the first data table A to be verified and its corresponding second data table to be verified according to the above method for verification, and so on, until the data verification of the first data table A to be verified, the first data table B to be verified, the first data table C to be verified, and their respective corresponding second data tables to be verified is completed.

It should be noted that the type of the verification task can be at least one of the following: data verification type for a specified row, data verification type for a specified time period, incremental data verification type, and full data verification type.

For example, if the verification task is a data verification type for a specified row, then according to the verification task, data verification is performed on the data in the N first data tables to be verified and their corresponding second data tables to be verified, which is equivalent to performing data verification on the data of the rows to be verified in the N first data tables to be verified and their corresponding second data tables to be verified; if the verification task is a data verification type for a specified time period, then according to the verification task, data verification is performed on the data in the N first data tables to be verified and their corresponding second data tables to be verified, which is equivalent to performing data verification on the data in the specified time period in the N first data tables to be verified and their corresponding second data tables to be verified; if the verification task is an incremental data verification type, then according to the verification task, data verification is performed on the data in the N first data tables to be verified and their corresponding second data tables to be verified, which is equivalent to performing data verification on the newly added data in the N first data tables to be verified and their corresponding second data tables to be verified; if the verification task is a full data verification type, then according to the verification task, data verification is performed on the data in the N first data tables to be verified and their corresponding second data tables to be verified, which is equivalent to performing data verification on all the data in the N first data tables to be verified and their corresponding second data tables to be verified.

In some embodiments of the present application, the method may further include: obtaining data verification results, and locating the data with differences according to the data verification results. The data verification results may include identification information of the data table where the data with differences is located, row identification and column identification of the data table where the data with differences is located, so the location of the data with differences can be determined according to the data verification results. In addition, the data with differences can also be repaired to achieve data consistency between databases.

According to the data verification method of the embodiment of the present application, the N first data tables to be verified are divided into M data table groups based on the sampling differences between the N first data tables to be verified, and the verification tasks corresponding to the M data table groups are sequentially assigned to the M thread groups. Based on the M thread groups, according to the verification tasks, data verification is performed on the data in the N first data tables to be verified and their respective corresponding second data tables to be verified. This solution groups the first data tables to be verified based on the sampling differences between the first data tables to be verified to reduce the sampling differences of the verification tasks corresponding to each data table group, thereby making the verification tasks corresponding to different concurrent thread groups more evenly distributed, thereby improving the efficiency of data verification as a whole.

Next, the process of dividing data table groups will be introduced in detail.

FIG2 is a flow chart of another data verification method provided by an embodiment of the present application. As shown in FIG2, based on the above embodiment, step 102 in FIG1 may include the following steps:

Step 201, determining the sampling time consumption of each first to-be-checked data table.

The sampling time consumption of each first data table to be verified refers to the time consumed for extracting a preset number of data samples from each first data table to be verified.

As a possible implementation method, the process of determining the sampling time of each first data table to be checked may include: for each first data table to be checked, simulating the process of taking a preset number of samples from the first data table to be checked, and taking the time consumed by the simulation process as the sampling time of the first data table to be checked.

Step 202: divide the N first to-be-checked data tables into M data table groups according to the sampling time consumption.

Since the structure and data volume of each first data table to be verified are different, the time consumption of extracting a certain amount of data samples from different first data tables to be verified is different. In the data verification process, the time consumption of the process of batch extracting data from the data table has a direct impact on the time consumption of the data verification process, so the N first data to be verified can be divided into M data table groups according to the sampling time consumption, so that the difference between the total sampling time consumption of the first data table to be verified contained in each data table group is as small as possible, so that the time consumed by each thread group in executing the corresponding verification task can be relatively close, so as to improve the efficiency of data verification as a whole.

As a possible implementation manner, according to the sampling time consumption, the implementation process of dividing the N first to-be-checked data tables into M data table groups may include the following steps:

Step 202-1, determining the average time consumption of M thread groups according to the sampling time consumption.

As a possible implementation method, the sampling time of each first data table to be checked in the N first data tables to be checked can be added up to obtain the total sampling time of the N first data tables to be checked; the total sampling time of the N first data tables to be checked is divided by M to obtain the average time of the M thread groups.

Step 202-2: divide the N first to-be-checked data tables into M data table groups according to the average time consumption and the sampling time consumption.

That is, the M first data tables to be checked are divided into M data table groups, so that the total sampling time of the first data tables to be checked contained in each data table is as close as possible to the average time.

As an example, the first round of data table group division can be performed in a preset manner to determine the time difference of each data table group, wherein the time difference of each data table group is the difference between the total time consumption and the average time consumption of the first data table group to be checked contained in each data table; the time differences of N data table groups are added up to obtain the total time difference; based on the time difference of each data table group and the total time difference, the next round of data table group division is performed, and it is iterated continuously until the total time difference is minimized and no longer changes.

According to the data verification method of the embodiment of the present application, when N first data tables to be verified are divided into M data table groups, the sampling time of each first data table to be verified is first determined, and based on the sampling time, the N first data tables to be verified are divided into M data table groups, so that the total sampling time of the first data tables to be verified contained in each data table group is close. In this way, the time consumption of data verification of each thread group can be close, so that resources can be fully utilized through concurrent processing, and the overall verification efficiency can be improved.

Next, a process of dividing the N first to-be-checked data tables into M data table groups according to the average time consumption and the sampling time consumption will be described in detail.

FIG3 is a flow chart of another data verification method provided by an embodiment of the present application. As shown in FIG3 , based on the above embodiment, the implementation process of step 202-2 in FIG2 may include the following steps:

Step 301 , sort the N first to-be-checked data tables in descending order of sampling time consumption.

Step 302: for each data table group, respectively divide the i-th first data table to be checked and the N-i+1-th first data table to be checked in the sorting result into the i-th data table group; wherein i is a positive integer, and M is less than or equal to M.

As shown in Figure 4, the first first data table to be checked and the Nth first data table to be checked in the sorting result are divided into the first data table group, the second first data table to be checked and the N-1th first data table to be checked in the sorting result are divided into the second data table group, the third first data table to be checked and the N-2th first data table to be checked in the sorting result are divided into the third data table group, and so on, the Mth first data table to be checked and the N-M+1th first data table to be checked are divided into the Mth data table group.

Step 303: If there are any remaining first data tables to be checked, the remaining first data tables to be checked are further divided into M data table groups according to the average time consumption and the sampling time consumption.

That is to say, if after step 302, there are still remaining first data tables to be checked that have not been divided into data table groups, the remaining first data tables to be checked can continue to be divided into M data table groups so that the difference between the total sampling time and the average time of the first data tables to be checked included in each data table group is minimized.

As a possible implementation manner, the implementation process of step 303 may include:

Step 303 - 1 , determining the total sampling time consumption of the first to-be-checked data table included in each data table group.

As an example, if after the division in step 302, the first data table group includes the first data table A to be checked and the first data table B to be checked, then the total sampling time of the first data table to be checked in the first data table group is the sum of the sampling times of the first data table A to be checked and the first data table B to be checked.

Step 303 - 2 , determining S target data table groups from the M data table groups; wherein S is a positive integer less than or equal to M, and the total sampling time corresponding to each target data table group is less than the average time.

As a possible implementation, for each data table group, the total sampling time corresponding to the data table group may be compared with the average time. If the total sampling time corresponding to the data table group is less than the average time, the data table group is used as the target data table group.

Step 303 - 3 : for each target data table group, take out a first data table to be checked from the remaining first data tables to be checked, and divide it into the target data table group.

Step 303 - 4 , returning to the step of determining the total sampling time consumption of the plurality of first to-be-checked data tables included in each data table group, until the N first to-be-checked data tables are all divided into M data table groups.

That is, as shown in FIG4 , the process of dividing the remaining first data table to be checked into M data table groups may include at least one round of division process. In each round of division process, one remaining first data table to be checked is allocated to each target data table group, and then the process returns to step 303-1 until all N first data table groups to be checked are divided into M data tables. If the total sampling time corresponding to the first data table group is greater than the average sampling time, the remaining first data table to be checked is no longer divided into the data table group.

Among them, when executing step 303-3, for each target data table group, a first data table to be checked can be taken out from the remaining first data tables to be checked according to the total sampling time, the average time and the sampling time of each remaining first data table to be checked of the multiple first data tables to be checked contained in the target data table group, and it can be divided into the target data table group, so that the total sampling time corresponding to the target data table group is close to the average time.

According to the data verification method of the embodiment of the present application, when dividing the data table groups, the N first data tables to be verified are first sorted in descending order of sampling time consumption, and for each data table group, the i-th first data table to be verified and the N-i+1-th first data table to be verified in the sorting results are respectively divided into the i-th data table group; if there are remaining first data tables to be verified, the remaining first data tables to be verified are further divided into M data table groups according to the average time consumption and the sampling time consumption, so that the total sampling time corresponding to each data table group is close to the average time consumption, so that the time consumed for data verification of each thread group can be close, which can not only make full use of resources through concurrent processing, but also improve the overall verification efficiency.

Next, the data verification process will be introduced in detail.

FIG5 is a flow chart of another data verification method provided by an embodiment of the present application. As shown in FIG5, the implementation process of step 104 in FIG1 may include:

Step 501 , based on the current thread group and according to the verification task, batch data verification is performed on the to-be-verified data corresponding to the current thread group according to the data column dimension.

It is understandable that since a data table can contain tens of thousands of data, if the data to be verified is verified row by row, the entire data verification process will take a long time. If batch data verification is performed according to the column dimension, the data verification efficiency can be greatly improved and the time consumption of the data verification process can be reduced.

Among them, batch data verification according to the column dimension refers to a method of verifying column by column, and data verification of a column of data can be completed through one data verification process.

As a possible implementation manner, the implementation process of step 501 may include:

Step 501-1, according to the verification task, take out the jth batch of first data to be verified from the first data table to be verified contained in the data table group corresponding to the current thread group, and at the same time take out the jth batch of second data to be verified from the corresponding second data table to be verified; wherein j is a positive integer.

In some embodiments of the present application, when taking out the jth batch of first data to be verified from the first data table to be verified contained in the data table group corresponding to the current thread group, the data can be taken according to the time range or according to the preset row range, and the present application does not limit this.

It should be noted that if the data table group corresponding to the current thread group includes multiple first data tables to be verified, the current thread group can perform data verification on the multiple first data tables to be verified and their corresponding second data tables to be verified in a serial manner.

Step 501 - 2 , sorting the j-th batch of first data to be verified and the j-th batch of second data to be verified according to the primary key, to obtain a sorted first result set and a sorted second result set.

It can be understood that the rows of the first result set and the second result set sorted by the primary key correspond to each other, which facilitates data verification.

Step 501 - 3 , mapping each column of data in the first result set into a first digital string of fixed length.

The first digital string mapped to each column of data includes the numbers corresponding to each numerical value in the column of data, and the length of the first digital string can be determined based on the actual scenario, which is not limited in the present application.

In some embodiments of the present application, the first digital string corresponding to each column of data in the first result set may be composed of numbers corresponding to each value in each column of data. That is, the implementation process of mapping each column of data in the first result set to a fixed-length first digital string may include: mapping each value in each column of data in the first result set to a fixed-length number, respectively, to obtain a fixed-length first digital string composed of numbers corresponding to each value in each column of data. The length of the first digital string corresponding to each column of data is the length of the fixed-length number corresponding to each value in the column of data * the number of rows of the column of data.

It should be noted that the mapping between each numerical value of each column of data in the first result set and a fixed-length number is a one-to-one relationship, that is, for each column of data, the number corresponding to the same numerical value is the same, and the numbers corresponding to different numerical values are also different. If a column of data in the first result set contains multiple identical numerical values, the multiple identical numerical values are mapped to the same number. If there are multiple identical numerical values in different columns of data in the first result set, the same numerical values in different columns can be mapped to the same number or to different numbers. In other words, when mapping each column of data in the first data set, a mapping relationship between each numerical value in the entire first data set can be constructed, or a mapping relationship between each numerical value can be constructed for each column of data in the first data set.

As a possible implementation, the process of mapping each value in each column of data of the first result set to a fixed-length number includes: based on a preset dictionary library, mapping each value in each column of data of the first result set to a fixed-length number to obtain a first digital string corresponding to each column of data. The dictionary library includes a mapping relationship between each value and a number.

As another possible implementation method, if a mapping relationship for each numerical value is constructed for each column of data in the first data set when mapping each column of data in the first data set, then the process of mapping each numerical value in each column of data in the first result set to a fixed-length number includes: as shown in Figure 6, for each column of data in the first result set, each numerical value in the column of data is mapped to a continuous fixed-length number according to the corresponding primary key. For example, each numerical value can be mapped to 1001, 1002, 1003..., if the same numerical value appears, it will be mapped to the same number based on the established mapping relationship, and the numbers corresponding to each numerical value in each column of data are concatenated in order to obtain the corresponding first digital string.

Step 501 - 4 , based on the mapping relationship of the first result set, map each column of data in the second result set into a second digital string of fixed length.

It can be understood that the purpose of data verification is to determine whether the data in the first result set is consistent with the data in the second result set. Usually, most of the data is consistent, but there may be some inconsistency in individual data. Therefore, in order to improve the efficiency of mapping, each column of data in the second result set can be directly mapped according to the mapping relationship of the first result set.

That is, when mapping each column of data in the second result set, each column of data in the first result set and its corresponding first numerical string are used as a mapping dictionary library.

As an example, if the mapping relationship of the first result set is constructed based on the dimensions of the entire first result set, then for each column of data in the second result set, when mapping each value in the column of data, first determine whether there is the same value in the first result set; if there is the same value in the first result set, then determine the target number mapped to the same value in the first result set, and map the corresponding value of the column of data in the second result set to the target number; if there is no same value in the first result set, then map the value to a fixed-length number that has not appeared in the mapping relationship constructed by the first result set.

As another example, if the mapping relationship of the first result set is constructed based on the column dimension, then for each column of data in the second result set, when mapping each value in the column of data, first determine whether the corresponding column of the first result set has the same value; if the corresponding column of the first result set has the same value, then determine the target number mapped to the same value in the corresponding column of the first result set, and map the corresponding value of the column of data in the second result set to the target number; if the corresponding column of the first result set does not have the same value, then map the value to a fixed-length number that has not appeared in the numbers mapped by the corresponding column of the first result set.

Step 501 - 5 , comparing the first digital string corresponding to each column of data in the first result set with the corresponding second digital string respectively, so as to implement data verification of the j-th batch of data.

It can be understood that if the first digital string corresponding to a column of data in the first result set is consistent with the corresponding second digital string, it means that the data in the column of the first result set is consistent with the data in the corresponding column of the second result set. If the first digital string corresponding to a column of data in the first result set is inconsistent with the corresponding second digital string, it means that there is a data inconsistency problem between the data in the column of the first result set and the data in the corresponding column of the second result set.

In some embodiments of the present application, if there is a difference between the first digital string corresponding to a column of data in the first result set and the corresponding second digital string, the location of the difference data can be achieved according to the difference position of the digital string. For example, as shown in Figure 6, if there is a difference between the first digital string and the second digital string at the pos position for a column of data, it can be determined that the first result set and the second result set have a difference in the (pos/N)+(pos%N>0?1:0)th row of the column, and the data here needs to be repaired. Where N is the length of the digital string mapped to each value, that is, pos is divided by the length of the digital string mapped to each value. If there is no remainder in the result, the quotient obtained is used as the number of rows where the difference is located. If there is a remainder in the result, the quotient obtained plus 1 is used as the number of rows where the difference is located.

Step 501-6, set j to j+1, and return to execute the step of taking out the j-th batch of first data to be verified from the first data table to be verified contained in the first data table group according to the verification task, and at the same time taking out the j-th batch of second data to be verified from the corresponding second data table to be verified, until the verification task corresponding to the first data group is completed.

In some embodiments of the present application, the implementation process of the above step 501-3 may also include: determining whether the number of rows of data in the first result set is consistent with that in the second result set; if the number of rows of data in the first result set is consistent with that in the second result set, mapping each column of data in the first result set to a first fixed-length digital string. If the number of rows of data in the first result set is inconsistent with that in the second result set, it means that there may be a problem of missing data in the first data table to be verified and the corresponding second data table to be verified, so for this situation, a row-by-row verification method can be adopted.

According to the data verification method of the embodiment of the present application, when performing data verification based on M thread groups, batch data verification is performed according to the data column dimension, each column of data is mapped to a fixed-length digital string, and batch data verification of each column of data is implemented by comparing the corresponding digital strings, thereby further improving the efficiency of data verification.

In order to implement the above embodiment, the present application provides a data verification device.

FIG7 is a structural block diagram of a data verification device provided in an embodiment of the present application. As shown in FIG7 , it includes:

The determination module 701 is used to determine N first data tables to be checked, and a second data table to be checked corresponding to each first data table to be checked; wherein N is a positive integer;

A grouping module 702, configured to divide the N first data tables to be checked into M data table groups based on sampling differences between the N first data tables to be checked; wherein M is a positive integer greater than 1;

An allocation module 703 is used to allocate the verification tasks corresponding to the M data table groups to the M thread groups in sequence;

The verification module 704 is used to perform data verification on the data in the N first to-be-verified data tables and their corresponding second to-be-verified data tables according to the verification task based on the M thread groups.

In some embodiments of the present application, the grouping module 702 is specifically used to:

Determine the sampling time of each first to-be-checked data table;

According to the sampling time consumption, the N first to-be-checked data tables are divided into M data table groups.

As a possible implementation manner, the grouping module 702 is further configured to:

For each first data table to be verified, a process of taking a preset number of samples from the first data table to be verified is simulated, and the duration of the simulation process is used as the sampling time of the first data table to be verified.

In some embodiments of the present application, the grouping module 702 is further configured to:

According to the sampling time, determine the average time of M thread groups;

According to the average time consumption and the sampling time consumption, the N first to-be-checked data tables are divided into M data table groups.

As an example, the grouping module 702 is further configured to:

Sort the N first to-be-checked data tables in descending order of sampling time consumption;

For each data table group, the i-th first data table to be checked and the N-i+1-th first data table to be checked in the sorting result are respectively divided into the i-th data table group; wherein i is a positive integer, and M is less than or equal to M;

If there are remaining first data tables to be checked, the remaining first data tables to be checked are further divided into M data table groups according to the average time consumption and the sampling time consumption.

As another example, the grouping module 702 is further configured to:

Determine the total sampling time of the first to-be-checked data table included in each data table group;

Determine S target data table groups from M data table groups; wherein S is a positive integer less than or equal to M, and the total sampling time corresponding to each target data table group is less than the average time;

For each target data table group, taking out a first data table to be checked from the remaining first data tables to be checked, and dividing it into the target data table group;

Return and execute to determine the total sampling time consumption of the multiple first data tables to be checked included in each data table group, until the N first data tables to be checked are all divided into M data table groups.

In some embodiments of the present application, the verification module 704 is specifically used to:

Based on the current thread group and the verification task, batch data verification is performed on the data to be verified corresponding to the current thread group according to the data column dimension.

As a possible implementation, the home module 704 is also used for:

According to the verification task, the jth batch of first data to be verified is taken out from the first data table to be verified contained in the data table group corresponding to the current thread group, and the jth batch of second data to be verified is taken out from the corresponding second data table to be verified;

Sort the j-th batch of first data to be verified and the j-th batch of second data to be verified according to the primary key to obtain a sorted first result set and a sorted second result set;

Map each column of data in the first result set to a first digital string of fixed length;

Based on the mapping relationship of the first result set, each column of data in the second result set is mapped into a second digital string of fixed length;

Compare the first digital string corresponding to each column of data in the first result set with the corresponding second digital string respectively, so as to implement data verification of the jth batch of data;

Set j to j+1, and return to execute the step of taking out the jth batch of first data to be verified from the first data table to be verified contained in the first data table group according to the verification task, and at the same time taking out the jth batch of second data to be verified from the corresponding second data table to be verified, until the verification task corresponding to the first data group is completed.

As another possible implementation, the verification module 704 is further configured to:

Determine whether the number of rows of data in the first result set is consistent with that in the second result set;

If the number of rows of data in the first result set is consistent with that in the second result set, each column of data in the first result set is mapped to a first digital string of a fixed length.

In some embodiments of the present application, the verification module 704 is further used to:

Get data verification results;

According to the data verification results, the data with differences are located.

It should be noted that the type of the verification task is at least one of the following: a data verification type for a specified row, a data verification type for a specified time period, an incremental data verification type, and a full data verification type.

According to the data verification device of the embodiment of the present application, the N first data tables to be verified are divided into M data table groups based on the sampling differences between the N first data tables to be verified, and the verification tasks corresponding to the M data table groups are sequentially assigned to the M thread groups. Based on the M thread groups, according to the verification tasks, data verification is performed on the data in the N first data tables to be verified and their respective corresponding second data tables to be verified. This solution groups the first data tables to be verified based on the sampling differences between the first data tables to be verified to reduce the sampling differences of the verification tasks corresponding to each data table group, thereby making the verification tasks corresponding to different concurrent thread groups more evenly distributed, thereby improving the efficiency of data verification as a whole.

It should be noted that the above explanations and descriptions in the data verification method embodiment are also applicable to the data verification device in the embodiment of the present application and will not be repeated here.

In order to implement the above embodiments, the present application provides an electronic device.

FIG8 is a block diagram of an electronic device provided in an embodiment of the present application. The electronic device may be a server, a computer, or other device. As shown in FIG8 , the electronic device includes:

A memory 810 and a processor 820, a bus 830 connecting different components (including the memory 810 and the processor 820), the memory 810 stores instructions executable by the processor 820; wherein the processor 880 is configured to execute the instructions to implement the data verification method described in the embodiment of the present application.

Bus 830 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

The electronic device 800 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by the electronic device 800, including volatile and non-volatile media, removable and non-removable media. The memory 810 can also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 840 and/or cache memory 850. The electronic device 800 can further include other removable/non-removable, volatile/non-volatile computer system storage media. Just as an example, the storage system 860 can be used to read and write non-removable, non-volatile magnetic media (not shown in Figure 8, usually referred to as "hard drive"). Although not shown in Figure 8, a disk drive for reading and writing a removable non-volatile disk (such as a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical media) can be provided. In these cases, each drive can be connected to the bus 830 through one or more data medium interfaces. The memory 810 may include at least one program product. The program product has a set (eg, at least one) of program modules. The program modules are configured to execute the functions of various embodiments of the present application.

A program/utility 880 having a set (at least one) of program modules 870 may be stored, for example, in the memory 810, such program modules 870 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 870 generally perform the functions and/or methods of the embodiments described herein.

The electronic device 800 may also communicate with one or more external devices 890 (e.g., keyboard, pointing device, display 891, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and/or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed via an input/output (I/O) interface 898. In addition, the electronic device 800 may also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and/or public network, such as the Internet) via a network adapter 893. As shown, the network adapter 893 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and/or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

The processor 860 executes various functional applications and data processing by running the programs stored in the memory 810 .

It should be noted that the implementation process and technical principles of the electronic device of this embodiment refer to the aforementioned explanation of the data verification method of the embodiment of the present application, and will not be repeated here.

In order to implement the above embodiments, the present application also proposes a computer storage medium.

When the instructions in the storage medium are executed by the processor of the server, the server can perform the data verification method as described above. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

The logic and/or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims (12)

1. A data verification method, comprising: Determine N first data tables to be verified, and a second data table to be verified corresponding to each of the first data tables to be verified; wherein N is a positive integer; Based on the sampling differences between the N first data tables to be checked, the N first data tables to be checked are divided into M data table groups; wherein M is a positive integer greater than 1; Allocate the verification tasks corresponding to the M data table groups to the M thread groups in sequence; Based on the M thread groups, and according to the verification task, data verification is performed on the data in the N first data tables to be verified and their respective corresponding second data tables to be verified; The dividing the N first data tables to be checked into M data table groups based on the sampling differences between the N first data tables to be checked includes: Determine the sampling time of each of the first data tables to be verified, wherein the sampling time of each of the first data tables to be verified refers to the time consumed for extracting a preset number of data samples from each of the first data tables to be verified; Determine the average time consumption of the M thread groups according to the sampling time consumption; According to the average time consumption and the sampling time consumption, the N first to-be-checked data tables are divided into M data table groups. 2. The method according to claim 1, characterized in that the step of determining the sampling time of each of the first to-be-checked data tables comprises: For each of the first data tables to be verified, a process of taking a preset number of samples from the first data table to be verified is simulated, and the duration of the simulation process is used as the sampling time of the first data table to be verified. 3. The method according to claim 1, characterized in that the dividing the N first to-be-checked data tables into M data table groups according to the average time consumption and the sampling time consumption comprises: Sorting the N first to-be-checked data tables in descending order of sampling time consumption; For each of the data table groups, respectively divide the i-th first data table to be checked and the N-i+1-th first data table to be checked in the sorting result into the i-th data table group; wherein i is a positive integer, and M is less than or equal to M; If there are remaining first data tables to be checked, the remaining first data tables to be checked continue to be divided into the M data table groups according to the average time consumption and the sampling time consumption. 4. The method according to claim 3, characterized in that the step of continuing to divide the remaining first data table to be checked into the M data table groups according to the average time consumption and the sampling time consumption comprises: Determine the total sampling time of the first to-be-checked data table included in each of the data table groups; Determine S target data table groups from the M data table groups; wherein S is a positive integer less than or equal to M, and the total sampling time corresponding to each of the target data table groups is less than the average time; For each target data table group, taking out a first data table to be checked from the remaining first data tables to be checked, and dividing it into the target data table group; Return to the step of determining the total sampling time consumption of the plurality of first to-be-checked data tables included in each of the data table groups until the N first to-be-checked data tables are all divided into the M data table groups. 5. The method according to claim 1, characterized in that the step of performing data verification on the data in the N first data tables to be verified and their respective corresponding second data tables to be verified based on the M thread groups and according to the verification task comprises: Based on the current thread group and according to the verification task, batch data verification is performed on the to-be-verified data corresponding to the current thread group according to the data column dimension. 6. The method according to claim 5, characterized in that the step of performing batch data verification on the to-be-verified data corresponding to the current thread group according to the verification task according to the data column dimension comprises: According to the verification task, taking out the jth batch of first data to be verified from the first data table to be verified included in the data table group corresponding to the current thread group, and taking out the jth batch of second data to be verified from the corresponding second data table to be verified; wherein j is a positive integer; sorting the j-th batch of first data to be verified and the j-th batch of second data to be verified according to the primary key to obtain a sorted first result set and a sorted second result set; Mapping each column of data in the first result set into a first digital string of fixed length; Based on the mapping relationship of the first result set, mapping each column of data in the second result set to a second digital string of fixed length; Compare the first digital string corresponding to each column of data in the first result set with the corresponding second digital string respectively, so as to implement data verification of the jth batch of data; Set j to j+1, and return to execute the step of taking out the jth batch of first data to be verified from the first data table to be verified contained in the first data table group according to the verification task, and at the same time taking out the jth batch of second data to be verified from the corresponding second data table to be verified, until the verification task corresponding to the first data group is completed. 7. The method according to claim 6, wherein mapping each column of data in the first result set into a first digital string of fixed length comprises: Determine whether the number of rows of data in the first result set is consistent with that in the second result set; If the number of rows of data in the first result set is consistent with that in the second result set, each column of data in the first result set is mapped to a first digital string of a fixed length. 8. The method according to claim 1, further comprising: Get data verification results; According to the data verification result, the data with differences are located. 9. The method according to any one of claims 1-8 is characterized in that the type of the verification task is at least one of the following: a data verification type for a specified row, a data verification type for a specified time period, an incremental data verification type, and a full data verification type. 10. A data verification device, comprising: A determination module, used to determine N first data tables to be checked, and a second data table to be checked corresponding to each of the first data tables to be checked; wherein N is a positive integer; A grouping module, configured to divide the N first data tables to be checked into M data table groups based on sampling differences between the N first data tables to be checked; wherein M is a positive integer greater than 1; An allocation module, used for allocating the verification tasks corresponding to the M data table groups to the M thread groups in sequence; A verification module, configured to perform data verification on the data in the N first to-be-verified data tables and their respective corresponding second to-be-verified data tables based on the M thread groups and according to the verification task; The dividing the N first data tables to be checked into M data table groups based on the sampling differences between the N first data tables to be checked includes: Determine the sampling time of each of the first data tables to be verified, wherein the sampling time of each of the first data tables to be verified refers to the time consumed for extracting a preset number of data samples from each of the first data tables to be verified; Determine the average time consumption of the M thread groups according to the sampling time consumption; According to the average time consumption and the sampling time consumption, the N first to-be-checked data tables are divided into M data table groups. 11. An electronic device, comprising: processor; a memory for storing executable instructions for the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 9. 12. A computer-readable storage medium, characterized in that when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 9.