CN116910050A - Data processing method, device, system and storage medium - Google Patents

Abstract

This application discloses a data processing method, device, system and storage medium. The method of this application includes: constructing a quality problem dictionary based on an experience database. The quality problem dictionary contains data quality problems and corresponding problem identifiers; obtaining the data to be processed. , identify the target data problems existing in the data to be processed; query the target data problems in the quality problem dictionary and obtain the corresponding target problem identification; add the target problem identification to the target data through the predefined fields; read the target data in the target data Multiple target fields are judged according to the preset fusion rules, and whether there is conflicting data is determined based on the judgment results; when there is conflicting data, the business application scenario of the current data is identified, and the authority is selected based on the identification results. Source judgment strategy: handle the trusted source authority level of each field in the conflicting data according to the authoritative source judgment strategy, and update the database.

Description

A data processing method, device, system and storage medium

Technical field

The present application relates to the field of data processing, and in particular, to a data processing method, device, system and storage medium.

Background technique

In modern businesses and organizations, data is generated and recorded through a variety of data sources and systems. Different departments and individuals often contribute to these data sources, leading to potential inconsistencies and inaccuracies in the data. This can pose significant challenges to data analysis, decision-making, and overall business efficiency. In the prior art, several methods have been tried to solve the data management problem. These methods include manual data validation, data deduplication algorithms, and basic data integration techniques.

However, these traditional approaches have limitations because they often lack a comprehensive approach to address all aspects of data management, including data cleaning, entity merging, and data aggregation. Manual verification processes are time-consuming and prone to human error, while existing data deduplication algorithms may not adequately handle complex data relationships. Additionally, current data aggregation techniques may not fully leverage authoritative sources or adequately address missing data issues.

Therefore, there is a need for an innovative data management and integration system that can efficiently clean, merge, and aggregate data from various sources while ensuring high accuracy, consistency, and availability of data.

Contents of the invention

In order to solve the above technical problems, this application provides a data processing method, which method includes:

Construct a quality problem dictionary based on the experience database, and the quality problem dictionary contains data quality problems and corresponding problem identifiers;

Obtain data to be processed and identify target data problems existing in the data to be processed;

Query the target data problem in the quality problem dictionary to obtain the corresponding target problem identifier;

Add the target problem identifier to the target data through a predefined field;

Read multiple target fields in the target data. The multiple target fields include: name, ID number, mobile phone number, date of birth, organization identification code and company name, and generate four types of target fields based on the aforementioned multiple target fields. Judgment factors, the four judgment factors are:

The first judgment factor: the combination of name and ID number, the second judgment factor: the combination of name and mobile phone number, the third judgment factor: the combination of name, date of birth and organization identification code, and the fourth judgment factor: the combination of name and company name. ;

The judgment strategy is: compare the data to be compared through the above four judgment factors. If in the data to be compared, one of the above four judgment factors matches, then the data to be compared is determined to be conflict data;

Wherein, the method for judging the match of the fourth judgment factor is:

When it is determined that the names are consistent, the semantic similarity between the company names in the data to be compared is calculated;

If the semantic similarity meets the preset threshold, it is determined that the fourth judgment factor matches;

When there is conflicting data, identify the business application scenario of the current data and select an authoritative source judgment strategy based on the identification results;

Process the trusted source authority level of each field in the conflict data according to the authoritative source judgment strategy, and update the database;

If there is no conflicting data, a unique identification code is generated for the current data and stored in the database.

Optionally, identify the business application scenario of the current data and select authoritative source judgment strategies based on the identification results, including:

If the current business application scenario is to process data from an existing historical database, select the first authoritative source judgment strategy. The first authoritative source judgment strategy includes:

Query the pre-built data authority source level table, use the data with the highest authority level from the conflict data as the business primary key according to the query results, and update the historical database.

Optionally, identify the business application scenario of the current data and select authoritative source judgment strategies based on the identification results, including:

If the current business application scenario is for warehousing and processing of real-time data, then select the second authoritative judgment strategy. The second authoritative judgment strategy includes:

Query the pre-built field authority source level table and compare the authority source levels corresponding to each field in the conflict data. Based on the comparison results, select the field with the highest authority source level for each field and generate the result data. The data is stored in the database, and the field authoritative source level is recorded with the level of the authoritative source of each field.

Optionally, after the result data is stored in the database, modification rules are set for the data after being stored in the database based on the pre-constructed field authority source level table. The modification rules include:

Identify the currently modified target field, and query the corresponding field authoritative source registration table based on the modified field;

Compare the authoritative source level of the current modifying user with the current authoritative source level of the target field based on the query results;

If the authoritative source level of the modifying user is greater than or equal to the current authoritative source level of the target field, the modification is allowed, otherwise the modification is rejected.

Optionally, if the modification is allowed, after the modification, the current authoritative source level of the target field is updated to the authoritative source level of the modifying user.

Optionally, identifying target data problems existing in the data to be processed includes:

Through predetermined pre-cleaning rules, the data to be processed is uniformly pre-cleaned. During the pre-cleaning process, target data that cannot be uniformly pre-cleaned and corresponding target data problems are marked.

A second aspect of this application provides a data processing device, which includes:

Constructing a quality problem dictionary module is used to construct a quality problem dictionary based on the experience database, where the quality problem dictionary contains data quality problems and corresponding problem identifiers;

The target data problem identification module is used to obtain the data to be processed and identify the target data problems existing in the data to be processed;

A target problem identification query module, used to query the target data problem in the quality problem dictionary and obtain the corresponding target problem identification;

A target problem identification adding module, configured to add the target problem identification to the target data through a predefined field;

The target field judgment and conflict detection module is used to read multiple target fields in the target data. The multiple target fields include: name, ID number, mobile phone number, date of birth, organization identification code and company name. , four judgment factors are generated based on the aforementioned multiple target fields, and the four judgment factors are:

The first judgment factor: the combination of name and ID number, the second judgment factor: the combination of name and mobile phone number, the third judgment factor: the combination of name, date of birth and organization identification code, and the fourth judgment factor: the combination of name and company name. ;

The judgment strategy is: compare the data to be compared through the above four judgment factors. If in the data to be compared, one of the above four judgment factors matches, then the data to be compared is determined to be conflict data;

Wherein, the method for judging the match of the fourth judgment factor is:

When it is determined that the names are consistent, the semantic similarity between the company names in the data to be compared is calculated;

If the semantic similarity meets the preset threshold, it is determined that the fourth judgment factor matches;

The authoritative source judgment strategy selection module is used to identify the business application scenario of the current data when there is conflicting data, and select the authoritative source judgment strategy based on the identification results;

A conflict data processing module, configured to process the trustworthy source authority level of each field in the conflict data according to the authoritative source judgment strategy, and update the database;

The unique identification code generation module is used to generate a unique identification code for the current data when there is no conflicting data and store it in the database.

The third aspect of this application provides a data processing system, the device includes:

Processors, memories, input-output units, and buses;

The processor is connected to the memory, the input and output unit and the bus;

The memory stores a program, and the processor calls the program to execute the first aspect and any optional method in the first aspect.

A fourth aspect of the present application provides a computer-readable storage medium. A program is stored on the computer-readable storage medium. When the program is executed on a computer, the program executes the first aspect and any one of the options optional in the first aspect. described method.

It can be seen from the above technical solutions that this application has the following advantages:

The method provided by this application identifies and marks problems existing in the data by building a quality problem dictionary and identification, which helps to accurately identify and handle data quality problems and improve the accuracy and credibility of the data.

By reading multiple fields in the target data and judging based on preset fusion rules, duplicate data and entities can be identified and merged into a unified entity, thus eliminating data conflicts and duplications and improving data quality. Consistency and completeness.

By identifying the business application scenario of the current data and selecting an appropriate authoritative source judgment strategy based on the identification results, the authoritative level of the trusted source for each field in the conflicting data can be processed to ensure the accuracy and credibility of the data.

The plan mentions selecting authoritative source judgment strategies based on business application scenarios, which makes data processing more flexible and adaptable to different business needs, ensuring the effectiveness of data processing methods in different scenarios.

By adding target problem identifiers to the target data through predefined fields, problems existing in the data can be quickly identified and processed, and the efficiency of data processing can be improved.

Description of the drawings

In order to explain the technical solutions in the present application more clearly, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic flow diagram of an embodiment of the data processing method provided in this application;

Figure 2 is a schematic diagram of the combination of judgment factors in this application;

Figure 3 is a schematic flow chart of the method of updating authoritative source levels based on modified rules in this application;

Figure 4 is a schematic structural diagram of the data processing device provided in this application;

Figure 5 is a schematic structural diagram of the data processing system provided in this application.

Detailed ways

It should be noted that the method provided by this application can be applied to terminals, systems, and servers. For example, the terminal can be a smartphone, computer, tablet, smart TV, smart watch, or portable computer terminal. It can be a fixed terminal such as a desktop computer. For convenience of explanation, in this application, the terminal is used as the execution subject for illustration.

This applicant declares: When collecting personal information through this program, it includes legal collection through public channels and collection with personal permission, and the collected personal information will not be illegally disclosed. All personal information involved in this application has obtained the permission of the relevant parties.

Specific embodiments of the methods provided in this application are described in detail below:

Referring to Figure 1, this application first provides an embodiment of a data processing method, which includes:

S101. Construct a quality problem dictionary based on the experience database. The quality problem dictionary includes data quality problems and corresponding problem identifiers;

In this step, a quality problem dictionary is first constructed based on the empirical database. This dictionary contains various possible data quality problems and corresponding problem identifiers. The problem identifiers are used to subsequently identify and mark quality problems existing in the target data. First, an experience database needs to be established, which is used to record various data quality problems that occurred in the past data processing process, as well as the corresponding problem identification. This experience database can be accumulated by professional data management teams, data quality experts or previous data processing experience. In the empirical database, common data quality problems from different data sources and different types of data are collected and sorted, such as missing data, wrong data, duplicate data, format errors, etc. Every data quality issue should have a corresponding issue identifier.

For example, first construct a quality problem dictionary, organize the data quality problems of information items such as company names, personal names, mobile phone numbers, etc. and assign problem identifiers. The following is an example of a quality problem dictionary:

S102. Obtain the data to be processed and identify target data problems existing in the data to be processed;

In this step, the data to be processed is obtained and identified to determine the target data problems existing in the data to be processed. This step is used to discover possible quality issues in the target data. Information. Then, through the algorithms and rules in the data processing method, the data to be processed is identified to determine the target data problems that may exist in it.

First, the data to be processed is obtained from various data sources and systems. These data can be structured data tables, records in the database, or unstructured data text, pictures, audio, etc. After obtaining the data to be processed, the data is identified by using predefined algorithms and rules to determine the target data problems that may exist in it. Target data questions can include but are not limited to the following:

Missing data: Certain fields or attributes lack necessary data values.

Bad data: Data values do not conform to the specified format, range, or logical conditions.

Duplicate data: There are multiple identical or nearly identical data records.

Format error: The format of the data does not match the expected format.

Logical error: The logical relationship between data is incorrect or inconsistent.

Once the target data problem is identified, the system can add problem identification to the corresponding location or field in the data based on the identification results. A problem identifier can be a predefined code, symbol, or specific value that marks the problematic portion of the data.

This step can also perform quality assessment on the data, such as calculating the proportion of missing data, detecting the logical consistency of the data, etc. These assessments can provide an overall picture of data quality and possible problems.

One way to implement this step is to perform unified pre-cleaning processing on the data to be processed through predetermined pre-cleaning rules. During the pre-cleaning process, mark the target data that cannot be uniformly pre-cleaned and the corresponding targets. Data issues.

First, develop a predetermined set of pre-cleaning rules. These rules can include data formatting, removing null values or outliers, fixing common errors, etc. Predetermined rules can be established based on experience databases or professional domain knowledge, or can be obtained through data analysis and data mining methods.

Use predetermined pre-cleaning rules to uniformly pre-clean the data to be processed. In this step, the system will automatically process the data according to predetermined rules, convert the data into a unified format and standards, fix common errors, and remove or fill missing values. During the pre-cleaning process, the system can mark target data that cannot be uniformly pre-cleaned according to predefined problem identifiers. These target data issues can be those that do not comply with predetermined rules, as well as other data quality issues that cannot be handled automatically.

S103. Query the target data problem in the quality problem dictionary and obtain the corresponding target problem identifier;

In the previous steps, possible data quality issues in the target data have been identified. In this step, the system will traverse the quality problem dictionary based on these target data problems and check one by one whether the problems in the dictionary match the target data problems. When it is found that there is a data quality problem in the dictionary that matches the target data problem, the system will obtain the problem identification corresponding to the problem. Then, add the problem identifier to the corresponding field or location in the target data to mark quality problems in the target data. If the target data issue does not find a match in the quality issue dictionary, the system can handle these unmatched issues individually, such as generating a new issue ID, or logging them as specific unknown issues. In order to improve query efficiency and reduce data processing time, appropriate data structures and algorithms can be used to query the quality problem dictionary, such as hash tables, binary tree search methods, dictionary trees, etc.

S104. Add the target problem identifier to the target data through a predefined field;

In step S104, the target problem identification obtained in the previous step S103 is added to the target data through predefined fields, thereby marking existing problems and providing a basis for subsequent processing. The specific implementation method can be as follows:

One or more fields can be predefined to store the target problem identification. These fields can be newly created or existing fields. In step S103, problems existing in the target data are identified by querying the quality problem dictionary, and corresponding target problem identifiers are obtained. Then, these target problem identifications are added to the corresponding positions or fields of the target data one by one through predefined fields.

The target problem ID can be added in different ways, such as recording the problem ID's code or symbol in a new field, or marking the problem ID as part of the target field. The marking method should be selected based on actual needs and data processing scenarios.

After the target problem identification is added to the target data, the data needs to be organized and stored. This may involve operations such as data cleaning, format conversion, and data archiving to ensure data consistency and integrity.

Through the above implementation method, step S104 can add the target problem identifier to the target data, thereby marking the problems existing in the data. In this way, the data can be classified, filtered, fused and other operations based on the problem identification in the subsequent processing stage, which improves the accuracy and efficiency of data processing. At the same time, the addition of target problem identification also provides convenience and basis for data quality assessment and problem troubleshooting.

S105. Read multiple target fields in the target data, judge the multiple target fields according to the preset fusion rules, and determine whether there is conflicting data based on the judgment results;

In step S105, the goal is to read multiple target fields in the target data and judge these fields according to preset fusion rules to determine whether there is conflicting data, that is, inconsistent data of the same entity. First, multiple target fields that require fusion judgment are read from the target data. These fields may contain data from different sources, different times, or different formats. According to the preset fusion rules, the multiple target fields read are compared and judged one by one. If the entity information represented by multiple fields is found to be inconsistent during the fusion judgment, that is, conflicting data exists, the system will mark or record the conflicting data. May be marked with a specific identifier, status value, or other means.

In this application, the multiple target fields identified can be name, ID number, mobile phone number, date of birth, organization identification code and company name. When executing the conflict data judgment strategy, these fields can be combined into judgment conditions. This application does not limit the specific field combination rules for data duplication checking. Below, this application provides a preferred embodiment, which includes a preferred judgment strategy:

Referring to Figure 2, four judgment factors are generated based on the aforementioned multiple target fields. The four judgment factors are:

The first judgment factor: the combination of name and ID number, the second judgment factor: the combination of name and mobile phone number, the third judgment factor: the combination of name, date of birth and organization identification code, and the fourth judgment factor: the combination of name and company name. ;

The judgment strategy is: compare the data to be compared through the above four judgment factors. If in the data to be compared, one of the above four judgment factors matches, the data to be compared is determined to be conflict data.

In this embodiment, four judgment factors are generated based on the foregoing multiple target fields, and each judgment factor is composed of different target fields. Then, the data to be compared is compared through these four judgment factors. If one judgment factor in the data to be compared matches any judgment factor of the target data, the data to be compared is determined to be conflicting data.

Generate judgment factors: According to the preferred judgment strategy, extract the name, ID number, mobile phone number, date of birth, organization identification code and company name fields from the target data, and use the first judgment factor, the second judgment factor, and the third judgment factor to The combination of the judgment factor and the fourth judgment factor generates four judgment factors.

The first judgment factor: consists of name and ID number.

The second judgment factor: consists of name and mobile phone number.

The third judgment factor: consists of name, date of birth and identification code of the organization to which it belongs.

The fourth judgment factor: consists of name and company name.

When comparing data, for the data to be compared, four judgment factors are also generated according to the combination of the first judgment factor, the second judgment factor, the third judgment factor and the fourth judgment factor.

Conflict data judgment: Compare the four judgment factors of the data to be compared with the four judgment factors of the target data. If there is a judgment factor in the data to be compared that matches any judgment factor of the target data, the data to be compared is determined to be conflicting data.

Through this optimal judgment strategy, conflicting data can be effectively identified, that is, data with the same entity but inconsistent information. Such data processing method helps to improve data quality and credibility, and is of great significance in subsequent data management and application processes.

Furthermore, this embodiment also provides a more specific implementation method, that is, for the fourth judgment factor, during data processing, the similarity of the company name is judged because in actual situations, there may be certain differences in the company name. Variations, typos, abbreviations, etc. result in company names expressed in different ways actually referring to the same company. This diversity and difference of company names may lead to misjudgments or omissions in the data fusion or matching process, affecting the accuracy and consistency of the data. Therefore, for the fourth judgment factor, if it is determined that the names are consistent, it is necessary to calculate the semantic similarity between the company names. If the calculated semantic similarity meets the preset threshold, it is determined that the fourth judgment factor matches, thereby confirming the data to be compared. for conflicting data.

Name matching is performed according to the above-mentioned preferred judgment strategy. After it is determined that the names are consistent, semantic similarity calculation is performed between the company name in the comparison data and the company name in the target data. Semantic similarity calculation can use natural language processing technology, such as text matching algorithms, word vector models, etc., to measure the semantic similarity between two business names. A preset semantic similarity threshold can be set to determine whether the company name of the data to be compared is similar to the company name of the target data to a certain extent. Compare the calculated semantic similarity with a preset threshold. If the semantic similarity meets the preset threshold, that is, the semantic similarity between the company names is higher than the set threshold, then the fourth judgment factor is determined to match, and the data to be compared is regarded as conflicting data. Otherwise, the fourth judgment factor does not match, and the processing ends.

S106. When there is conflicting data, identify the business application scenario of the current data, and select an authoritative source judgment strategy based on the identification results;

In step S106, when conflicting data is discovered, the business application scenario of the current data needs to be identified, and an appropriate authority source judgment strategy is selected based on this scenario to process the conflicting data, thereby determining the trusted source authority level of the field. Such processing can ensure the accuracy and credibility of data in different business application scenarios.

In this step, it is necessary to determine the business application scenarios of the current data based on the business needs and application scenarios of data management. This scenario may involve different data processing objectives, data usage methods, and requirements for data accuracy. Based on the identified business application scenarios, a set of authoritative source judgment strategies are predefined. These strategies may match the needs of different business application scenarios to ensure the authority and credibility of the data in that scenario. Based on the selected authoritative source judgment strategy, authoritative judgment is made on each field in the conflict data. Different authoritative evaluation algorithms, data source considerations, and other methods may be used to determine the authoritative level of a field's trusted source. Based on the authoritative judgment results of the conflicting data fields, the authority level information of the corresponding fields in the database is updated. In this way, during subsequent data processing and application, corresponding decisions and uses can be made based on the authority level of the field.

For step S106, this application provides two possible application data processing scenarios. One is a data processing scenario where a historical database already exists. For this scenario, the data processing goal is to insert new data into the historical database. This Sometimes it is necessary to adjust the authoritative source level of the data. In this scenario, the authoritative source judgment strategy is:

If the current business application scenario is to process data from an existing historical database, select the first authoritative source judgment strategy. The first authoritative source judgment strategy includes:

Query the pre-built data authority source level table, use the data with the highest authority level from the conflict data as the business primary key according to the query results, and update the historical database.

This application provides a first authoritative source judgment strategy. The first authoritative source judgment strategy is described below:

Build a data authority source level table in advance to record the authority level information of different data sources. This table contains the identification of different data sources and the corresponding authority levels, which are used to indicate the trustworthiness of the data source. For the discovered conflicting data, the authority level information of each data source involved in the conflicting data is obtained by querying the data authority source level table.

Based on the query results of the data authority source level table, select the data source with the highest authority level as the source of the business primary key. That is, in conflicting data, the information provided by the data source with the highest authoritative level shall prevail, and this data shall be used as the business primary key. The information in the conflicting data, identified using the business primary key, is then updated to the historical database. In this way, the corresponding data in the historical database will be updated with information from the highest authoritative data source, ensuring the authority and accuracy of the data.

By selecting the first authoritative source judgment strategy and determining the business primary key based on the data authority source level table, you can ensure that the data processing in the historical database is authoritative and credible. This strategy makes full use of the authoritative level information of the data, giving priority to the most credible data sources for data updates, avoiding data confusion and errors, and at the same time improving the quality and availability of historical database data.

For another data processing scenario, the goal is to process real-time data. In this scenario, the authoritative source judgment strategy is required to respond to real-time data processing. Therefore, this application also provides a strategy, which is defined in this article as the second Authoritative source judgment strategy, the second authoritative source judgment strategy is described below:

If the current business application scenario is for warehousing and processing of real-time data, then select the second authoritative judgment strategy. The second authoritative judgment strategy includes:

Query the pre-built field authority source level table and compare the authority source levels corresponding to each field in the conflict data. Based on the comparison results, select the field with the highest authority source level for each field and generate the result data. The data is stored in the database, and the field authoritative source level is recorded with the level of the authoritative source of each field.

First, a field authority source level table is built in advance to record the authority source level information of each field. The table contains the identification of each field and the corresponding authority level, which is used to indicate the trustworthiness of each field. For the discovered conflicting data, query the field authority source level table to obtain the authority level information of each field in the conflicting data. Based on the query results, compare each field in the conflicting data and select the field with the highest authoritative source level as the result data. That is, in the conflict data, the information of the field with the highest authority level shall prevail to generate the result data. The selected result data is processed and stored in the database. In this way, the data stored in the database will be based on the field information with the highest authority level, ensuring the authority and credibility of the data stored in the database.

The second authoritative source judgment strategy is suitable for real-time data processing scenarios and can process and store conflicting data in a timely manner. Since real-time data may change frequently in a short period of time, using a second authoritative source judgment strategy can quickly process the data when it arrives, update the database in a timely manner, and maintain the real-time nature and flexibility of the data.

This strategy is dynamically selected based on the authoritative source level of each field in the actual conflict data, and is suitable for different data scenarios and business needs. This adaptability can cope with changes in data sources and diversity of business scenarios, improving the flexibility and adaptability of data processing.

By selecting fields with the highest authoritative source level as result data to be stored in the database, the storage structure of the database can be optimized. Only the most credible data information is retained, reducing the storage of redundant data and optimizing the space utilization efficiency of the database.

S107. Process the trusted source authority level of each field in the conflict data according to the authoritative source judgment strategy, and update the database.

In step S107, the authoritative source authority level of each field in the conflict data is processed according to the selected authoritative source judgment strategy, and the processed data is updated to the database. Such a process helps ensure that the data in the database is authoritative and reliable.

In this embodiment, the authoritative level of the trusted source of each field in the conflicting data is processed according to the selected authoritative source judgment strategy. Update the trusted source authority level information of each field in the processed conflict data to the database. In this way, the data stored in the database will contain the latest authoritative level information of trusted sources, ensuring the authority and reliability of the data in the database.

When updating the database, conflicting data needs to be merged to merge inconsistent data of the same entity into one record. During the merging process, field information with the highest authority level is given priority to ensure the authority and accuracy of the final data.

S108. If there is no conflicting data, generate a unique identification code for the current data and store it in the database.

In step S107, if no conflicting data is found in step S105, it means that the current data does not require further fusion and processing, and the data is already unique. In this step, a unique identification code can be generated for the current data and the data can be stored in the database, that is, the data can be saved to the database.

For the current data, a certain algorithm or rule can be used to generate a unique identification code. The identification code can be a number, string or other form of unique identifier used to distinguish different data entities.

The above provides an embodiment of the data processing method in this application, aiming to solve data quality problems and information confusion that may occur in the process of data management from multiple data sources. The method includes three stages: cleaning, fusion, and aggregation. Each stage is configured with corresponding algorithms and steps to ensure the accuracy and consistency of the data.

The advantages of this embodiment are:

Effectively handle data quality issues: By pre-building a dictionary of quality issues and fusion rules, data quality issues can be efficiently identified and dealt with, improving the accuracy and credibility of the data.

Adaptive processing of different data scenarios: Select different authoritative source judgment strategies according to business application scenarios, so that data processing can flexibly adapt to different data sources and business needs.

Data aggregation and fusion improve data integrity: through aggregation rules and fusion rules, the integrity and consistency of data are ensured and data quality is improved.

Database optimization and unique identification code generation: Optimize the database storage structure, improve the space utilization efficiency of the database, and generate a unique identification code for each piece of data to ensure the uniqueness of the data.

For the second authoritative source judgment strategy provided in the above embodiment, this application also provides a dynamic update method for the authoritative source level. This method can automatically update the authoritative source level of each field to a higher level, thereby ensuring Data modification permissions will not be confused. The method is described below. The method includes the following steps: In the method provided by this embodiment, after the result data is entered into the database, based on the pre-constructed field authoritative source level table, modifications are set for the data after being entered into the database. Rules, the modification rules include:

S001. Identify the currently modified target field, and query the corresponding field authoritative source registration table based on the modified field;

S002. Compare the current authoritative source level of the modified user with the current authoritative source level of the target field according to the query results;

S003. If the authoritative source level of the modifying user is greater than or equal to the current authoritative source level of the target field, the modification is allowed;

S004. If the authoritative source level of the modifying user is less than the current authoritative source level of the target field, the modification will be rejected.

S005. After modification, update the current authoritative source level of the target field to the authoritative source level of the modifying user.

In the above embodiment, the dynamic update method ensures that the data modification permissions are not confused and prevents low-privilege users from mistakenly modifying high-privilege fields, thereby maintaining the authority and integrity of the data. The specific description of this method is as follows:

Set modification rules based on the field authority source level table: First, after the data is entered into the database, set modification rules for the entered data based on the pre-built field authority source level table. The field authority source level table records the authority source level information of each field and is used to indicate the credibility of each field.

When a user modifies data in the database, the system will identify the currently modified target field. According to the modified target field, query the corresponding field authority source level information in the field authority source level table. Get the authoritative source level of the current modifying user and compare it with the current authoritative source level of the target field. If the authoritative source level of the current modifying user is greater than or equal to the current authoritative source level of the target field, the modification is allowed. Otherwise, the modification is rejected.

In the aforementioned dynamic update method, if it is confirmed that modification is allowed after comparing the permission level, then the modification operation on the target field is performed.

After modification, the current authoritative source level of the target field is updated to the authoritative source level of the modifying user. In this way, the authority source level of the target field will be updated with the permission level of the modifying user, maintaining the consistency of data management. This ensures that the data will not be modified by users with a lower authority source level.

By updating the current authoritative source level of the target field to the authoritative source level of the modifying user, dynamic adjustment of data permissions is achieved. Only users with sufficiently high permissions can modify the data, and the authority source level of the modified field will be automatically updated to ensure that the authority and integrity of the data are maintained. Such a dynamic update method helps ensure the security and reliability of data management and provides a more flexible way to control data permissions.

The above embodiments describe the data processing method in this application. The following describes the devices, systems and storage media involved in this application:

Referring to Figure 4, this application provides a data processing device, including:

The quality problem dictionary building module 401 is used to build a quality problem dictionary based on the experience database. The quality problem dictionary contains data quality problems and corresponding problem identifiers;

The target data problem identification module 402 is used to obtain the data to be processed and identify the target data problems existing in the data to be processed;

The target problem identification query module 403 is used to query the target data problem in the quality problem dictionary and obtain the corresponding target problem identification;

The target problem identification adding module 404 is used to add the target problem identification to the target data through a predefined field;

The target field judgment and conflict detection module 405 is used to read multiple target fields in the target data, judge the multiple target fields according to the preset fusion rules, and determine whether there is conflicting data based on the judgment results;

The authoritative source judgment strategy selection module 406 is used to identify the business application scenario of the current data when there is conflicting data, and select an authoritative source judgment strategy based on the identification results;

The conflict data processing module 407 is configured to process the trusted source authority level of each field in the conflict data according to the authoritative source judgment policy, and update the database.

Optionally, the target field judgment and conflict detection module 405 is specifically used to:

Four judgment factors are generated based on the aforementioned multiple target fields. The four judgment factors are:

The first judgment factor: the combination of name and ID number, the second judgment factor: the combination of name and mobile phone number, the third judgment factor: the combination of name, date of birth and organization identification code, and the fourth judgment factor: the combination of name and company name. ;

The judgment strategy is: compare the data to be compared through the above four judgment factors. If in the data to be compared, one of the above four judgment factors matches, the data to be compared is determined to be conflict data.

Optionally, the target field judgment and conflict detection module 405 is specifically used to:

When it is determined that the names are consistent, the semantic similarity between the company names in the data to be compared is calculated;

If the semantic similarity meets the preset threshold, it is determined that the fourth judgment factor matches.

Optionally, the authoritative source judgment strategy selection module 406 is specifically configured to: if the current business application scenario is to process data from an existing historical database, select a first authoritative source judgment strategy. The first authoritative source judgment strategy Strategies include:

Query the pre-built data authority source level table, use the data with the highest authority level from the conflict data as the business primary key according to the query results, and update the historical database.

Optionally, the authoritative source judgment strategy selection module 406 is specifically configured to: if the current business application scenario is for warehousing and processing of real-time data, select a second authoritative judgment strategy. The second authoritative judgment strategy includes:

Query the pre-built field authority source level table and compare the authority source levels corresponding to each field in the conflict data. Based on the comparison results, select the field with the highest authority source level for each field and generate the result data. The data is stored in the database, and the field authoritative source level is recorded with the level of the authoritative source of each field.

Optionally, a modification rule setting module 408 is also included for:

After the result data is entered into the database, modification rules are set for the entered data based on the pre-constructed field authority source level table. The modification rules include:

Identify the currently modified target field, and query the corresponding field authoritative source registration table based on the modified field;

Compare the authoritative source level of the current modifying user with the current authoritative source level of the target field based on the query results;

If the authoritative source level of the modifying user is greater than or equal to the current authoritative source level of the target field, the modification is allowed, otherwise the modification is rejected.

Also used for:

If the modification is allowed, after the modification, the current authoritative source level of the target field is updated to the authoritative source level of the modifying user.

Optionally, a unique identification code generation module 409 is also included for:

If there is no conflicting data, a unique identification code is generated for the current data and stored in the database.

Also included is a unified pre-cleaning module 410 for:

Through predetermined pre-cleaning rules, the data to be processed is uniformly pre-cleaned. During the pre-cleaning process, target data that cannot be uniformly pre-cleaned and corresponding target data problems are marked.

Referring to Figure 5, this application also provides a data processing system, including:

Processor 501, memory 502, input and output unit 503, bus 504;

The processor 501 is connected to the memory 502, the input and output unit 503 and the bus 504;

The memory 502 stores a program, and the processor 501 calls the program to execute any of the above data processing methods.

The present application also relates to a computer-readable storage medium. A program is stored on the computer-readable storage medium. The characteristic is that when the program is run on a computer, it causes the computer to perform any of the above methods.

Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be described again here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), magnetic disk or optical disk and other media that can store program code.

Claims (9)

1. A method of data processing, the method comprising:

constructing a quality problem dictionary according to an experience database, wherein the quality problem dictionary comprises data quality problems and corresponding problem identifiers;

acquiring data to be processed, and identifying a target data problem in the data to be processed;

inquiring the target data problem in the quality problem dictionary to obtain a corresponding target problem identifier;

adding the target problem identification to the target data through a predefined field;

reading a plurality of target fields in the target data, the plurality of target fields comprising: name, ID card number, mobile phone number, birth date, affiliated organization identification code and enterprise name, and generating four judgment factors according to the target fields, wherein the four judgment factors are respectively:

first judgment factor: name and identification card number combination, second judgment factor: name and phone number combination, third judgment factor: name and date of birth and the organization identifier code combination to which it belongs, and a fourth judgment factor: name and business name combinations;

the judgment strategy is as follows: comparing the data to be compared through the four judging factors, and if one judging factor of the four judging factors is matched in the data to be compared, determining the data to be compared as conflict data;

The method for judging the fourth judgment factor matching comprises the following steps:

after the names are consistent, calculating semantic similarity between enterprise names in the data to be compared;

if the semantic similarity meets a preset threshold, determining that the fourth judgment factor is matched;

when conflict data exists, the current service application scene of the data is identified, and an authoritative source judgment strategy is selected according to the identification result;

processing the authority level of the trusted source of each field in the conflict data according to the authority source judging strategy, and updating a database;

if no conflict data exists, generating a unique identification code for the current data, and warehousing.

2. The data processing method according to claim 1, wherein identifying a service application scenario of the current data, and selecting an authoritative source determination policy based on the identification result comprises:

if the current business application scene is used for processing the data of the existing historical database, selecting a first authority source judgment strategy, wherein the first authority source judgment strategy comprises the following steps:

and inquiring a pre-constructed data authority source level table, taking the data with the highest authority level from the conflict data as a business primary key according to an inquiry result, and updating the historical database.

3. The data processing method according to claim 1, wherein identifying a service application scenario of the current data, and selecting an authoritative source determination policy based on the identification result comprises:

if the current business application scene is used for carrying out warehouse entry processing on real-time data, selecting a second authority judgment strategy, wherein the second authority judgment strategy comprises the following steps:

inquiring a field authority source grade table constructed in advance, comparing authority source grades corresponding to each field in the conflict data, selecting the field with the highest authority source grade for each field according to a comparison result, generating result data, and warehousing the result data, wherein the field authority source grade records the authority source grade of each field.

4. A data processing method according to claim 3, wherein, when the result data is put in storage, a modification rule is set for the put data based on a field authority source level table constructed in advance, the modification rule comprising:

identifying a currently modified target field, and inquiring a corresponding field authority source registry according to the modified field;

comparing the current authority source level of the current modification user with the current authority source level of the target field according to the query result;

And if the authority source level of the modification user is greater than or equal to the current authority source level of the target field, allowing modification, otherwise refusing modification.

5. The data processing method of claim 4, wherein if modification is allowed, updating the current authoritative source class of the target field to the authoritative source class of the modifying user after modification.

6. The data processing method according to claim 1, wherein the identifying the target data problem existing in the data to be processed includes:

and carrying out unified pre-cleaning treatment on the data to be treated through a preset pre-cleaning rule, and marking target data and corresponding target data problems which cannot be subjected to the unified pre-cleaning treatment in the pre-cleaning treatment process.

7. A data processing apparatus, the apparatus comprising:

the quality problem dictionary module is used for constructing a quality problem dictionary according to the experience database, wherein the quality problem dictionary comprises data quality problems and corresponding problem identifiers;

the target data problem identification module is used for acquiring data to be processed and identifying target data problems in the data to be processed;

The target problem identification inquiring module is used for inquiring the target data problem in the quality problem dictionary to obtain a corresponding target problem identification;

a target problem identification adding module for adding the target problem identification to the target data through a predefined field;

the target field judging and collision detecting module is used for reading a plurality of target fields in the target data, wherein the target fields comprise: name, ID card number, mobile phone number, birth date, affiliated organization identification code and enterprise name, and generating four judgment factors according to the target fields, wherein the four judgment factors are respectively:

first judgment factor: name and identification card number combination, second judgment factor: name and phone number combination, third judgment factor: name and date of birth and the organization identifier code combination to which it belongs, and a fourth judgment factor: name and business name combinations;

the judgment strategy is as follows: comparing the data to be compared through the four judging factors, and if one judging factor of the four judging factors is matched in the data to be compared, determining the data to be compared as conflict data;

the method for judging the fourth judgment factor matching comprises the following steps:

After the names are consistent, calculating semantic similarity between enterprise names in the data to be compared;

if the semantic similarity meets a preset threshold, determining that the fourth judgment factor is matched;

the authority source judgment policy selection module is used for identifying the current service application scene of the data when the conflict data exists, and selecting an authority source judgment policy according to the identification result;

the conflict data processing module is used for processing the credible source authority level of each field in the conflict data according to the authority source judgment strategy and updating a database;

and the unique identification code generation module is used for generating a unique identification code for the current data and warehousing when no conflict data exist.

8. A system for data processing, the apparatus comprising:

a processor, a memory, an input-output unit, and a bus;

the processor is connected with the memory, the input/output unit and the bus;

the memory holds a program which the processor invokes to perform the method of any one of claims 1 to 6.

9. A computer readable storage medium having a program stored thereon, which when executed on a computer performs the method of any of claims 1 to 6.