CN111797078A - Data cleaning method, model training method, device, storage medium and equipment - Google Patents

Abstract

The embodiment of the application discloses a data cleaning method, a model training method, a device, a storage medium and equipment, wherein data to be cleaned needing data cleaning can be firstly obtained, the cleaning requirement of the data to be cleaned is obtained, then a target cleaning rule used for carrying out data cleaning on the data to be cleaned is determined according to the obtained data to be cleaned, the cleaning requirement and a pre-trained cleaning rule classification model, and finally the data to be cleaned is subjected to data cleaning according to the determined target cleaning rule, so that the cleaning effect of the data to be cleaned meets the cleaning requirement. Therefore, as long as the cleaning rule classification model is obtained through pre-training, the cleaning rule classification model can be subsequently utilized to automatically clean data without excessive manual participation, so that the labor cost of data cleaning is reduced, and the efficiency of data cleaning is improved.

Description

Data cleaning method, model training method, device, storage medium and equipment

technical field

The present application relates to the technical field of data processing, and in particular, to a data cleaning method, a model training method, an apparatus, a storage medium and a device.

Background technique

At present, how to process massive data has become a test that electronic devices have to face, and the primary task of data processing is data cleaning. ". However, when performing data cleaning in related technologies, it often relies on manual domain knowledge, experience, etc., resulting in a large amount of human resource consumption and high labor costs for data cleaning.

SUMMARY OF THE INVENTION

The embodiments of the present application provide a data cleaning method, a model training method, an apparatus, a storage medium and equipment, which can reduce the labor cost of data cleaning.

In a first aspect, an embodiment of the present application provides a data cleaning method, which is applied to an electronic device, and the data cleaning method includes:

Obtain the data to be cleaned that needs to be cleaned;

Obtain the cleaning requirements of the data to be cleaned;

According to the data to be cleaned, the cleaning requirements and the pre-trained cleaning rule classification model, determine a target cleaning rule for performing data cleaning on the data to be cleaned;

Perform data cleaning on the data to be cleaned according to the target cleaning rule, so that the cleaning effect on the data to be cleaned meets the cleaning requirement;

The cleaning rule classification model uses the cleaning rule feature representing the cleaning rule as the target output, and the joint feature representing the sample data to be cleaned corresponding to the cleaning rule and the cleaning effect as the training input, and is obtained by model training.

In a second aspect, an embodiment of the present application provides a model training method, which is applied to an electronic device, and the model training method includes:

Acquire a plurality of cleaning rules, and acquire sample data to be cleaned corresponding to each of the cleaning rules;

Obtain the cleaning effect of performing data cleaning on the corresponding sample data to be cleaned by each of the cleaning rules;

acquiring the joint features of each of the sample data to be cleaned and the corresponding cleaning effects, and acquiring the cleaning rule features of each of the cleaning rules;

Taking each of the joint features as a training input, and using the cleaning rule feature corresponding to each of the joint features as a target output, model training is performed to obtain a cleaning rule classification model.

In a third aspect, an embodiment of the present application provides a data cleaning device, which is applied to electronic equipment, and the data cleaning device includes:

The data acquisition module is used to acquire the data to be cleaned that needs to be cleaned;

a requirement acquisition module, used to acquire the cleaning requirements of the data to be cleaned;

a rule determination module, configured to determine a target cleaning rule for performing data cleaning on the data to be cleaned according to the data to be cleaned, the cleaning requirement and a pre-trained cleaning rule classification model;

a data cleaning module, configured to perform data cleaning on the data to be cleaned according to the target cleaning rule, so that the cleaning effect on the data to be cleaned meets the cleaning requirement;

The cleaning rule classification model uses the cleaning rule feature representing the cleaning rule as the target output, and the joint feature representing the sample data to be cleaned corresponding to the cleaning rule and the cleaning effect as the training input, and is obtained by model training.

In a fourth aspect, an embodiment of the present application provides a model training device, which is applied to electronic equipment, and the model training device includes:

a first acquisition module, configured to acquire a plurality of cleaning rules, and acquire sample data to be cleaned corresponding to each of the cleaning rules;

a second acquisition module, configured to acquire the cleaning effect of performing data cleaning on the corresponding sample data to be cleaned by each of the cleaning rules;

a third acquisition module, configured to acquire the joint features of each of the sample data to be cleaned and their corresponding cleaning effects, and to acquire the cleaning rule features of each of the cleaning rules;

The model training module is configured to use each of the joint features as a training input, and use the cleaning rule feature corresponding to each of the joint features as a target output to perform model training to obtain a cleaning rule classification model.

In a fifth aspect, the embodiments of the present application provide a storage medium on which a computer program is stored, and when the computer program runs on a computer, the computer is made to execute the data cleaning method provided by the embodiments of the present application. steps, or cause the computer to execute the steps in the model training method provided by the embodiments of the present application.

In a sixth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, the memory has a computer program, and the processor is used to execute the data provided by the embodiment of the present application by calling the computer program Steps in the cleaning method, or perform the steps in the model training method provided by the embodiments of the present application.

In the embodiment of the present application, the electronic device may first obtain the data to be cleaned that needs to be cleaned, and obtain the cleaning requirement of the data to be cleaned, and then determine the classification model according to the obtained data to be cleaned, the cleaning requirement, and the pre-trained cleaning rule classification model. The target cleaning rules for cleaning the data to be cleaned are determined, and finally the data to be cleaned is cleaned according to the determined target cleaning rules, so that the cleaning effect of the data to be cleaned meets the cleaning requirements. Therefore, as long as the cleaning rule classification model is obtained by pre-training, the cleaning rule classification model can be used to automatically clean the data in the future without excessive manual participation, which not only reduces the labor cost of data cleaning, but also improves data cleaning. s efficiency.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 is a schematic structural diagram of a panoramic perception architecture provided by an embodiment of the present application.

FIG. 2 is a schematic flowchart of a data cleaning method provided by an embodiment of the present application.

FIG. 3 is another schematic flowchart of a data cleaning method provided by an embodiment of the present application.

FIG. 4 is a schematic diagram of obtaining target cleaning rules by an electronic device according to a cleaning rule classification model in an embodiment of the present application.

FIG. 5 is a schematic flowchart of a model training method provided by an embodiment of the present application.

FIG. 6 is another schematic flowchart of the model training method provided by the embodiment of the present application.

FIG. 7 is a schematic diagram of an application scenario of model training in an embodiment of the present application.

FIG. 8 is a schematic structural diagram of a data cleaning apparatus provided by an embodiment of the present application.

FIG. 9 is a schematic structural diagram of a model training apparatus provided by an embodiment of the present application.

FIG. 10 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

FIG. 11 is another schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

Please refer to the drawings, wherein the same component symbols represent the same components, and the principles of the present application are exemplified by being implemented in a suitable computing environment. The following description is based on illustrated specific embodiments of the present application and should not be construed as limiting other specific embodiments of the present application not detailed herein.

With the miniaturization and intelligence of sensors, electronic devices such as mobile phones and tablet computers integrate more and more sensors, such as light sensors, distance sensors, position sensors, acceleration sensors, and gravity sensors, and so on. Electronic devices can collect more data with less power consumption through the sensors they configure. At the same time, the electronic device will also collect data related to its own state and data related to the state of the user during the running process, and so on. Generally speaking, electronic devices can obtain data related to the external environment (such as temperature, light, location, sound, weather, etc.), data related to user status (such as posture, speed, usage habits, personal basic information, etc.) and electronic devices. Status-related data (such as power consumption, resource usage, network status, etc.). In this embodiment of the present application, these data that can be acquired by the electronic device are recorded as panoramic data.

In the embodiment of the present application, in order to be able to process the data obtained by the electronic device, a panoramic perception architecture is proposed. Please refer to FIG. 1. FIG. 1 is a schematic structural diagram of a panoramic perception architecture provided by an embodiment of the application, applied to an electronic device, and includes a bottom-up information perception layer, a data processing layer, a feature extraction layer, a context modeling layer, and an intelligent service layer.

As the bottom layer of the panoramic perception architecture, the information perception layer is used to obtain raw data that can describe various types of user scenarios, that is, panoramic data. The information perception layer is composed of a plurality of sensors for data collection, including but not limited to the distance sensor shown in the figure for detecting the distance between the electronic device and external objects, and the magnetic field information for detecting the environment in which the electronic device is located. Magnetic field sensor, light sensor for detecting light information of the environment where electronic equipment is located, acceleration sensor for detecting acceleration data of electronic equipment, fingerprint sensor for collecting user's fingerprint information, Hall for sensing magnetic field information Sensor, position sensor for detecting the current geographic location of the electronic device, gyroscope for detecting the angular velocity of the electronic device in various directions, inertial sensor for detecting the motion data of the electronic device, for sensing the attitude of the electronic device Information attitude sensors, barometers used to detect the air pressure of the environment where the electronic device is located, and heart rate sensors used to detect the user's heart rate information, etc.

As the second bottom layer of the panoramic perception architecture, the data processing layer is used to process the original data obtained by the information perception layer to eliminate noise and inconsistency in the original data. Among them, the data processing layer can perform data cleaning, data integration, data transformation, data reduction and other processing on the data obtained by the information perception layer.

As the middle layer of the panoramic perception architecture, the feature extraction layer is used to perform feature extraction on the data processed by the data processing layer, so as to extract the features included in the data. Among them, the feature extraction layer can extract features or process the extracted features by filtering method, packaging method, integration method and other methods.

The filtering method refers to filtering the extracted features to remove redundant feature data. The packing method is used to filter the extracted features. The integration method refers to the integration of multiple feature extraction methods to construct a more efficient and accurate feature extraction method for feature extraction.

As the next level of the panoramic perception architecture, the context modeling layer is used to construct a model according to the features extracted by the feature extraction layer, and the obtained model can be used to represent the state of the electronic device, the user state, or the environment state, etc. For example, the scenario modeling layer can construct a key value model, a pattern identification model, a graph model, an entity relationship model, an object-oriented model, etc. according to the features extracted by the feature extraction layer.

As the highest layer of the panoramic perception architecture, the intelligent service layer is used to provide intelligent services according to the model constructed by the context modeling layer. For example, the intelligent service layer can provide users with basic application services, can provide system intelligent optimization services for electronic devices, and can also provide users with personalized intelligent services.

In addition, the panoramic perception architecture also includes an algorithm library, which includes but is not limited to the illustrated Markov algorithm, implicit Dirichlet distribution algorithm, Bayesian classification algorithm, support vector machine, and K-means clustering algorithm. , K-Nearest Neighbor Algorithm, Conditional Random Field, Residual Network, Long Short-Term Memory Network, Convolutional Neural Network and Recurrent Neural Network.

Embodiments of the present application first provide a data cleaning method. The execution body of the data cleaning method may be the data cleaning apparatus provided in the embodiments of the present application, or an electronic device integrated with the data cleaning apparatus, wherein the data cleaning apparatus may adopt hardware or software implementation. The electronic device may be a device equipped with a processor and having processing capabilities, such as a smart phone, a tablet computer, a palmtop computer, a notebook computer, or a desktop computer.

Based on the data cleaning method provided by the embodiment of the present application, the information perception layer provides the collected panoramic data to the data processing layer; the data processing layer uses the panoramic data from the information perception layer as the data to be cleaned that needs to be cleaned, And perform data cleaning on it, and provide the cleaned data to the feature extraction layer; the feature extraction layer performs feature extraction on the data from the data processing layer to obtain features that can characterize the data, and provides the extracted features to the scenario modeling. layer; the scenario modeling layer is based on the features from the feature extraction layer, and uses the model obtained by modeling to represent the state of the electronic device or the user state or the environment state, etc.; finally, the intelligent service layer is constructed according to the scenario modeling layer. The model provides corresponding intelligent services, such as basic application services, system optimization services, personalized services, etc.

Please refer to FIG. 2, which is a schematic flowchart of a data cleaning method provided by an embodiment of the present application. The data cleaning method is implemented in the data processing layer of the panoramic perception architecture. As shown in FIG. 2, the data cleaning method provided by the embodiment of the present application The flow of the method can be as follows:

In 101, data to be cleaned that needs to be cleaned is obtained.

For example, the electronic device can obtain the data to be cleaned that needs to be cleaned locally, it can also obtain the data to be cleaned that needs to be cleaned from other electronic devices, and it can also obtain the data to be cleaned that needs to be cleaned from the network, etc. .

In 102, the cleaning requirements of the data to be cleaned are obtained.

Those of ordinary skill in the art can understand that real-world data is often multi-dimensional, incomplete, noisy and inconsistent. The purpose of data cleaning is to fill in missing values, smooth noise and identify outliers, Correct inconsistencies in data, etc.

In the embodiment of the present application, after acquiring the data to be cleaned that needs to be cleaned, the electronic device further acquires the cleaning requirement of the data to be cleaned. In layman's terms, the cleaning requirement describes the cleaning effect that the data to be cleaned needs to be cleaned. For example, the original data to be cleaned contains data of multiple dimensions, and these dimensions are often not independent, that is to say, maybe one of them There is an association between several dimensions, and maybe there is no need for me. In this way, the cleaning requirement of the data to be executed can be to reduce the dimension of the data to be cleaned to a specified dimension.

It can be understood by those skilled in the art that the cleaning requirement depends on the actual requirement for data processing performed by the electronic device, which is not specifically limited in this embodiment of the present application.

In 103, according to the data to be cleaned, the cleaning requirements and the pre-trained cleaning rule classification model, a target cleaning rule for performing data cleaning on the data to be cleaned is determined.

It should be noted that, in the embodiment of the present application, the electronic device is configured with a cleaning rule classification model for selecting which cleaning rule to clean the data to be cleaned, and the cleaning rule classification model uses the cleaning rule feature representing the cleaning rule as the target output. , and the joint features representing the sample data to be cleaned corresponding to the cleaning rules and their cleaning effects are used as training inputs, and are obtained by model training.

For example, all possible cleaning rules can be pre-integrated, and the sample data to be cleaned corresponding to each cleaning rule and its cleaning effect can be collected at the same time; Combined features of cleaning effect; then, each joint feature is used as training input, and the cleaning rule feature corresponding to each joint feature is used as target output, and model training is performed according to a preset training algorithm, so as to train to obtain which cleaning method is used to select The rule is a cleaning rule classification model for data cleaning of the data to be cleaned.

Therefore, after the electronic device obtains the data to be cleaned that needs to be cleaned, and obtains the cleaning requirement of the data to be cleaned, the electronic device can input the data to be cleaned and the cleaning requirement into the cleaning rule classification model, so that the cleaning rule classification model outputs A cleaning rule that can perform data cleaning on the data to be cleaned and the cleaning effect meets the cleaning requirements, and the cleaning rule is used as the target cleaning rule for data cleaning of the data to be cleaned.

In 104, data cleaning is performed on the data to be cleaned according to the target cleaning rule, so that the cleaning effect of the data to be cleaned meets the aforementioned cleaning requirements.

In the embodiment of the present application, after determining the target cleaning rule for cleaning the data to be cleaned, the electronic device can perform data cleaning on the data to be cleaned according to the target cleaning rule, so that the cleaning effect of the data to be cleaned meets the aforementioned cleaning requirements, Finally get the required data.

As can be seen from the above, in the embodiment of the present application, the electronic device can first obtain the data to be cleaned that needs to be cleaned, and obtain the cleaning requirements of the data to be cleaned, and then obtain the data to be cleaned, the cleaning requirements and the pre-trained cleaning rules according to the obtained data to be cleaned. The classification model determines the target cleaning rules for cleaning the data to be cleaned, and finally cleans the data to be cleaned according to the determined target cleaning rules, so that the cleaning effect of the data to be cleaned meets the cleaning needs. Therefore, as long as the cleaning rule classification model is obtained by pre-training, the cleaning rule classification model can be used to automatically clean the data in the future without excessive manual participation, which not only reduces the labor cost of data cleaning, but also improves data cleaning. s efficiency.

Please refer to FIG. 3 , which is another schematic flowchart of a data cleaning method provided by an embodiment of the present application. The data cleaning method can be applied to electronic equipment, and the process of the data cleaning method can include:

In 201, sensor data collected by a sensor is acquired, and the acquired sensor data is used as data to be cleaned.

It should be noted that an electronic device is usually configured with a variety of sensors, through which the environment where it is located, its own movement, and the like can be sensed. The sensors configured in the electronic device include but are not limited to gravity sensors, acceleration sensors, positioning sensors (such as satellite positioning sensors, base station positioning sensors, etc.), sound sensors, and light sensors.

However, the sensor data collected by these sensors are not all required by the electronic device, which requires the electronic device to clean the sensor data and clean the actual data from it.

Therefore, in this embodiment of the present application, the electronic device may acquire sensor data collected by sensors configured by itself, and use the acquired sensor data as data to be cleaned.

In 202, the cleaning requirements of the data to be cleaned are obtained.

Those of ordinary skill in the art can understand that real-world data is often multi-dimensional, incomplete, noisy and inconsistent. The purpose of data cleaning is to fill in missing values, smooth noise and identify outliers, Correct inconsistencies in data, etc.

In the embodiment of the present application, after acquiring the data to be cleaned that needs to be cleaned, the electronic device further acquires the cleaning requirement of the data to be cleaned. In layman's terms, the cleaning requirement describes the cleaning effect that the data to be cleaned needs to be cleaned. For example, the original data to be cleaned contains data of multiple dimensions, and these dimensions are often not independent, that is to say, maybe one of them There is an association between several dimensions, and maybe there is no need for me. In this way, the cleaning requirement of the data to be executed can be to reduce the dimension of the data to be cleaned to a specified dimension.

It should be noted that those of ordinary skill in the art can understand that the cleaning requirement depends on the actual requirement for data processing performed by the electronic device, which is not specifically limited in this embodiment of the present application.

In 203, the combined characteristics of the data to be cleaned and the cleaning requirement are obtained.

In 204, the obtained joint features are input into the cleaning rule classification model, and the cleaning rule features output by the cleaning rule classification model are obtained.

In 205, a cleaning rule matching the cleaning rule feature output by the cleaning rule classification model is determined as a target cleaning rule for performing data cleaning on the data to be cleaned.

It should be noted that, in the embodiment of the present application, the electronic device is configured with a cleaning rule classification model for selecting which cleaning rule to clean the data to be cleaned, and the cleaning rule classification model uses the cleaning rule feature representing the cleaning rule as the target output. , and the joint features representing the sample data to be cleaned corresponding to the cleaning rules and their cleaning effects are used as training inputs, and are obtained by model training.

For example, all possible cleaning rules can be pre-integrated, and the sample data to be cleaned corresponding to each cleaning rule and its cleaning effect can be collected at the same time; Combined features of cleaning effect; then, each joint feature is used as training input, and the cleaning rule feature corresponding to each joint feature is used as target output, and model training is performed according to a preset training algorithm, so as to train to obtain which cleaning method is used to select The rule is a cleaning rule classification model for data cleaning of the data to be cleaned.

Therefore, after the electronic device obtains the data to be cleaned that needs to be cleaned, and obtains the cleaning requirement of the data to be cleaned, the electronic device can input the data to be cleaned and the cleaning requirement into the cleaning rule classification model, so that the cleaning rule classification model outputs A cleaning rule that can perform data cleaning on the data to be cleaned and the cleaning effect meets the cleaning requirements, and the cleaning rule is used as the target cleaning rule for data cleaning of the data to be cleaned.

It should be noted that, in the embodiment of the present application, the data to be cleaned and the cleaning requirements are input into the cleaning rule classification model, not the data to be cleaned and the cleaning requirements themselves are input into the cleaning rule classification model, but the data to be cleaned can be represented. and cleaning requirement features are input into the cleaning rule classification model.

Therefore, in the embodiment of the present application, after acquiring the data to be cleaned and the cleaning requirement of the data to be cleaned, the electronic device further acquires the combined feature of the data to be cleaned and the cleaning requirement, and uses the combined feature to analyze the data to be cleaned and its cleaning requirements. Combined depth characterization of cleaning requirements.

After obtaining the joint features of the data to be cleaned and the cleaning requirements, the electronic device inputs the joint features into the pre-trained cleaning rule classification model for processing. On the other hand, the cleaning rule classification model processes the input joint features and outputs the corresponding cleaning rule features. The cleaning rule features represent the cleaning rules that can clean the data to be cleaned and the cleaning effect meets the cleaning requirements.

After the electronic device obtains the cleaning rule feature output by the cleaning rule classification model, it further determines a cleaning rule matching the cleaning rule feature, and uses the cleaning rule as a target cleaning rule for data cleaning of the data to be cleaned.

For example, referring to FIG. 4 , the electronic device obtains the joint feature A of the data to be cleaned and its cleaning requirements, processes the cleaning rule classification model inputted by the joint feature A, and obtains the cleaning rule feature A output by the cleaning rule classification model, Match out cleaning rule A as the target cleaning rule.

In 206, data cleaning is performed on the data to be cleaned according to the target cleaning rule, so that the cleaning effect of the data to be cleaned meets the cleaning requirement.

In the embodiment of the present application, after determining the target cleaning rule for cleaning the data to be cleaned, the electronic device can perform data cleaning on the data to be cleaned according to the target cleaning rule, and obtain the cleaning effect of the data to be cleaned that satisfies the corresponding cleaning effect. need.

In one embodiment, "obtaining the combined characteristics of data to be cleaned and cleaning requirements" may include:

According to the generative adversarial network, the joint features of the data to be cleaned and the cleaning requirements are obtained.

In the embodiment of the present application, considering that the generative adversarial network can generate more sample data based on the existing data, and has strong feature learning ability, the electronic device can obtain the data to be cleaned and the joint feature of the cleaning requirement according to the generative adversarial network. .

Among them, when the electronic device obtains the combined features of the data to be cleaned and the cleaning requirements, the data to be cleaned and the cleaning requirements are formed into a data pair, which is represented as <data to be cleaned, cleaning requirements>, and then constructed according to the generative adversarial network <Data to be cleaned, Cleaning Requirements > Joint Features.

It should be noted that, in other implementation manners, a person of ordinary skill in the art can select an appropriate feature construction manner according to actual requirements to obtain the combined features of the data to be cleaned and the cleaning requirements.

In one embodiment, "determining a cleaning rule that matches the cleaning rule feature output by the cleaning rule classification model" includes:

(1) Obtain the similarity between the cleaning rule features output by the cleaning rule classification model and the cleaning rule features of a plurality of pre-stored cleaning rules;

(2) Taking the cleaning rule whose similarity reaches the preset similarity as the cleaning rule matched with the cleaning rule feature output by the cleaning rule classification model.

It should be noted that, in the embodiment of the present application, the cleaning rule matched with the cleaning rule feature output by the cleaning rule classification model refers to the difference between the cleaning rule feature of the cleaning rule and the cleaning rule feature output by the cleaning rule classification model. The similarity reaches the preset similarity.

Therefore, when the electronic device determines the cleaning rule that matches the cleaning rule feature output by the cleaning rule classification model, it can first obtain the similarity between the cleaning rule feature output by the cleaning rule classification model and the cleaning rule features of multiple pre-stored cleaning rules and then take the cleaning rule whose similarity reaches the preset similarity as the cleaning rule matched with the cleaning rule feature output by the cleaning rule classification model, that is, the target cleaning rule used for subsequent data cleaning of the data to be cleaned.

For example, suppose that the electronic device has pre-stored cleaning rule feature A of cleaning rule A, cleaning rule feature B of cleaning rule B, and cleaning rule feature C of cleaning rule C, and the preset similarity is configured as 85%. If the similarity between the cleaning rule feature A of the cleaning rule A and the cleaning rule feature output by the cleaning rule classification model is 40%, the cleaning rule feature B of the cleaning rule B is similar to the cleaning rule feature output by the cleaning rule classification model. The degree of similarity is 45%, and the similarity between the cleaning rule feature C of the cleaning rule C and the cleaning rule feature output by the cleaning rule classification model is 86%. It can be seen that the cleaning rule feature C of the cleaning rule C and the cleaning rule classification model output cleaning The similarity of the rule features reaches a preset similarity (85%). At this time, the electronic device determines the cleaning rule C to match the cleaning rule features output by the cleaning rule classification model.

Wherein, when calculating the similarity between the two cleaning rule features, the electronic device can use the feature distance between the two cleaning rule features to measure the similarity between the two cleaning rule features, that is, to calculate the two The feature distance between cleaning rule features (any one of the feature distances can be selected by those of ordinary skill in the art according to actual needs, such as Euclidean distance, Manhattan distance, Chebyshev distance, and cosine distance, etc.), as one of the two cleaning rule features. similarity between. Any one of the characteristic distances can be selected by those of ordinary skill in the art according to actual needs.

In one embodiment, "performing data cleaning on the data to be cleaned according to the target cleaning rule" includes:

One or more cleaning functions corresponding to the target cleaning rule are called to clean the data to be cleaned.

It should be noted that, in the embodiments of the present application, each cleaning rule is composed of one or more cleaning functions, and the cleaning functions are used to actually implement cleaning operations, including but not limited to missing value processing, normalization processing, noise elimination processing, etc. Wait. Wherein, the cleaning function itself can be written by the relevant technical personnel using computer programming languages (such as C language, Java language, Python language, etc.), such as regular expressions, filter functions, SQL expressions, and the like.

Therefore, when the electronic device cleans the data to be cleaned according to the target cleaning rule, it can call one or more cleaning functions corresponding to the target cleaning rule to clean the data to be cleaned, so that the cleaning effect of the data to be cleaned meets the aforementioned cleaning requirements. Finally get the required data.

Please refer to FIG. 5. FIG. 5 is a model training method provided by an embodiment of the present application. The model training method is used to train a cleaning rule classification model required in the data cleaning method provided by the embodiment of the present application. The execution of the model training method The main body may be the model training apparatus provided in the embodiments of the present application, or an electronic device integrating the model training apparatus, where the model training apparatus may be implemented in hardware or software. As shown in FIG. 5 , the process of the model training method provided by the embodiment of the present application may be as follows:

In 301, a plurality of cleaning rules are obtained, and sample data to be cleaned corresponding to each cleaning rule is obtained.

In this embodiment of the present application, a cleaning rule-oriented database may be created on the electronic device in advance, where the cleaning rule-oriented database includes a cleaning rule sub-database, a sample data sub-database to be cleaned, and a cleaning effect sub-database.

During model training, the electronic device can integrate all possible cleaning rules and store these cleaning rules in the cleaning rules sub-database. For example, the electronic device stores the acquired cleaning rules in the cleaning rules sub-database in the form of character strings.

In addition, for the acquired cleaning rules and stored in the cleaning rule sub-database, the electronic device further acquires the sample data to be cleaned corresponding to each cleaning rule, and stores the sample data to be cleaned in the sample data to be cleaned sub-database. In the database, for example, the sample data to be cleaned is stored in the sample sub-database to be cleaned, and the sample data to be cleaned of the digital type is still stored in the sample sub-database to be cleaned in the digital type.

It should be noted that the electronic device can obtain the sample data to be cleaned locally, also can obtain the sample data to be cleaned from other electronic devices, and can also obtain the sample data to be cleaned from the Internet.

In 302, a cleaning effect of performing data cleaning on the corresponding sample data to be cleaned by each cleaning rule is obtained.

In the embodiment of the present application, after acquiring a plurality of cleaning rules and the sample data to be cleaned corresponding to each cleaning rule, the electronic device further acquires the cleaning effect of performing data cleaning on the corresponding sample data to be cleaned by each cleaning rule, and uses these cleaning rules. Cleaning effects are stored in the cleaning effects sub-database. For example, the cleaning effect can be stored in the cleaning effect sub-database in the form of a table.

In 303, a joint feature of each sample data to be cleaned and its corresponding cleaning effect is acquired, and a cleaning rule feature of each cleaning rule is acquired.

In the embodiment of the present application, for the obtained sample data to be cleaned and the corresponding cleaning effect, the electronic device also obtains the joint feature of each sample data to be cleaned and the corresponding cleaning effect, and uses the joint feature to analyze the sample data to be cleaned and the corresponding cleaning effect. The corresponding cleaning effect is characterized by joint depth.

In addition, the electronic device also acquires cleaning rule features of each cleaning rule, and uses the cleaning rule features to characterize the cleaning rules. For example, the electronic device may acquire lexical features of one or more cleaning functions corresponding to each cleaning rule, as cleaning rule features of each cleaning rule.

In 304, model training is performed using each joint feature as a training input and the cleaning rule feature corresponding to each joint feature as a target output to obtain a cleaning rule classification model.

In the embodiment of the present application, after obtaining the joint features of each sample data to be cleaned and their corresponding cleaning effects, and obtaining the cleaning rule features of each cleaning rule, the electronic device can use each joint feature as a training input, The cleaning rule feature corresponding to each joint feature is used as the target output, and the model is trained according to the preset training algorithm to obtain a cleaning rule classification model for automatically selecting cleaning rules.

Among them, the training algorithm is a machine learning algorithm, and the machine learning algorithm can realize various functions through continuous feature learning. For example, a data to be cleaned and its corresponding cleaning requirements can be given, and automatic selection can be performed on the data to be cleaned. Cleaning rules for data cleaning and the cleaning effect meets the cleaning requirements. Machine learning algorithms can include: decision tree models, logistic regression models, Bayesian models, neural network models, clustering models, and the like.

In addition, the algorithm types of machine learning algorithms can be divided according to various situations. For example, machine learning algorithms can be divided into: supervised learning algorithms, unsupervised learning algorithms, semi-supervised learning algorithms, reinforcement learning algorithms, etc. Wait.

Under supervised learning, the input data is called "training data", and each set of training data has a clear identification or result, such as "spam" and "non-spam" in the anti-spam system, and for handwritten digit recognition. "1", "2", "3", "4" etc. When establishing the recognition model, supervised learning establishes a learning process, compares the scene type information with the actual results of the "training data", and continuously adjusts the recognition model until the scene type information of the model reaches an expected accuracy rate. Common application scenarios of supervised learning are classification problems and regression problems. Common algorithms are Logistic Regression and Back Propagation Neural Network.

In unsupervised learning, the data is not specifically identified and the model is identified to infer some intrinsic structure of the data. Common application scenarios include the learning of association rules and clustering. Common algorithms include Apriori algorithm and k-Means algorithm.

Semi-supervised learning algorithm, in which the input data is partially identified, this learning model can be used for type recognition, but the model first needs to learn the internal structure of the data in order to organize the data reasonably to make predictions. Application scenarios include classification and regression, and algorithms include some extensions to commonly used supervised learning algorithms that first attempt to model unlabeled data and then make predictions on labeled data. Graph Inference or Laplacian SVM, etc.

Reinforcement learning algorithm. In this learning mode, the input data is used as feedback to the model. Unlike the supervised model, the input data is only used as a way to check whether the model is right or wrong. In reinforcement learning, the input data is directly fed back to the model. The model must adjust to this immediately. Common application scenarios include dynamic systems and robot control. Common algorithms include Q-Learning and Temporal difference learning.

In addition, machine learning algorithms can also be divided into:

Regression algorithms, common regression algorithms include: Ordinary Least Square, Logistic Regression, Stepwise Regression, Multivariate Adaptive Regression Splines and Local Scatter Smoothing Estimated (LocallyEstimated Scatterplot Smoothing).

Instance-based algorithms include k-Nearest Neighbor (KNN), Learning Vector Quantization (LVQ), and Self-Organizing Map (SOM).

Regularization methods, common algorithms include: Ridge Regression, Least Absolute Shrinkage and Selection Operator (LASSO), and Elastic Net.

Decision tree algorithm, common algorithms include: Classification and Regression Tree (CART), ID3 (Iterative Dichotomiser 3), C4.5, Chi-squared Automatic Interaction Detection (CHAID), Decision Stump, Random Forest (Random Forest) , Multivariate Adaptive Regression Splines (MARS) and Gradient Boosting Machine (GBM).

Bayesian method algorithms, including: Naive Bayesian algorithm, Averaged One-Dependence Estimators (AODE), and Bayesian Belief Network (BBN).

Please refer to FIG. 6 , which is another schematic flowchart of a model training method provided by an embodiment of the present application. The model training method can be applied to electronic equipment, and the process of the model training method can include:

In 401, a plurality of cleaning rules and sample data to be cleaned corresponding to each cleaning rule are acquired.

In this embodiment of the present application, a cleaning rule-oriented database may be created on the electronic device in advance, where the cleaning rule-oriented database includes a cleaning rule sub-database, a sample data sub-database to be cleaned, and a cleaning effect sub-database.

During model training, the electronic device can integrate all possible cleaning rules and store these cleaning rules in the cleaning rules sub-database. For example, the electronic device stores the acquired cleaning rules in the cleaning rules sub-database in the form of character strings.

In addition, for the acquired cleaning rules and stored in the cleaning rule sub-database, the electronic device further acquires the sample data to be cleaned corresponding to each cleaning rule, and stores the sample data to be cleaned in the sample data to be cleaned sub-database. In the database, for example, the sample data to be cleaned is stored in the sample sub-database to be cleaned, and the sample data to be cleaned of the digital type is still stored in the sample sub-database to be cleaned in the digital type.

It should be noted that the electronic device can obtain the sample data to be cleaned locally, also can obtain the sample data to be cleaned from other electronic devices, and can also obtain the sample data to be cleaned from the Internet.

In 402, a cleaning effect of performing data cleaning on the corresponding sample data to be cleaned by each cleaning rule is obtained.

In the embodiment of the present application, after acquiring a plurality of cleaning rules and the sample data to be cleaned corresponding to each cleaning rule, the electronic device further acquires the cleaning effect of performing data cleaning on the corresponding sample data to be cleaned by each cleaning rule, and uses these cleaning rules. Cleaning effects are stored in the cleaning effects sub-database. For example, the cleaning effect can be stored in the cleaning effect sub-database in the form of a table.

In 403, the joint features of each sample data to be cleaned and their corresponding cleaning effects are obtained according to the generative adversarial network.

In the embodiment of the present application, for the obtained sample data to be cleaned and the corresponding cleaning effect, the electronic device also obtains the joint feature of each sample data to be cleaned and the corresponding cleaning effect, and uses the joint feature to analyze the sample data to be cleaned and the corresponding cleaning effect. The corresponding cleaning effect is characterized by joint depth.

Considering that the generative adversarial network can generate more sample data based on the existing data, and has strong feature learning ability, the electronic device can obtain the joint feature of the sample data to be cleaned and the cleaning effect according to the generative adversarial network.

Wherein, when the electronic device obtains the joint features of the sample data to be cleaned and the corresponding cleaning effect, the sample data to be cleaned and the corresponding cleaning effect are formed into a data pair, which is represented as <sample data to be cleaned, cleaning effect>, and then according to Generative adversarial network constructs joint features of <sample data to be cleaned, cleaning effect>.

In 404, lexical features of one or more cleaning functions corresponding to each cleaning rule are acquired according to the encoder neural network, as cleaning rule features of each cleaning rule.

In the embodiment of the present application, the electronic device further acquires the cleaning rule feature of each cleaning rule, and uses the cleaning rule feature to characterize the cleaning rule.

It should be noted that, in the embodiments of the present application, each cleaning rule is composed of one or more cleaning functions, and the cleaning functions are used to actually implement cleaning operations, including but not limited to missing value processing, normalization processing, noise elimination processing, etc. Wait. Wherein, the cleaning function itself can be written by the relevant technical personnel using computer programming languages (such as C language, Java language, Python language, etc.), such as regular expressions, filter functions, SQL expressions, and the like.

When the electronic device acquires the cleaning rule characteristics of each cleaning rule, for any cleaning rule, the electronic device performs word segmentation on one or more cleaning functions corresponding to the cleaning rule, obtains the vocabulary sequence of the cleaning rule, and then inputs the word sequence into the word sequence. Go to the encoder neural network for encoding processing, and obtain a lexical feature vector with representational ability, which is used as the cleaning rule feature of the aforementioned cleaning rule.

For example, for a cleaning rule, the electronic device performs word segmentation on it to obtain a word sequence C=(c ₁ , c ₂ , ..., c _n ), and after encoding the word sequence C into the encoder neural network to obtain The lexical feature vector V=(v ₁ , v ₂ , ... v _n ), and the lexical feature vector V is used as the cleaning rule feature of the cleaning rule.

It should be noted that the embodiments of the present application do not limit the specific model and topology of the encoder neural network. A single-layer recurrent neural network can be used for training to obtain an encoder neural network, or a multi-layer recurrent neural network can be used for training. To obtain the encoder neural network, a convolutional neural network, a variant thereof, or a neural network with other network structures can also be used for training to obtain an encoder neural network. For example, in this embodiment of the present application, a recurrent neural network may be used to construct an encoder neural network.

In 405, each joint feature is used as a training input, and the cleaning rule feature corresponding to each joint feature is used as a target output, and a conditional recurrent neural network is used for model training to obtain a cleaning rule classification model.

In the embodiment of the present application, after obtaining the joint features of each sample data to be cleaned and their corresponding cleaning effects, and obtaining the cleaning rule features of each cleaning rule, the electronic device can use each joint feature as a training input, The cleaning rule feature corresponding to each joint feature is used as the target output, and the conditional recurrent neural network is used for model training to obtain a cleaning rule classification model for automatically selecting cleaning rules.

For a clearer understanding of the embodiment of the present application, please refer to FIG. 7 , which is a schematic diagram of an application scenario of model training in the embodiment of the present application.

First, a database for cleaning rules is constructed, including three sub-databases, namely, a sub-database of cleaning rules, a sub-database of sample data to be cleaned, and a sub-database of cleaning effects. Integrate all possible cleaning rules, collect the sample data to be cleaned corresponding to each cleaning rule and its cleaning effect, store the cleaning rules in the form of strings in the cleaning rules sub-database, and store the sample data to be cleaned in the sample to be cleaned. In the data sub-database, the cleaning effect is stored in the cleaning effect sub-database in the form of a table.

Secondly, the encoder neural network constructed by the recurrent neural network is used to encode all cleaning rules, and the corresponding lexical feature vector is obtained, which is used as the cleaning rule feature of each cleaning rule. At the same time, a data pair is constructed between the sample data to be cleaned and its cleaning effect corresponding to each cleaning rule, which is expressed as <sample data to be cleaned, cleaning effect>, and the <sample data to be cleaned, cleaning effect> of each cleaning rule is generated by using the generative adversarial network. Effect> Perform learning to obtain <sample data to be cleaned, cleaning effect> joint features.

Finally, the joint feature of <sample data to be cleaned, cleaning effect> corresponding to each cleaning rule is used as the training input, and its lexical feature vector is used as the target output, and the conditional recurrent neural network is used for model training to obtain the cleaning rule classification model.

Therefore, as long as the data to be cleaned that needs to be cleaned and the trained cleaning rule classification model to which the cleaning requirements are input are input, the cleaning rules output by the cleaning rule classification model can be obtained, and the data to be cleaned can be cleaned by using the cleaning rules. The cleaning effect can meet the cleaning needs.

Embodiments of the present application further provide a data cleaning device. Please refer to FIG. 8 , which is a schematic structural diagram of a data cleaning apparatus provided by an embodiment of the present application. The data cleaning device is applied to electronic equipment, and the data cleaning device includes a data acquisition module 501, a demand acquisition module 502, a rule determination module 503 and a data cleaning module 504, as follows:

A data acquisition module 501, configured to acquire data to be cleaned that needs to be cleaned;

A requirement acquisition module 502, configured to acquire cleaning requirements of the data to be cleaned;

The rule determination module 503 is configured to determine the target cleaning rules for performing data cleaning on the data to be cleaned according to the data to be cleaned, the cleaning requirements and the pre-trained cleaning rule classification model;

The data cleaning module 504 is configured to perform data cleaning on the data to be cleaned according to the target cleaning rules, so that the cleaning effect of the data to be cleaned meets the aforementioned cleaning requirements;

Among them, the cleaning rule classification model uses the cleaning rule feature representing the cleaning rule as the target output, and the joint feature representing the to-be-cleaned sample data corresponding to the cleaning rule and its cleaning effect as the training input, and is obtained by model training.

In one embodiment, when determining a target cleaning rule for performing data cleaning on the data to be cleaned according to the data to be cleaned, the cleaning requirements and the pre-trained cleaning rule classification model, the rule determination module 503 can be used to:

Obtain the combined characteristics of the data to be cleaned and the cleaning requirements;

Input the obtained joint features into the cleaning rule classification model, and obtain the cleaning rule features output by the cleaning rule classification model;

A cleaning rule matching the cleaning rule feature output by the cleaning rule classification model is determined as a target cleaning rule for performing data cleaning on the data to be cleaned.

In one embodiment, when determining the cleaning rule matching the cleaning rule feature output by the cleaning rule classification model, the rule determination module 503 may be used to:

Obtaining the similarity between the cleaning rule features output by the cleaning rule classification model and the cleaning rule features of multiple pre-stored cleaning rules;

The cleaning rule whose similarity reaches the preset similarity is used as the cleaning rule matched with the cleaning rule feature output by the cleaning rule classification model.

In one embodiment, when performing data cleaning on the data to be cleaned according to the target cleaning rule, the data cleaning module 504 may be used to:

One or more cleaning functions corresponding to the target cleaning rule are called to clean the data to be cleaned.

In one embodiment, when acquiring data to be cleaned that needs to be cleaned, the data acquisition module 501 can be used to:

The sensor data collected by the sensor is acquired, and the acquired sensor data is used as the data to be cleaned.

The embodiment of the present application also provides a model training device. Please refer to FIG. 9 , which is a schematic structural diagram of a model training apparatus provided by an embodiment of the present application. The model training device is applied to electronic equipment, and the data cleaning device includes a first acquisition module 601, a second acquisition module 602, a third acquisition module 603 and a model training module 604, as follows:

The first acquisition module 601 is configured to acquire a plurality of cleaning rules, and acquire sample data to be cleaned corresponding to each cleaning rule;

The second acquiring module 602 is configured to acquire the cleaning effect of performing data cleaning on the corresponding sample data to be cleaned by each cleaning rule;

The third acquisition module 603 is configured to acquire the joint features of each sample data to be cleaned and its corresponding cleaning effect, and acquire the cleaning rule feature of each cleaning rule;

The model training module 604 is configured to use each joint feature as a training input, and use the cleaning rule feature corresponding to each joint feature as a target output to perform model training to obtain a cleaning rule classification model.

In one embodiment, when acquiring the cleaning rule features of each cleaning rule, the third acquiring module 603 may be used to:

The lexical features of one or more cleaning functions corresponding to each cleaning rule are acquired as cleaning rule features of each cleaning rule.

In one embodiment, when acquiring the lexical features of one or more cleaning functions corresponding to each cleaning rule, the third acquiring module 603 may be used to:

The lexical features of one or more cleaning functions corresponding to each cleaning rule are obtained according to the encoder neural network.

In one embodiment, when acquiring the joint features of the sample data to be cleaned and their corresponding cleaning effects, the third acquisition module 603 can be used to:

The joint feature of each sample data to be cleaned and its corresponding cleaning effect is obtained according to the generative adversarial network.

In one embodiment, when using each joint feature as a training input and the cleaning rule feature corresponding to each joint feature as a target output for model training to obtain a cleaning rule classification model, the model training module 604 can be used for:

Taking each joint feature as the training input and the cleaning rule feature corresponding to each joint feature as the target output, the conditional recurrent neural network is used for model training, and the cleaning rule classification model is obtained.

An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the stored computer program is executed on a computer, the computer is made to execute the steps in the data cleaning method provided by this embodiment, or The computer is caused to execute the steps in the model training method provided by this embodiment. The storage medium may be a magnetic disk, an optical disk, a read only memory (Read Only Memory, ROM,) or a random access device (Random Access Memory, RAM), and the like.

An embodiment of the present application further provides an electronic device, including a memory and a processor. The processor executes the steps in the data cleaning method provided by this embodiment by calling a computer program stored in the memory, or executes the model provided by this embodiment. Steps in the training method.

In one embodiment, an electronic device is also provided. Referring to FIG. 10 , the electronic device includes a processor 701 and a memory 702 . The processor 701 is electrically connected to the memory 702 .

The processor 701 is the control center of the electronic device, uses various interfaces and lines to connect various parts of the entire electronic device, executes the electronic device by running or loading the computer program stored in the memory 702, and calling the data stored in the memory 702. various functions and process data.

The memory 702 can be used to store software programs and modules, and the processor 701 executes various functional applications and data processing by running the computer programs and modules stored in the memory 702 . The memory 702 may mainly include a stored program area and a stored data area, wherein the stored program area may store an operating system, a computer program (such as a sound playback function, an image playback function, etc.) required for at least one function, and the like; Data created by the use of electronic equipment, etc. Additionally, memory 702 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, memory 702 may also include a memory controller to provide processor 701 access to memory 702 .

In this embodiment of the present application, the processor 701 in the electronic device loads the instructions corresponding to the processes of one or more computer programs into the memory 702 according to the following steps, and is executed by the processor 701 and stored in the memory 702 A computer program that implements various functions, as follows:

Obtain the data to be cleaned that needs to be cleaned;

Obtain the cleaning requirements of the data to be cleaned;

According to the data to be cleaned, the cleaning requirements and the pre-trained cleaning rule classification model, determine the target cleaning rules for data cleaning of the data to be cleaned;

Data cleaning is performed on the data to be cleaned according to the target cleaning rules, so that the cleaning effect of the data to be cleaned meets the aforementioned cleaning requirements;

Among them, the cleaning rule classification model uses the cleaning rule feature representing the cleaning rule as the target output, and the joint feature representing the to-be-cleaned sample data corresponding to the cleaning rule and its cleaning effect as the training input, and is obtained by model training.

Alternatively, the processor 701 in the electronic device loads the instructions corresponding to the processes of one or more computer programs into the memory 702 according to the following steps, and the processor 701 runs the computer program stored in the memory 702, thereby Implement various functions, as follows:

Obtain multiple cleaning rules, and obtain sample data to be cleaned corresponding to each cleaning rule;

Obtain the cleaning effect of data cleaning performed on the corresponding sample data to be cleaned by each cleaning rule;

Obtain the joint features of each sample data to be cleaned and its corresponding cleaning effect, and obtain the cleaning rule features of each cleaning rule;

Taking each joint feature as the training input, and taking the cleaning rule feature corresponding to each joint feature as the target output, model training is performed to obtain a cleaning rule classification model.

Please refer to FIG. 11 , which is another schematic structural diagram of the electronic device provided by the embodiment of the present application. The difference from the electronic device shown in FIG. 10 is that the electronic device further includes components such as an input unit 703 and an output unit 704 .

The input unit 703 can be used to receive input numbers, character information or user feature information (such as fingerprints), and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

The output unit 704 may be used to display information input by the user or information provided to the user, such as a screen.

In this embodiment of the present application, the processor 701 in the electronic device loads the instructions corresponding to the processes of one or more computer programs into the memory 702 according to the following steps, and is executed by the processor 701 and stored in the memory 702 A computer program that implements various functions, as follows:

Obtain the data to be cleaned that needs to be cleaned;

Obtain the cleaning requirements of the data to be cleaned;

According to the data to be cleaned, the cleaning requirements and the pre-trained cleaning rule classification model, determine the target cleaning rules for data cleaning of the data to be cleaned;

Data cleaning is performed on the data to be cleaned according to the target cleaning rules, so that the cleaning effect of the data to be cleaned meets the aforementioned cleaning requirements.

In one embodiment, when determining a target cleaning rule for performing data cleaning on the data to be cleaned according to the data to be cleaned, cleaning requirements, and a pre-trained cleaning rule classification model, the processor 701 may execute:

Obtain the combined characteristics of the data to be cleaned and the cleaning requirements;

Input the obtained joint features into the cleaning rule classification model, and obtain the cleaning rule features output by the cleaning rule classification model;

A cleaning rule matching the cleaning rule feature output by the cleaning rule classification model is determined as a target cleaning rule for performing data cleaning on the data to be cleaned.

In one embodiment, when determining the cleaning rule matching the cleaning rule feature output by the cleaning rule classification model, the processor 701 may execute:

Obtaining the similarity between the cleaning rule features output by the cleaning rule classification model and the cleaning rule features of multiple pre-stored cleaning rules;

The cleaning rule whose similarity reaches the preset similarity is used as the cleaning rule matched with the cleaning rule feature output by the cleaning rule classification model.

In one embodiment, when performing data cleaning on the data to be cleaned according to the target cleaning rule, the processor 701 may execute:

One or more cleaning functions corresponding to the target cleaning rule are called to clean the data to be cleaned.

In one embodiment, when acquiring data to be cleaned that needs to be cleaned, the processor 701 may execute:

The sensor data collected by the sensor is acquired, and the acquired sensor data is used as the data to be cleaned.

Alternatively, the processor 701 in the electronic device loads the instructions corresponding to the processes of one or more computer programs into the memory 702 according to the following steps, and the processor 701 runs the computer program stored in the memory 702, thereby Implement various functions, as follows:

Obtain multiple cleaning rules, and obtain sample data to be cleaned corresponding to each cleaning rule;

Obtain the cleaning effect of data cleaning performed on the corresponding sample data to be cleaned by each cleaning rule;

Obtain the joint features of each sample data to be cleaned and its corresponding cleaning effect, and obtain the cleaning rule features of each cleaning rule;

Taking each joint feature as the training input, and taking the cleaning rule feature corresponding to each joint feature as the target output, model training is performed to obtain a cleaning rule classification model.

In one embodiment, when acquiring the cleaning rule characteristics of each cleaning rule, the processor 701 may execute:

The lexical features of one or more cleaning functions corresponding to each cleaning rule are acquired as cleaning rule features of each cleaning rule.

In one embodiment, when acquiring the lexical features of one or more cleaning functions corresponding to each cleaning rule, the processor 701 may execute:

The lexical features of one or more cleaning functions corresponding to each cleaning rule are obtained according to the encoder neural network.

In one embodiment, when acquiring the joint features of the sample data to be cleaned and their corresponding cleaning effects, the processor 701 may execute:

The joint feature of each sample data to be cleaned and its corresponding cleaning effect is obtained according to the generative adversarial network.

In one embodiment, when using each joint feature as a training input and the cleaning rule feature corresponding to each joint feature as a target output to perform model training to obtain a cleaning rule classification model, the processor 701 may execute:

Taking each joint feature as the training input and the cleaning rule feature corresponding to each joint feature as the target output, the conditional recurrent neural network is used for model training, and the cleaning rule classification model is obtained.

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

It should be noted that, for the data cleaning method/model training method of the embodiment of the present application, ordinary testers in the art can understand that all or part of the process of implementing the data cleaning method/model training method of the embodiment of the present application can be implemented through a computer. The computer program can be stored in a computer-readable storage medium, such as a memory of an electronic device, and executed by at least one processor in the electronic device. During the execution process The process of the embodiment such as the data cleaning method/model training method may be included. The storage medium may be a magnetic disk, an optical disk, a read-only memory, a random access memory, or the like.

For the data cleaning device/model training device of the embodiment of the present application, each functional module thereof may be integrated in a processing chip, or each module may exist physically alone, or two or more modules may be integrated in one module. . The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules. If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk or an optical disk, etc. .

A data cleaning method, model training method, device, storage medium and device provided by the embodiments of the present application have been described in detail above. The principles and implementations of the present application are described with specific examples in this article. The above embodiments The description is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, there will be changes in the specific embodiments and application scope. In summary, the above , the contents of this specification should not be construed as limiting the application.

Claims (15)

1. A data cleaning method is applied to electronic equipment and is characterized by comprising the following steps:

acquiring data to be cleaned, which needs to be cleaned;

acquiring the cleaning requirement of the data to be cleaned;

determining a target cleaning rule for cleaning the data to be cleaned according to the data to be cleaned, the cleaning requirement and a pre-trained cleaning rule classification model;

performing data cleaning on the data to be cleaned according to the target cleaning rule, so that the cleaning effect on the data to be cleaned meets the cleaning requirement;

the cleaning rule classification model is obtained by performing model training by using a cleaning rule characteristic representing a cleaning rule as a target output and a combined characteristic representing sample data to be cleaned corresponding to the cleaning rule and a cleaning effect of the sample data to be cleaned as a training input.

2. The data cleaning method of claim 1, wherein the determining a target cleaning rule for data cleaning of the data to be cleaned according to the data to be cleaned, the cleaning requirement and a pre-trained cleaning rule classification model comprises:

acquiring the data to be cleaned and the combined characteristics of the cleaning requirements;

inputting the combined features into the cleaning rule classification model to obtain cleaning rule features output by the cleaning rule classification model;

and determining the cleaning rule matched with the cleaning rule characteristic as the target cleaning rule.

3. The data cleansing method of claim 2, wherein the determining the cleansing rule matching the cleansing rule feature comprises:

acquiring the similarity between the cleaning rule features and the cleaning rule features of a plurality of pre-stored cleaning rules;

and determining the cleaning rule with the similarity reaching the preset similarity as the cleaning rule matched with the cleaning rule characteristic.

4. The data cleansing method of claim 1, wherein the data cleansing of the data to be cleansed according to the target cleansing rule comprises:

and calling one or more cleaning functions corresponding to the target cleaning rule to perform data cleaning on the data to be cleaned.

5. The data cleaning method of claim 1, wherein obtaining data to be cleaned for which data cleaning is required comprises:

and acquiring sensor data acquired by a sensor, and taking the sensor data as data to be cleaned.

6. A model training method is applied to electronic equipment and is characterized by comprising the following steps:

acquiring a plurality of cleaning rules and acquiring sample data to be cleaned corresponding to each cleaning rule;

acquiring a cleaning effect of each cleaning rule for cleaning data of the corresponding sample data to be cleaned;

acquiring the combined characteristics of the sample data to be cleaned and the corresponding cleaning effect thereof, and acquiring the cleaning rule characteristics of the cleaning rules;

and performing model training by taking each joint feature as training input and taking the cleaning rule feature corresponding to each joint feature as target output to obtain a cleaning rule classification model.

7. The data cleansing method of claim 6, wherein the obtaining cleansing rule features of each of the cleansing rules comprises:

and acquiring the vocabulary characteristics of one or more cleaning functions corresponding to each cleaning rule as the cleaning rule characteristics of each cleaning rule.

8. The method of claim 7, wherein the obtaining the vocabulary characteristics of the one or more cleansing functions corresponding to each of the cleansing rules comprises:

and acquiring the vocabulary characteristics of one or more cleaning functions corresponding to each cleaning rule according to the neural network of the encoder.

9. The method according to claim 8, wherein the obtaining of the joint feature of each sample data to be cleaned and the corresponding cleaning effect thereof comprises:

and acquiring the combined characteristics of the sample data to be cleaned and the corresponding cleaning effect thereof according to the generated countermeasure network.

10. The data cleaning method of claim 6, wherein performing model training using each of the joint features as a training input and using the cleaning rule feature corresponding to each of the joint features as a target output to obtain a cleaning rule classification model, comprises:

and taking each joint feature as training input, taking the cleaning rule feature corresponding to each joint feature as target output, and performing model training by using a conditional cycle neural network to obtain the cleaning rule classification model.

11. A data cleaning device is applied to electronic equipment and is characterized by comprising:

the data acquisition module is used for acquiring data to be cleaned, which needs to be cleaned;

the requirement acquisition module is used for acquiring the cleaning requirement of the data to be cleaned;

the rule determining module is used for determining a target cleaning rule for cleaning the data to be cleaned according to the data to be cleaned, the cleaning requirement and a pre-trained cleaning rule classification model;

the data cleaning module is used for cleaning the data to be cleaned according to the target cleaning rule, so that the cleaning effect of the data to be cleaned meets the cleaning requirement;

the cleaning rule classification model is obtained by performing model training by using a cleaning rule characteristic representing a cleaning rule as a target output and a combined characteristic representing sample data to be cleaned corresponding to the cleaning rule and a cleaning effect of the sample data to be cleaned as a training input.

12. A model training device applied to electronic equipment is characterized by comprising:

the first acquisition module is used for acquiring a plurality of cleaning rules and acquiring sample data to be cleaned corresponding to each cleaning rule;

the second acquisition module is used for acquiring the cleaning effect of each cleaning rule for cleaning the data of the corresponding sample data to be cleaned;

the third acquisition module is used for acquiring the combined characteristics of the sample data to be cleaned and the cleaning effect corresponding to the sample data to be cleaned and acquiring the cleaning rule characteristics of the cleaning rules;

and the model training module is used for performing model training by taking each joint feature as training input and taking the cleaning rule feature corresponding to each joint feature as target output to obtain a cleaning rule classification model.

13. A storage medium having stored thereon a computer program for causing a computer to perform a data cleansing method according to any one of claims 1 to 5 or a model training method according to any one of claims 6 to 10 when the computer program is run on the computer.

14. An electronic device comprising a processor and a memory, the memory storing a computer program, wherein the processor is configured to perform the data cleansing method according to any one of claims 1 to 5 by invoking the computer program.

15. An electronic device comprising a processor and a memory, the memory storing a computer program, wherein the processor is configured to perform the model training method of any one of claims 6 to 10 by invoking the computer program.

Images