WO2018000269A1 - Data annotation method and system based on data mining and crowdsourcing - Google Patents

Abstract

A data annotation method based on data mining and crowdsourcing, comprising: acquiring raw data to be annotated (S101); performing classification and crowdsourcing distribution on the raw data by using an integrated algorithm (S102); acquiring a crowdsourcing annotation result and performing automatic review on the same by using an integrated algorithm, filtering and marking a questionable annotation result (S103); and outputting a crowdsourcing annotation result that has gone through the automatic review, the crowdsourcing annotation result including the questionable annotation result (S104). Questionable annotation results having possible problems are searched for and found from among all of the crowdsourcing annotation results, and the questionable annotation results are marked to facilitate reviewing and modifications thereof, thereby significantly facilitating the finding of questionable annotation results and increasing the annotation quality of the outputted results. The method organically combines the technique of data mining and crowdsourcing platforms, thus enabling a massive amount of accurately annotated data to be generated and the annotation costs to be effectively reduced at the same time.

Description

Data mining method and system based on data mining and crowdsourcing Technical field

The invention relates to the technical field of data annotation, in particular to a data annotation method and system based on data mining and crowdsourcing.

Background technique

In recent years, with the development of crowdsourcing technology, the use of crowdsourcing technology for data labeling has attracted the attention of researchers. Crowdsourcing technology is a distributed problem solving method. This technology uses the wisdom and strength of everyone to solve tasks that are difficult for computers to solve, especially data annotation, object recognition, etc., which are very simple for humans, but are very difficult tasks for computers. Many annotation tasks, such as text annotation, image classification, etc., can be published to the Internet through the crowdsourcing platform, and marked by ordinary users from the Internet. Ordinary users complete the data annotation task and receive the financial rewards provided by the publisher.

The advantage of the crowdsourcing platform is that the processing is fine, and when the scale is large enough, comprehensive and in-depth data processing results can be obtained. The disadvantages are large investment, low efficiency, and small data processing. Moreover, the annotators are ordinary users from the Internet, and the quality of the annotation is not guaranteed compared with the traditional expert annotation.

Therefore, how to reduce the labeling cost of the label data and improve the efficiency and quality of the labeling is a technical problem that needs to be solved in the technical field.

Summary of the invention

The object of the present invention is to provide a data mining method and system based on data mining and crowdsourcing, so as to reduce the labeling cost of the label data and improve the efficiency and quality of the labeling.

The object of the present invention is achieved by the following technical solutions:

A data mining method based on data mining and crowdsourcing, including:

Obtain the original data to be labeled;

Classification and crowdsourcing of the raw data using an integrated algorithm;

Obtain the crowdsourcing labeling results, use the integrated algorithm, automate the auditing of the crowdsourcing labeling results, filter out the problem labeling results, and mark the problem labeling results;

The results of the crowdsourced annotations that are automatically reviewed are output, and the results of the crowdsourcing annotations include the problem annotation results.

Preferably, the problem labeling result includes a low quality labeling result, and the step of automatically using the integrated algorithm to automatically perform the crowdsourcing labeling result, screening the question labeling result, and marking the question labeling result comprises:

According to the historical labeling database and the comparison rule, the crowdsourcing labeling result is analyzed, and the low-quality labeling result is obtained and labeled, wherein the comparison rule includes at least one of similarity comparison, cluster analysis and validity check.

Preferably, the step of analyzing the crowdsourcing labeling result according to the historical labeling database and the comparison rule according to the historical labeling database and the comparison rule, and the step of obtaining the low-quality labeling result and marking the step specifically includes:

According to the global history labeling database, the similarity comparison is performed on the crowdsourcing labeling result. If the similarity between the crowdsourcing labeling result and the labeling result in the historical labeling database reaches the threshold, the remaining labels that conflict with the crowdsourcing labeling result are collided. The result is marked as a low quality label result.

Preferably, the step of analyzing the crowdsourcing labeling result according to the historical labeling database and the comparison rule, and obtaining the low-quality labeling result and marking the step specifically includes:

According to the historical labeling database of the labeler, clustering analysis is performed on the crowdsourcing labeling result. If the crowdsourcing labeling result belongs to the clustering category, the labeling result of the remaining deviation from the crowdsourcing labeling result exceeding the threshold is marked as Low quality labeling results.

Preferably, the problem labeling result includes an error labeling result, and the step of automatically checking the crowdsourcing labeling result, obtaining the problem labeling result, and marking the question labeling result specifically includes:

According to the intent recognition rule, the data intent is compared with the crowdsourcing labeling result, and the result of the screening machine classification conflicts with the crowdsourcing labeling result is an error labeling result and is marked.

Preferably, the step of comparing the data intent with the crowdsourcing labeling result according to the intent identification rule, and the step of screening the machine classification conflicting with the crowdsourcing labeling result as an error labeling result and marking comprises:

It is determined whether the crowdsourcing labeling result conforms to the intent sentence matching template corresponding to the manual labeling intention, and if it does not match, it is marked as an error labeling result.

Preferably, the step of comparing the data intent with the crowdsourcing labeling result according to the intent identification rule, and the step of screening the machine classification conflicting with the crowdsourcing labeling result as an error labeling result and marking comprises:

It is determined whether the crowdsourcing labeling result includes any words in the intent word bag corresponding to the manual labeling intention, and if not, marking the error labeling result.

Preferably, the integrated algorithm includes at least a clustering algorithm and an annotation rule template, and the step of classifying and crowdsourcing the original data using the integrated algorithm specifically includes: according to the clustering algorithm and the labeling rule template The raw data is sorted and distributed.

Preferably, the step of outputting the result of the crowdsourcing labeling that is automatically audited comprises:

Outputs the statistical results of each caller's labeling task completion and the problem labeling results in each of the labeler's labeling tasks.

A data mining system based on data mining and crowdsourcing, including:

a capture module for obtaining raw data to be labeled;

a distribution module for classifying and crowdsourcing the original data using an integrated algorithm;

The processing module is configured to obtain the crowdsourcing labeling result, use an integrated algorithm, automatically perform an audit of the crowdsourcing labeling result, filter out the problem labeling result, and mark the problem labeling result;

An output module, configured to output an automated audited crowdsourcing labeling result, where the crowdsourcing labeling result includes a problem labeling result.

Compared with the prior art, the present invention has the following advantages: in the existing crowdsourcing technology, the labeler is an ordinary user from the Internet, and the quality of the labeling is not guaranteed, and the labeling method adopted in the present invention includes: obtaining the labeling station. Raw data required; distribute the original data according to preset rules; obtain crowdsourcing labeling results, and automatically audit the crowdsourcing labeling results, obtain problem labeling results, and mark problem labeling results; output crowdsourcing labeling Results and questions are annotated with results. In this way, the crowdsourcing labeling results can be reviewed, so that the results of the problematic labeling can be found out from all the crowdsourcing labeling results, and the result labeling results can be marked, so that the problem labeling results can be easily reviewed. And modification, great convenience to find out the problematic labeling results, improve the quality of the output of the output. The invention organically combines the data mining technology with the crowdsourcing platform, so as to have a large amount of accurate labeling data, and effectively reduce the labeling cost.

DRAWINGS

1 is a flowchart of a data mining and crowdsourcing-based data annotation method according to Embodiment 1 of the present invention;

2 is a schematic diagram of a data mining and crowdsourcing based data annotation system in accordance with a second embodiment of the present invention.

detailed description

Although the flowcharts describe various operations as a sequential process, many of the operations can be implemented in parallel, concurrently or concurrently. The order of the operations can be rearranged. Processing may be terminated when its operation is completed, but may also have additional steps not included in the figures. Processing can correspond to methods, functions, procedures, subroutines, subroutines, and the like.

Computer devices include user devices and network devices. The user equipment or the client includes but is not limited to a computer, a smart phone, a PDA, etc.; the network device includes but is not limited to a single network server, a server group composed of multiple network servers, or a cloud computing-based computer or network server. cloud. The computer device can operate alone to carry out the invention, and can also access the network and implement the invention through interoperation with other computer devices in the network. The network in which the computer device is located includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a VPN network, and the like.

The terms "first," "second," and the like may be used herein to describe the various elements, but the elements should not be limited by these terms, and the terms are used only to distinguish one element from another. The term "and/or" used herein includes any and all combinations of one or more of the associated listed items. When a unit is referred to as being "connected" or "coupled" to another unit, it can be directly connected or coupled to the other unit, or an intermediate unit can be present.

The terminology used herein is for the purpose of describing the particular embodiments, The singular forms "a", "an", It is also to be understood that the terms "comprising" and """ Other features, integers, steps, operations, units, components, and/or combinations thereof.

The invention will now be further described with reference to the drawings and preferred embodiments.

Embodiment 1

As shown in FIG. 1 , a data annotation method based on data mining and crowdsourcing is disclosed in the embodiment, including:

S101. Acquire original data to be marked.

S102. Perform classification and crowdsourcing distribution of the original data by using an integrated algorithm.

S103. Obtaining crowdsourcing labeling results, using an integrated algorithm, performing an automated audit of the crowdsourcing labeling results, screening out the problem labeling results, and marking the problem labeling results;

S104. Output an automated audited crowdsourcing labeling result, where the crowdsourcing labeling result is included Including the problem labeling results.

The range of data marked includes, but is not limited to, text, images, audio, statistics, and other data.

In the existing crowdsourcing technology, the labeler is an ordinary user from the Internet, and the quality of the labeling is not guaranteed, and the labeling method adopted in the present invention includes: S101, obtaining original data to be labeled; S102, using an integrated algorithm Classification and crowdsourcing distribution of the original data; S103, obtaining crowdsourcing labeling results, using an integrated algorithm, automatically reviewing the crowdsourcing labeling results, screening out the problem labeling results, and marking the problem labeling results; S104 And outputting the crowdsourcing labeling result that has been automatically reviewed, and the crowdsourcing labeling result includes the problem labeling result. In this way, the crowdsourcing labeling results can be reviewed, so that the results of the problematic labeling can be found out from all the crowdsourcing labeling results, and the result labeling results can be marked, so that the problem labeling results can be easily reviewed. And modification, great convenience to find out the problematic labeling results, improve the quality of the output of the output. The invention organically combines the data mining technology with the crowdsourcing platform, so as to have a large amount of accurate labeling data, and effectively reduce the labeling cost. The invention can be applied to the technical field of robot interaction, and is convenient for the robot to collect the labeled data, so that the robot can collect the required high quality data and better interact with the human.

According to one example, the problem labeling result includes a low quality labeling result, the step of automatically reviewing the crowdsourcing labeling result, obtaining the problem labeling result, and marking the question labeling result specifically includes:

According to the historical labeling database and the comparison rule, the crowdsourcing labeling result is analyzed, and the low-quality labeling result is obtained and labeled, wherein the comparison rule includes at least one of similarity comparison, cluster analysis and validity check. The low-quality labeling result is specifically a possible low-quality labeling, specifically the result of the suspected low-quality labeling, and as a suspect object, further specific inspection is needed.

According to another example, according to the historical annotation database and the comparison rule, the root analyzes the crowdsourcing labeling result according to the historical labeling database and the comparison rule, and the step of obtaining the low-quality labeling result and marking the step specifically includes:

According to the global history labeling database, the similarity comparison is performed on the crowdsourcing labeling result. If the similarity between the crowdsourcing labeling result and the labeling result in the historical labeling database reaches the threshold, the remaining labels that conflict with the crowdsourcing labeling result are collided. The result is marked as a low quality label result. This allows you to screen out low-quality labeling results for further screening.

According to another example, the step of analyzing the crowdsourcing labeling result according to the historical labeling database and the comparison rule, and obtaining the low-quality labeling result and marking the step specifically includes:

According to the historical labeling database of the labeler, clustering analysis is performed on the crowdsourcing labeling result. If the crowdsourcing labeling result belongs to the clustering category, the labeling result of the remaining deviation from the crowdsourcing labeling result exceeding the threshold is marked as Low quality labeling results. This allows you to screen out low-quality labeling results for further screening.

According to another example, the problem labeling result includes an error labeling result, the step of automatically reviewing the crowdsourcing labeling result, obtaining the problem labeling result, and marking the question labeling result specifically includes:

According to the intent recognition rule, the data intent is compared with the crowdsourcing labeling result, and the result of the screening machine classification conflicts with the crowdsourcing labeling result is an error labeling result and is marked. In this way, the error labeling results can be screened for further screening.

According to another example, the step of comparing the data intent with the crowdsourcing labeling result according to the intent recognition rule, and the step of screening the machine classification conflicting with the crowdsourcing labeling result as an error labeling result and marking comprises:

It is determined whether the crowdsourcing labeling result conforms to the intent sentence matching template corresponding to the manual labeling intention, and if it does not match, it is marked as an error labeling result. In this way, the error labeling results can be screened for further screening.

According to another example, the step of comparing the data intent with the crowdsourcing labeling result according to the intent recognition rule, and the step of screening the machine classification conflicting with the crowdsourcing labeling result as an error labeling result and marking comprises:

It is determined whether the crowdsourcing labeling result includes any words in the intent word bag corresponding to the manual labeling intention, and if not, marking the error labeling result. In this way, the error labeling results can be screened for further screening.

According to another example, the integrated algorithm includes at least a clustering algorithm and an annotation rule template, and the step of classifying and crowdsourcing the original data data using the integrated algorithm specifically includes: according to a clustering algorithm and An annotation rule template classifies and distributes the raw data.

According to another example, the step of outputting the crowdsourced annotation result by the automated review specifically includes:

Outputs the statistical results of each caller's labeling task completion and the problem labeling results in each of the labeler's labeling tasks. In this way, the statistical results of the annotations and the problematic annotation results can be completed, and not only the specific conditions of the annotations can be known, but also the data of the annotation accuracy of the labeler can be understood accordingly, so as to select a better labeler.

In the case of being closer to the actual application, for example, when the marked data is a corpus, the specific process of the S103 to automatically review the results of the crowdsourcing labeling includes:

Before the automated review, the crowdsourcing labeling results need to be summarized (ie input);

Then count the number of annotations to obtain the actual completion of the current task of the labeler;

Through the similarity comparison, clustering, validity test, etc., the historical annotation database is cited to analyze whether there is a "low quality labeling". As an example corresponding to the above low quality labeling result, it can be automatically labeled as "possible low quality". For example, different types of corpus have the same label; or, similarly similar corpus have different labels. For example, for corpora with similar influences, such as “I am going to eat” and “I am going to eat”, most of them are labeled “Go to dinner”, and one of them will mark “I am going to eat”. Going to sing, this needs to mark this labeling result as a low-quality labeling result, such as the above-mentioned "possible low quality";

Use the intent recognition rule to compare with the result of the manual labeling, and filter out the label items that conflict with the machine classification and the manual labeling, such as the above-mentioned error labeling result, which can be automatically marked as “probable error”;

In this way, the data automatically marked in this round will mark the review focus for the next manual sampling, greatly reducing the workload.

In the audit, you need to count the number of annotations. Specifically, you can count the total number of annotation entries and the statistics of the labels that must be marked.

For labeling results marked as "possibly low quality", specific,

Refer to the global history annotation database for similarity comparison. If the similarity of the two labeled corpus itself reaches a certain threshold, then the "crowdsourcing labeling result" is compared. If there is a conflict in the “crowdsourcing labeling result”, it is marked as “possibly low quality”;

As another example, the author's historical annotation database is referenced for clustering. If the corpus of this article is located within a certain cluster category (ie, the natural language content is similar), then the "crowdsourcing labeling result" is compared. If the historical “crowdout labeling result” deviates less, and the current labeling result deviates significantly from the historical labeling set range, it is marked as “possibly low quality”.

For the labeling result labeled "Possible Error", specifically,

Determining whether the labeled corpus meets the intent sentence matching template corresponding to the intention of manual marking, and if not, marking the "probable error";

As another example manner, it is determined whether the labeled corpus contains any vocabulary in the intent word bag corresponding to the manual labeling intention, and if not included, it is marked as “probable error”.

In the audit, statistical tools are used to collect statistics on the results of crowdsourcing, including the use of natural language processing tools, “primary review” of crowdsourcing labeling results, and automatic labeling of “high-probability error labels”. class.

The output results include the statistical results of outputting each of the caller's labeling tasks, and the labeling entries that each of the labeler may have problems with the task.

Embodiment 2

As shown in FIG. 2, in this embodiment, a data annotation system based on data mining and crowdsourcing is disclosed, including:

The capture module 201 is configured to obtain original data to be labeled;

a distribution module 202, configured to perform classification and crowdsourcing distribution of the original data using an integrated algorithm;

The processing module 203 is configured to obtain the crowdsourcing labeling result, use an integrated algorithm, automatically perform an audit on the crowdsourcing labeling result, filter out the problem labeling result, and mark the problem labeling result;

The output module 204 is configured to output an automated audited crowdsourcing labeling result, where the crowdsourcing labeling result includes a problem labeling result.

The data annotation system disclosed in this embodiment includes: a capture module 201 for acquiring original data to be labeled; a distribution module 202 for classifying and distributing the original data using an integrated algorithm; and processing module 203, used to obtain the crowdsourcing labeling result, use an integrated algorithm, automatically review the crowdsourcing labeling result, filter out the problem labeling result, and mark the problem labeling result; the output module 204 is used to output the automated auditing public. The package labeling result includes the problem labeling result in the crowdsourcing labeling result. In this way, the crowdsourcing labeling results can be reviewed, so that the results of the problematic labeling can be found out from all the crowdsourcing labeling results, and the result labeling results can be marked, so that the problem labeling results can be easily reviewed. And modification, great convenience to find out the problematic labeling results, improve the quality of the output of the output. The invention organically combines the data mining technology with the crowdsourcing platform, so as to have a large amount of accurate labeling data, and effectively reduce the labeling cost.

According to one example, the question labeling result includes a low quality labeling result, where The management module is specifically configured to: analyze the crowdsourcing labeling result according to the historical labeling database and the comparison rule, and obtain and label the low quality labeling result, wherein the comparison rule includes at least the similarity comparison, the cluster analysis, and the validity check. One.

According to another example, the processing module is specifically configured to perform a similarity comparison on the crowdsourcing labeling result according to the global history labeling database, and if the similarity between the crowdsourcing labeling result and the labeling result in the historical labeling database reaches a threshold , the remaining labeling results that conflict with the crowdsourcing labeling result are marked as low quality labeling results.

According to another example, clustering analysis is performed on the crowdsourcing labeling result according to the labeler's historical labeling database. If the crowdsourcing labeling result belongs to the clustering category, the remaining deviation from the crowdsourcing labeling result is exceeded. The labeling result of the threshold is marked as a low quality labeling result.

According to another example, the question labeling result includes an error labeling result, and the processing module is specifically configured to:

According to the intent recognition rule, the data intent is compared with the crowdsourcing labeling result, and the result of the screening machine classification conflicts with the crowdsourcing labeling result is an error labeling result and is marked.

According to another example, the processing module is specifically configured to: determine whether the crowdsourcing labeling result meets the intent sentence matching template corresponding to the manual labeling intention, and if not, mark the error labeling result.

According to another example, the processing module is specifically configured to: determine whether the crowdsourcing labeling result includes any words in the intent word bag corresponding to the manual labeling intention, and if not, mark the error labeling result.

According to another example, the integrated algorithm includes at least a clustering algorithm and an annotation rule template, and the distribution module is specifically configured to: classify and distribute the original data according to a clustering algorithm and an annotation rule template.

According to another example, the output module is specifically configured to: output a statistical result of each of the labeler's labeling task completion status and a question labeling result in each of the labeler's labeling tasks.

The above is a further detailed description of the present invention in connection with the specific preferred embodiments, and the specific embodiments of the present invention are not limited to the description. It will be apparent to those skilled in the art that the present invention may be made without departing from the spirit and scope of the invention.

Claims (18)

A data annotation method based on data mining and crowdsourcing, characterized in that it comprises: Obtain the original data to be labeled; Classification and crowdsourcing of the raw data using an integrated algorithm; Obtain the crowdsourcing labeling results, use the integrated algorithm, automate the auditing of the crowdsourcing labeling results, filter out the problem labeling results, and mark the problem labeling results; The results of the crowdsourced annotations that are automatically reviewed are output, and the results of the crowdsourcing annotations include the problem annotation results. The data labeling method according to claim 1, wherein the question labeling result comprises a low-quality labeling result, and the integrated algorithm is used to automatically review the crowdsourcing labeling result, and screen the problem labeling result, and The steps of marking the problem labeling result specifically include: According to the historical labeling database and the comparison rule, the crowdsourcing labeling result is analyzed, and the low-quality labeling result is obtained and labeled, wherein the comparison rule includes at least one of similarity comparison, cluster analysis and validity check. The data labeling method according to claim 2, wherein the step of analyzing the crowdsourcing labeling result according to the historical labeling database and the comparison rule, and obtaining the low-quality labeling result and marking the step specifically comprises: According to the global history labeling database, the similarity comparison is performed on the crowdsourcing labeling result. If the similarity between the crowdsourcing labeling result and the labeling result in the historical labeling database reaches the threshold, the remaining labels that conflict with the crowdsourcing labeling result are collided. The result is marked as a low quality label result. The data labeling method according to claim 2, wherein the step of analyzing the crowdsourcing labeling result according to the historical labeling database and the comparison rule, and obtaining the low-quality labeling result and marking the step specifically comprises: According to the historical labeling database of the labeler, clustering analysis is performed on the crowdsourcing labeling result. If the crowdsourcing labeling result belongs to the clustering category, the labeling result of the remaining deviation from the crowdsourcing labeling result exceeding the threshold is marked as Low quality labeling results. The data labeling method according to claim 1, wherein the question labeling result comprises an error labeling result, and the preset algorithm is used to automatically review the crowdsourcing labeling result, and screen the problem labeling result, and The steps of marking the problem labeling result specifically include: Compare the data intent with the crowdsourced labeling result according to the intent recognition rule, and filter the machine points. The class conflicts with the result of the crowdsourcing annotation and marks the result with an error. The data labeling method according to claim 5, wherein the comparing the data intent with the crowdsourcing labeling result according to the intent recognition rule, and screening the machine classification and the crowdsourcing labeling result conflicting result labeling and marking The steps specifically include: It is determined whether the crowdsourcing labeling result conforms to the intent sentence matching template corresponding to the manual labeling intention, and if it does not match, it is marked as an error labeling result. The data labeling method according to claim 5, wherein the comparing the data intent with the crowdsourcing labeling result according to the intent recognition rule, and screening the machine classification and the crowdsourcing labeling result conflicting result labeling and marking The steps specifically include: It is determined whether the crowdsourcing labeling result includes any words in the intent word bag corresponding to the manual labeling intention, and if not, marking the error labeling result. The data annotation method according to claim 1, wherein the integrated algorithm comprises at least a clustering algorithm and an annotation rule template, and the step of classifying and crowdsourcing the original data using the integrated algorithm Specifically, the method includes: classifying and distributing the original data according to a clustering algorithm and an annotation rule template. The data labeling method according to claim 1, wherein the step of outputting the crowdsourced labeling result that is automatically audited comprises: Outputs the statistical results of each caller's labeling task completion and the problem labeling results in each of the labeler's labeling tasks. A data annotation system based on data mining and crowdsourcing, characterized in that it comprises: a capture module for obtaining raw data to be labeled; a distribution module for classifying and crowdsourcing the original data using an integrated algorithm; The processing module is configured to obtain the crowdsourcing labeling result, use an integrated algorithm, automatically perform an audit of the crowdsourcing labeling result, filter out the problem labeling result, and mark the problem labeling result; An output module, configured to output an automated audited crowdsourcing labeling result, where the crowdsourcing labeling result includes a problem labeling result. The data labeling system according to claim 10, wherein the problem labeling result comprises a low quality labeling result, and the processing module is specifically configured to: analyze the crowdsourcing labeling result according to the historical labeling database and the comparison rule, A low quality labeling result is obtained and labeled, wherein the comparison rule includes at least one of a similarity comparison, a cluster analysis, and a validity check. The data annotation system of claim 11 wherein said processing module The block is specifically used to: compare the similarity degree of the crowdsourcing labeling result according to the global history labeling database, and if the similarity between the crowdsourcing labeling result and the labeling result in the historical labeling database reaches a threshold, the remaining and the crowdsourcing are marked The result of the conflicting labeling results is marked as a low quality labeling result. The data labeling system according to claim 11, wherein the crowdsourcing labeling result is clustered according to the labeler's historical labeling database, and if the crowdsourcing labeling result belongs to the clustering category, the remaining The result of the labeling of the crowdsourcing labeling result that exceeds the threshold is marked as a low-quality labeling result. The data labeling system according to claim 10, wherein the problem labeling result comprises an error labeling result, and the processing module is specifically configured to: According to the intent recognition rule, the data intent is compared with the crowdsourcing labeling result, and the result of the screening machine classification conflicts with the crowdsourcing labeling result is an error labeling result and is marked. The data labeling system according to claim 14, wherein the processing module is configured to: determine whether the crowdsourcing labeling result conforms to an intent sentence matching template corresponding to the manual labeling intention, and if not, Marked as an error labeling result. The data labeling system according to claim 14, wherein the processing module is configured to: determine whether the crowdsourcing labeling result includes any words in the intent word bag corresponding to the manual labeling intention, if not included , marked as an error labeling result. The data labeling system according to claim 10, wherein the integrated algorithm comprises at least a clustering algorithm and an annotation rule template, and the distribution module is specifically configured to: use the clustering algorithm and the labeling rule template to Data is classified and distributed. The data labeling system according to claim 10, wherein the output module is specifically configured to: output a statistical result of each of the labeler's labeling task completion status and a question labeling result in each of the labeler's labeling tasks.

Images