WO2020114100A1 - Information processing method and apparatus, and computer storage medium - Google Patents

Abstract

An information processing method, an information processing apparatus, and a computer storage medium. The method comprises: performing clustering processing on text in an original text set by using a preset clustering mode to obtain multiple first cluster sets (110); and performing clustering processing on text in each first cluster set by using a preset clustering mode to obtain multiple second cluster sets, the preset clustering mode being a first preset clustering mode or a second preset clustering mode (120).

Description

Information processing method, device and computer storage medium Technical field

The present invention relates to the field of computer technology, and in particular, to an information processing method, device, and computer storage medium.

Background technique

In the era of information explosion, there is an increasing demand for rapid and accurate collation of massive amounts of information. To meet this demand, many applications have emerged, such as information retrieval, literature review, personality recommendation, and intelligent question answering. In these applications, text clustering technology is the key core technology. Text clustering technology has become an important means of effective organization, summary and navigation of text information.

Unsupervised machine learning provides some clustering techniques, including partition-based methods, hierarchical clustering methods, density-based methods, grid-based methods, model-based methods, self-organizing mapping neural network methods, ant colony-based methods Method etc. The complexity of these methods is relatively high, and it is difficult to deal with large-scale text clustering.

If a slightly simpler clustering algorithm is used for clustering, the current processing scheme is to first use a clustering algorithm for clustering, and further process the residual text of the last unsuccessful clustering. This clustering method has no complementary or progressive performance, but is simply a "chimney" process, and each clustering actually uses a different method or standard, making the final clustering results inconsistent.

Summary of the invention

To solve the existing technical problems, embodiments of the present invention provide an information processing method, device, and computer storage medium.

To achieve the above objective, the technical solutions of the embodiments of the present invention are implemented as follows:

An embodiment of the present invention provides an information processing method. The method includes: performing clustering on a text in an original text set by using a preset clustering method to obtain a plurality of first cluster sets; for each first cluster The text in the class set is clustered using a preset clustering method to obtain multiple second cluster sets; the preset clustering method is the first preset clustering method or the second preset clustering method.

An embodiment of the present invention also provides an information processing apparatus, the apparatus includes: a first clusterer, configured to perform clustering processing on the text in the original text set by using a preset clustering method to obtain multiple first clusters Cluster set; a second clusterer, configured to perform clustering on the text in each first cluster set by using a preset clustering method to obtain multiple second cluster sets; wherein, the preset cluster The mode is the first preset clustering mode or the second preset clustering mode.

An embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method according to the embodiment of the present invention are implemented.

An embodiment of the present invention also provides an information processing apparatus, including: a processor and a memory for storing a computer program that can be run on the processor, where the processor is used to execute the present invention when the computer program is run The steps of the method described in the examples.

An embodiment of the present invention discloses an information processing method. The method includes: performing clustering processing on a text in an original text set by using a preset clustering method to obtain a plurality of first cluster sets; for each first cluster The text in the cluster set is clustered using a preset clustering method to obtain multiple second cluster sets; wherein, the preset clustering method is the first preset clustering method or the second preset clustering method .

BRIEF DESCRIPTION

FIG. 1 is a first schematic flowchart of an information processing method according to an embodiment of the present invention;

2 is a second schematic flowchart of an information processing method according to an embodiment of the present invention;

FIG. 3 is a third schematic flowchart of an information processing method according to an embodiment of the present invention;

4 is a fourth schematic flowchart of an information processing method according to an embodiment of the present invention;

5 is a schematic structural diagram of an information processing apparatus according to an embodiment of the present invention.

detailed description

The present invention will be described in further detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not intended to limit the present invention.

An embodiment of the present invention provides an information processing method. As shown in FIG. 1, the method includes:

Step 110: Perform clustering on the text in the original text set by using a preset clustering method to obtain multiple first cluster sets.

In an alternative embodiment of the present invention, the text in the original text collection may be a massive amount of data acquired in different application systems such as a digital library, an information retrieval database, and so on. The text may be divided according to a preset standard. As an example, according to an application scenario, a sentence, ten sentences, or a paragraph are divided into one text.

As an optional example, the text of the original text set is clustered to obtain multiple first cluster sets. The first clustering set is the clustering set obtained after the original text set undergoes the first clustering process.

Step 120: Perform a clustering process on the text in each first clustering set by using a preset clustering method to obtain multiple second clustering sets.

When the process of two clustering processes is shown in FIG. 2, as an optional example, the text in each first cluster set is clustered separately to obtain multiple second cluster sets. The obtained multiple second clustering sets are used as the clustering result obtained after the information processing of the present invention. The second clustering set is a clustering set obtained by the first clustering set after the second clustering process.

In this embodiment of the present invention, the preset clustering mode is a first preset clustering mode or a second preset clustering mode. In other words, in this embodiment of the present invention, the first preset clustering method or the second preset clustering method may be used to perform two clustering processes on the text in the original text set, or the first preset clustering method may be used to Perform a clustering process on the text in the original text collection to obtain a first clustering set, and then use a second preset clustering method to cluster the text in the first clustering set, or use a second preset clustering method Perform a clustering process on the text in the original text set to obtain a first cluster set, and then perform a clustering process on the text in the first cluster set using a first preset clustering method.

In this embodiment, the first preset clustering mode and the second preset clustering mode are determined based on the efficiency requirements and/or precision requirements of text clustering.

In an optional embodiment of the present invention, if text clustering focuses on clustering efficiency, and the first preset clustering method has high clustering efficiency, the preset clustering methods may all use the first preset clustering method, that is, The first preset clustering method is used to cluster the original text collection, and the first preset clustering method is used to cluster the first clustering collection; if text clustering concerns clustering accuracy, and the second preset The clustering method has high clustering accuracy, so the clustering method of the drama club can all adopt the second preset clustering method, that is, the second preset clustering method is used to cluster the original text collection, and the second preset clustering is used. Clustering the first clustering set; if text clustering focuses on clustering efficiency and clustering accuracy at the same time, the preset clustering method may use the first preset clustering mode and the second preset clustering mode, That is, the first preset clustering method is used to cluster the original text collection, and the second preset clustering method is used to cluster the first clustering collection; or, the second preset clustering method is used to cluster the original text The clusters are clustered, and the first clustering cluster is clustered using the first preset clustering method. Or, if text clustering does not require clustering results, you can arbitrarily choose the clustering method for clustering.

In this way, different combinations of clustering methods can be selected according to clustering requirements, so that clustering processing is more flexible and effective, and can meet the needs of different application scenarios.

In an alternative embodiment of the present invention, when the preset clustering mode is the first preset clustering mode, as shown in FIG. 3, the clustering process is performed using the preset clustering mode, including:

Step 310: Extract each text in the to-be-processed text set, and express each text as a signature vector.

In an optional embodiment of the present invention, the to-be-processed text set may be the original text set or the first cluster set, that is, the cluster processing in this embodiment is directed to the clustering in step 110 and/or step 120 Class processing; that is, perform the first clustering process on the original text set according to the first preset clustering method or perform the second clustering process on the first cluster set after the clustering process has been performed.

In an optional embodiment of the present invention, the step of processing the to-be-processed text set to obtain a signature vector includes (not shown in the drawings of the specification):

Step 3101: Obtain a word sequence of text, and obtain a weight of each word in the word sequence, and obtain a weighted word sequence based on obtaining the weight of each word in the word sequence.

Preprocessing the text to obtain the word sequence of the text. The pre-processing step includes: performing word segmentation and removing stop words on the text. According to a preset weight algorithm, the weight of each word in the word sequence is calculated to obtain a weighted word sequence. As an optional example, a TF-IDF algorithm is used to perform weight calculation on the text to obtain a sequence of weighted words of the text.

As an example, the text1 in the text collection to be processed is "I want to apply for in-app purchase ZTE mobile phone". After preprocessing, the resulting word sequence is [apply|in-app purchase|ZTE|mobile phone]. The TF-IDF algorithm is used to calculate the weight of each word in the obtained word sequence, and the weighted word sequence of text1 is [application, 3.12|internal purchase, 8.90|ZTE, 5.54|mobile phone, 1.89].

Step 3102: Perform a hash operation on the weighted word sequence to obtain a weighted hash value sequence.

A hash algorithm is used to calculate the hash value of each word in the word sequence. Convert each word in the word sequence into a binary number of digits N according to a preset, and multiply each bit of the hash value of each word by the weight of the word to obtain the text Sequence of weighted hashes.

As an example, if N is set to 128, that is, each word is converted into 128 bits by a hash algorithm, then after each word in the word sequence corresponding to text1 undergoes a hash algorithm, the resulting hash value sequence is [ 100101...010,3.12|...|000110...100,1.89], multiplying each bit of each word by the corresponding weight to obtain the weighted hash sequence of the text is [3.12 -3.12-3.12 3.12-3.12 3.12...-3.12 3.12-3.12|...|-1.89-1.89-1.891.891.89-1.89...].

Step 3103: Perform merge processing on each weighted hash value in the weighted hash value sequence to obtain a weighted hash value corresponding to the text.

In an optional embodiment of the present invention, each hash value in the weighted hash value sequence is added bit by bit to obtain a weighted hash value corresponding to the text.

As an example, the weighted hash value sequence [3.12-3.12-3.123.12-3.12...-3.123.12-3.12|...|-1.89-1.89-1.891.891.81.89-1.89. ..], add the four weighted hash values bit by bit to get [5.743.91-1.182.31-12.34-7.71...-3.64-0.1121.29].

Step 3104: Perform binary processing on the weighted hash value to obtain a binary signature vector.

In an optional embodiment of the present invention, the weighted hash value obtained in the above step is binarized to obtain a binary signature vector of the text. As an optional example, the weighted hash value is processed bit by bit. When the value of the bit is positive, the bit is 1, and when the value of the bit is negative, the bit is 0.

As an example, the weighted hash value of text1 [5.74 3.91-1.18 2.31-12.34-7.71...-3.64-0.11 21.29] is binarized, and the 128-bit binary signature vector corresponding to text1 is [ 1 1 0 0 1 0 0 0 0 1].

Step 320: Perform segmentation processing on the signature vector of each text, obtain multiple signature vector segments, and perform clustering processing on each signature vector segment.

In an optional embodiment of the present invention, the signature vectors of the text are segmented according to preset parameters to obtain multiple signature vector segments, and each signature vector segment is clustered. The preset parameters may continue to be set according to calculation requirements. The specific steps (not shown in the drawings in the specification) include:

Step 3201: Segment the binary signature vector of each text to obtain multiple binary signature vector segments.

In an optional embodiment of the present invention, if the binary signature vector is divided into b segments, then each segment of the N-bit signature vector contains r bits, where N=b*r.

Step 3202: Perform hash operation processing on each binary signature vector segment to obtain a hash value corresponding to the binary signature vector segment.

In an optional embodiment of the present invention, the b-stage binary signature vectors are respectively hashed to obtain the hash value of each stage of the binary signature vector.

Step 3203: Divide the corresponding text with the same hash value into the same clustering set.

In an optional embodiment of the present invention, the hash value of each segment of the binary signature vector obtained in the above step is classified, and the corresponding text with the same hash value is divided into the same clustering set.

In this way, when the first preset clustering method is adopted, the text is reduced by using the hash algorithm, and the weighting algorithm is used, which not only reduces the difficulty of calculation, improves efficiency, but also ensures the accuracy of calculation.

In an alternative embodiment of the present invention, when the preset clustering mode is the second preset clustering mode, as shown in FIG. 4, the clustering process using the preset clustering mode includes:

Step 410: Calculate the similarity between any two texts in the text set to be processed.

In an optional embodiment of the present invention, as described above, the to-be-processed text set may be the original text set or the first cluster set, that is, the cluster processing in this embodiment is directed to step 110 and/or step The clustering process in 120; that is, the first clustering process is performed on the original text set according to the second preset clustering method or the second clustering is performed on the first clustering set after the clustering process has been performed Class processing.

Based on a preset similarity algorithm, the similarity between any two texts in the to-be-processed text set is calculated.

Step 420: Perform clustering processing on the text in the to-be-processed text set based on the calculation result of the similarity.

In an optional embodiment of the present invention, according to a preset similarity classification algorithm, the calculation result of the similarity obtained above is classified to obtain a clustering set conforming to the similarity classification algorithm.

In an alternative embodiment of the present invention, the steps of the preset similarity algorithm specifically include a semantic similarity algorithm and a syntactic similarity algorithm, wherein the semantic similarity algorithm between any two texts specifically includes (in the drawings of the specification (Not shown):

Step A: Use the preset corpus to train the text in the text set to be processed to obtain the word vector matrix of the text.

其中，

表示具有|V|行d _w列的矩阵；V为语料集中所有词汇构成的词汇表，|V|为该词汇表中的词汇个数。若单词w在矩阵中的次序为m，则由该模型得到的词向量可以表示为

其中，

为第m行的向量。 As an example, the Word2Vec method can be used for word vector training, and the vector length is set to d _w (optionally, d _{w is} set to 400), then the Word2Vec method obtained by training is a matrix

among them,

Represents a matrix with |V| rows and d _w columns; V is a vocabulary composed of all words in the corpus, and |V| is the number of vocabularies in the vocabulary. If the order of the word w in the matrix is m, the word vector obtained by the model can be expressed as

among them,

Is the vector of the mth row.

Step B: For any two texts, calculate the semantic similarity based on the semantic distance.

Step B1: Pre-process the text in the text set to be processed to obtain the word sequence of the text.

In an alternative embodiment of the present invention, the preprocessing operations include: word segmentation of the text, removal of stop words, and so on.

t2的词序列为

其中，

为t1的第m个词，m为正整数；

为t2的第n个词，n为正整数。 As an example, select any two texts t1 and t2 for preprocessing to get the sequence of t1 words

The word sequence of t2 is

among them,

Is the m-th word of t1, m is a positive integer;

Is the nth word of t2, n is a positive integer.

Step B2: Calculate the semantic similarity of the corresponding two words in any two texts in the text set.

In an optional embodiment of the present invention, the calculation formula (1) of the word sense similarity is:

Among them, sim _cos (w ₁ , w ₂ ) is the word meaning similarity of the word w ₁ and the word w ₂ ; v(w ₁ ) is the word vector of the word w ₁ ; v(w ₂ ) is the word vector of the word w ₂ ; |v(w ₁ )| is the length of the word vector of the word w ₁ ; |v(w ₂ )| is the length of the word vector of the word w ₂ .

In an optional embodiment of the present invention, the formula (2) for calculating the semantic similarity between t1 and t2 is:

为t1中词

与t2的词义距离；

为t2中词

与t1的词义距离；根据公式(1)计算t1和t2之间任意两个词的词义相似度

among them,

Is the word in t1

Meaning distance from t2;

Is a word in t2

The semantic distance from t1; calculate the semantic similarity of any two words between t1 and t2 according to formula (1)

As an example, according to formula (2), the semantic similarity score1 between t1 and t2 is calculated.

In an alternative embodiment of the present invention, the syntax similarity algorithm between any two texts specifically includes (not shown in the drawings of the specification):

Step A: Pre-process the text in the text set to be processed to obtain the word sequence of the text.

In an alternative embodiment of the present invention, the preprocessing operations include: word segmentation of the text, removal of stop words, and so on.

As an example, the texts text1 and text2 are subjected to word segmentation, stop word removal and other operations to obtain word sequences t1 and t2.

Step B: Perform dependency syntax analysis on the word sequence corresponding to the text to obtain a syntactic similarity between any two texts.

Step B1: Use a preset syntax analysis tool to perform a dependency syntax analysis on the word sequence of the two texts to obtain the number of valid word collocation pairs between the two texts.

As an example, the natural language processing open source package of Stanford University or the natural language processing open source package of Fudan University is used to analyze the dependency syntax of t1 and t2, and the number of effective word collocation pairs in t1 and t2 is calculated, respectively p ₁ and p ₂ .

Step B2: According to the number of valid word collocation pairs between the two texts, obtain the syntactic similarity between the texts.

In an optional embodiment of the present invention, the syntactic similarity according to the calculation formula (3) is:

score2=|p ₁ -p ₂ | (3)

As an example, p ₁ and p ₂ are calculated according to formula (3) to obtain the syntactic similarity score2 of the text1 and text2.

In an optional embodiment of the present invention, the steps of the preset similarity algorithm specifically include:

Step A: Obtain the semantic similarity between any two texts according to the preset semantic similarity algorithm.

Step B: According to the preset syntactic similarity algorithm, the syntactic similarity between any two texts is obtained.

Step C: Based on the calculated semantic similarity and syntactic similarity, a similarity between any two texts is obtained.

In an alternative embodiment of the present invention, the calculation of the similarity between any two texts (4) is:

score=α*score1+β*score2 (4)

Among them, score is the similarity between any two texts; score1 is the semantic similarity between any two texts; score2 is the syntactic similarity between any two texts; α is the weight of semantic similarity 0<=α <=1, β is the weight of syntactic similarity 0<=β<=1, α+β=1, the values of α and β can be set according to the calculation requirements, as an optional example, both α and β Set to 0.5.

In an alternative embodiment of the present invention, the preset similarity classification algorithm includes:

Step A: Calculate the first similarity between two first texts in the text set to be processed, and determine whether the first similarity exceeds a first preset threshold; the first text is the text set to be processed Any text in.

In an optional embodiment of the present invention, the first similarity is the similarity between any two texts in the text set to be processed, and the first preset threshold can be set according to calculation needs, and the setting range can be from 0 to 1. As an example, the first preset threshold is set to 0.5.

In an optional embodiment of the present invention, in step 410, based on the preset similarity algorithm, a similarity score between any two texts in the to-be-processed text set is calculated.

Step B: When the first similarity exceeds the first preset threshold, divide the two first texts into the same clustering set.

In an optional embodiment of the present invention, when the calculated score exceeds a first preset threshold, the two first texts corresponding to the score are divided into the same clustering set. When the calculated score is less than the first preset threshold, the two texts corresponding to the score are not divided into the same clustering set. That is, the similarity between any two texts in the clustering set exceeds the first preset threshold.

In an optional embodiment of the present invention, the preset similarity classification algorithm may further include:

Step A: separately calculate a second similarity between the second text in the to-be-processed text set and the first text in the clustering set, and determine whether the second similarity exceeds the first preset Threshold; the second text is other text than the first text in the text to be processed.

In an optional embodiment of the present invention, the second similarity is the similarity between any two texts in the text set to be processed. The similarity between any two texts in the clustering set is not less than the first preset threshold.

In an optional embodiment of the present invention, in step 410, based on the preset similarity algorithm, the second text in the to-be-processed text set and any text in the clustering set are calculated. Similarity, and determine whether the multiple similarity scores are not less than a first preset threshold.

(1<＝i<＝m，m,i皆为正整数)的对应的相似度score _i ¹²，并判断所述相似度score _i ¹²与第一预设阈值的大小关系。 As an optional example, in step 410, based on the preset similarity algorithm, the second text t ^{2 in the} text set to be processed and the text in the cluster set P1 are calculated

(1<=i<=m, m, i are all positive integers) corresponding similarity score _i ¹² , and determine the magnitude relationship between the similarity score _i ¹² and the first preset threshold.

Step B: When the second similarities all exceed the first preset threshold, obtain a second average value of the similarity between the second text and the text in the cluster set, and determine the second Whether the mean value of similarity exceeds a second preset threshold; when the mean value of the second similarity exceeds the second preset threshold, the cluster set is determined as a candidate cluster set.

所述相似度均值即为第二相似度均值。判断所述第二相似度均值与第二预设阈值的大小关系，所述第二预设阈值可以根据计算需要进行设置，设置的范围可以为0至1。作为一种示例，将第二预设阈值设定为0.5。当所述第二相似度均值都超过第二预设阈值时，将所述聚类集合确定为候选聚类集合。 In an optional embodiment of the present invention, when the plurality of similarity scores all exceed the first preset threshold, the multiple scores are added and the corresponding average value of similarity is calculated

The mean value of similarity is the second mean value of similarity. Judging the magnitude relationship between the second similarity average and the second preset threshold, the second preset threshold can be set according to calculation needs, and the setting range can be 0 to 1. As an example, the second preset threshold is set to 0.5. When all the average values of the second similarities exceed the second preset threshold, the cluster set is determined as a candidate cluster set.

判断所述第二相似度均值与第二预设阈值的大小关系，当所述第二相似度均值都超过第二预设阈值时，将所述聚类集合C确定为候选聚类集合。 As an example, when the similarity score _i ¹² all exceeds the first preset threshold, the multiple scores are added and the corresponding second average similarity value is calculated

Judging the magnitude relationship between the second average similarity and the second preset threshold, and when all the second similarity averages exceed the second preset threshold, determine the cluster set C as a candidate cluster set.

Step C: Obtain all candidate cluster sets corresponding to the second text, use the candidate cluster set that satisfies the preset requirements in the candidate cluster set as the target cluster set, and divide the second text into the Target cluster collection.

In an optional embodiment of the present invention, steps A and B are used to obtain all candidate cluster sets corresponding to the second text. Obtain a target cluster set from all the candidate sets according to a preset requirement, and add the second text to the target cluster set.

In an alternative embodiment of the present invention, the preset requirement may be to select the cluster set with the highest average similarity as the target cluster set.

其中，第二相似度均值最大的为

根据预设要求，将候选聚类集合P1确定为目标聚类集合，将所述第二文本t ²加入至所述聚类集合P1中。 As an example, using steps A and B to obtain all candidate cluster sets corresponding to the second text are P1, P2...Pn, respectively, and the corresponding mean values of the second similarities are

Among them, the largest mean value of the second similarity is

According to a preset requirement, the candidate cluster set P1 is determined as the target cluster set, and the second text t ^{2 is} added to the cluster set P1.

In this way, when the second preset clustering method is adopted, the semantic similarity and syntactic similarity between the texts are calculated, and the value of the parameter can be set to meet the needs of different application scenarios and improve the accuracy of clustering.

Specific example one

When the application scenario is a personality recommendation scenario, the clustering requirements are: high efficiency requirements and low precision requirements, then the methods provided by the embodiments of the present invention include:

S101: Select the first preset clustering method to perform two clustering processes, and cluster the information text in the system into multiple second clustering sets.

S102: Match a second cluster set related to the user according to the user attributes and historical data; if the user's historical access information text exists in the second cluster set, or the label information of the second cluster set matches the user The attributes match, and the specific method is defined by the personality recommendation scenario.

S103: Treat all information in the matched second cluster set as recommendation information and return it to the user.

In this way, the above-mentioned applications can accurately and quickly implement personalized recommendations with good controllability.

Specific example two

When the application scenario is an intelligent question and answer scenario, the clustering requirements are: low efficiency requirements and high precision requirements, then the methods provided by the embodiments of the present invention include:

S201: Select the first preset clustering method to perform two clustering processes, and cluster the question and answer library in the system into multiple second clustering sets.

S202: Analyze all samples in multiple second cluster sets, and select the recommended samples of the second cluster set. The specific recommendation scheme is defined by the intelligent question and answer scenario.

S203: Configure a standard answer for the recommended samples in the multiple second cluster sets, so that each sample in the multiple second cluster sets and the standard answer form a question and answer pair, and place them in the question and answer database.

In this way, the above application can automatically and accurately expand the question and answer library of intelligent question and answer, avoiding a lot of manual operations.

Specific example three

When the application scenario is an information retrieval scenario, the clustering requirements are: high efficiency requirements and high precision requirements, then the methods provided by the embodiments of the present invention include:

S301: Select a first preset clustering method and a second preset clustering method for clustering processing, and cluster the index text in the system into multiple second clustering sets.

S302: When an external retrieval request is received, the request information is matched with one or more samples in the second cluster set one by one to obtain a matching second cluster set; the method of clustering matching is defined by a specific information retrieval scenario .

S303: Match all the samples in the matching second cluster set with the external search request, and the matched samples are returned to the user as a search result.

In this way, the above applications can accurately and quickly realize information retrieval, and avoid the computational overhead of large-scale index information retrieval one by one.

Specific example four

When the application scenario is a digital library scenario, the clustering requirements are: low efficiency requirements and low precision requirements, then the methods provided by the embodiments of the present invention include:

S401: Two clustering methods can be freely matched, for example, first select the first preset clustering method, and then select the second preset clustering method, cluster all the data samples in the digital library to obtain multiple The second cluster collection.

S402: Use the subject model and other methods to obtain the label information of each second cluster set.

S403: The samples of the second cluster set use the label information of the set. Since one sample may exist in multiple second cluster sets, each data sample has multiple label information.

In this way, the above-mentioned applications can add tag information to large-scale material texts, such as adding category information to books, thereby facilitating the system management of the digital library.

An embodiment of the present invention also provides an information processing apparatus. As shown in FIG. 5, the information processing apparatus 500 includes:

The first clusterer 510 is configured to perform clustering processing on the text in the original text set in a preset clustering manner to obtain multiple first cluster sets.

The second clusterer 520 is configured to perform clustering processing on the text in each first cluster set by using a preset clustering method to obtain multiple second cluster sets.

Wherein, the preset clustering mode is the first preset clustering mode or the second preset clustering mode.

In an embodiment, the first preset clustering mode and the second preset clustering mode are determined based on the efficiency requirements and/or precision requirements of text clustering.

In an embodiment, the first clusterer 510 or the second clusterer 520 is used to extract each text in the set of text to be processed, and represent each text as a signature vector; for each text, the signature vector Perform segmentation processing to obtain multiple signature vector segments, and perform clustering processing on each signature vector segment.

In an embodiment, the first clusterer 510 or the second clusterer 520 is used to obtain a word sequence of text, and to obtain the weight of each word in the word sequence, based on obtaining the word sequence The weight of each word obtains a sequence of weighted words; hashing the sequence of weighted words to obtain a sequence of weighted hash values; combining each weighted hash value in the sequence of weighted hash values Processing to obtain a weighted hash value corresponding to the text; performing binarization processing on the weighted hash value to obtain a binary signature vector.

In an embodiment, the first clusterer 510 or the second clusterer 520 is used to segment the binary signature vector of each text to obtain multiple binary signature vector segments; for each binary The signature vector segment is hashed to obtain the hash value corresponding to the binary signature vector segment; the corresponding text with the same hash value is divided into the same clustering set.

In an embodiment, the first clusterer 510 or the second clusterer 520 is used to calculate the similarity between any two texts in the set of text to be processed. The text in the text collection to be processed is clustered.

In an embodiment, the first clusterer 510 or the second clusterer 520 is used to calculate a first similarity between two first texts in the set of text to be processed, and determine the first similarity Whether the degree exceeds a first preset threshold; the first text is any text in the to-be-processed text set; when the first similarity exceeds the first preset threshold, the two A text is divided into the same clustering set.

In an embodiment, the first clusterer 510 or the second clusterer 520 is respectively used to calculate between the second text in the to-be-processed text set and the first text in the cluster set The second similarity, determine whether all the second similarities exceed the first preset threshold; the second text is other text than the first text in the text to be processed; when the When all the second similarities exceed the first preset threshold, obtain a second average value of the second similarity between the second text and the first text in the cluster set, and determine whether the second average value of the similarity Exceeding a second preset threshold; when the mean value of the second similarity exceeds the second preset threshold, determining the cluster set as a candidate cluster set; obtaining all candidate clusters corresponding to the second text Set, taking the candidate cluster set that satisfies the preset requirements among the candidate cluster sets as the target cluster set, and dividing the second text into the target cluster set.

In an embodiment, the to-be-processed text set is the original text set or the first cluster set.

In this way, the clustering process is performed twice by the preset clustering method, and the subsequent clustering process is to perform clustering processing again on the text in the first clustering set, and the hierarchical clustering processing method is adopted to avoid In the case of inconsistent clustering results, on the other hand, the clustering accuracy and clustering efficiency are greatly improved. In addition, different combinations of clustering methods can be selected according to clustering requirements, which makes clustering processing more flexible and effective, and can meet the needs of different application scenarios.

For the device embodiments of the present invention, refer to the method embodiments of the present invention described above.

In the above embodiments of the invention, the first clusterer 510 and the second clusterer 520 in the information processing device 500 can be implemented by a CPU, DSP, MCU, or FPGA in practical applications.

It should be noted that when the information processing apparatus provided in the above embodiment performs information processing, only the above-mentioned division of each program module is used as an example for illustration. In practical applications, the above-mentioned processing may be allocated by different program modules according to needs. That is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the information processing apparatus and the information processing method embodiment provided in the above embodiments belong to the same concept. For the specific implementation process, see the method embodiment, and details are not described here.

An embodiment of the present invention also provides a computer-readable storage medium on which an executable program is stored, and when the executable program is executed by a processor, any of the foregoing information processing methods is implemented.

An embodiment of the present invention also provides an information processing apparatus including: a processor and a memory for storing a computer program that can be run on the processor, wherein the processor runs when the computer program is run To execute any information processing method implemented in the embodiments of the present invention.

It is understood that the memory may be implemented by any type of volatile or non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be read-only memory (ROM, Read Only Memory), programmable read-only memory (PROM, Programmable Read-Only Memory), erasable programmable read-only memory (EPROM, Erasable Programmable Read- Only Memory), Electrically Erasable Programmable Read Only Memory (EEPROM, Electrically Erasable Programmable Read-Only Memory), Magnetic Random Access Memory (FRAM, Ferromagnetic Random Access Memory), Flash Memory (Flash Memory), Magnetic Surface Memory , Compact disc, or read-only compact disc (CD-ROM, Compact, Read-Only Memory); the magnetic surface memory can be a disk storage or a tape storage. The volatile memory may be a random access memory (RAM, Random Access Memory), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM, Static Random Access Memory), synchronous static random access memory (SSRAM, Synchronous Static Random Access Memory), dynamic random access Memory (DRAM, Dynamic Random Access), Synchronous Dynamic Random Access Memory (SDRAM, Synchronous Dynamic Random Access Memory), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM, Double Data Rate, Synchronous Dynamic Random Access Random Access Memory), enhanced Type synchronous dynamic random access memory (ESDRAM, Enhanced Synchronous Dynamic Random Access Memory), synchronous connection dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), direct memory bus random access memory (DRRAM, Direct Rambus Random Access Random Access Memory ). The memories described in the embodiments of the present invention are intended to include but are not limited to these and any other suitable types of memories.

The method disclosed in the foregoing embodiments of the present invention may be applied to a processor, or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in the processor or instructions in the form of software. The aforementioned processor may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor may implement or execute the disclosed methods, steps, and logical block diagrams in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present invention may be directly implemented and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium. The storage medium is located in a memory. The processor reads the information in the memory and completes the steps of the foregoing method in combination with its hardware.

In an embodiment, the information processing apparatus may be implemented by one or more application specific integrated circuits (ASIC, Application Integrated Circuit), DSP, programmable logic device (PLD, Programmable Logic Device), complex programmable logic device (CPLD, Complex Programmable Logic (Device), FPGA, general-purpose processor, controller, MCU, microprocessor (Microprocessor), or other electronic components to implement the aforementioned method.

In the several embodiments provided in this application, it should be understood that the disclosed device and method may be implemented in other ways. The device embodiments described above are only schematics. For example, the division of the units is only a division of logical functions. In actual implementation, there may be other division methods, such as: multiple units or components may be combined, or Can be integrated into another system, or some features can be ignored, or not implemented. In addition, the coupling or direct coupling or communication connection between the displayed or discussed components may be through some interfaces, and the indirect coupling or communication connection of the device or unit may be electrical, mechanical, or other forms of.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units; Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, the functional units in the embodiments of the present invention may all be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above integration The unit can be implemented in the form of hardware, or in the form of hardware plus software functional units.

Those of ordinary skill in the art may understand that all or part of the steps to implement the above method embodiments may be completed by program instructions related hardware. The foregoing program may be stored in a computer-readable storage medium, and when the program is executed, The steps of the above method embodiments are included; and the foregoing storage medium includes various media that can store program codes, such as a mobile storage device, ROM, RAM, magnetic disk, or optical disk.

Alternatively, if the above integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention can be embodied in the form of software products in essence or part of contributions to the existing technology. The computer software products are stored in a storage medium and include several instructions for A computer device (which may be a personal computer, a server, or a network device, etc.) executes all or part of the methods described in the embodiments of the present invention. The foregoing storage media include various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

The above are only specific embodiments of the present invention, but the scope of protection of the present invention is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present invention. It should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims (20)

An information processing method, wherein the method includes: Perform clustering on the text in the original text collection by using a preset clustering method to obtain multiple first cluster collections; Perform clustering on the text in each first clustering set by using a preset clustering method to obtain multiple second clustering sets; The preset clustering mode is the first preset clustering mode or the second preset clustering mode. The method according to claim 1, wherein The first preset clustering mode and the second preset clustering mode are determined based on the efficiency requirements and/or precision requirements of text clustering. The method according to claim 1, wherein the clustering using the preset clustering method includes: Extract each text in the to-be-processed text collection and represent each text as a signature vector; Segment the signature vectors of each text to obtain multiple signature vector segments, and cluster each signature vector segment. The method of claim 3, wherein the expressing each text as a signature vector includes: Obtaining a word sequence of text, and obtaining the weight of each word in the word sequence, and obtaining a weighted word sequence based on obtaining the weight of each word in the word sequence; Hashing the weighted word sequence to obtain a weighted hash value sequence; Performing merge processing on each weighted hash value in the weighted hash value sequence to obtain a weighted hash value corresponding to the text; Perform binary processing on the weighted hash value to obtain a binary signature vector. The method according to claim 4, wherein the segmenting the signature vector of each text to obtain multiple signature vector segments, and clustering each signature vector segment includes: Segment the binary signature vector of each text to obtain multiple binary signature vector segments; Perform a hash operation on each binary signature vector segment to obtain a hash value corresponding to the binary signature vector segment; Divide the corresponding text with the same hash value into the same clustering set. The method according to claim 1, wherein the clustering using the preset clustering method includes: Calculate the similarity between any two texts in the text set to be processed, and perform clustering processing on the texts in the text set to be processed based on the calculation result of the similarity. The method according to claim 6, wherein the clustering processing of the text in the to-be-processed text set based on the calculation result of the similarity comprises: Calculate a first similarity between two first texts in the to-be-processed text set, and determine whether the first similarity exceeds a first preset threshold; the first text is any of the to-be-processed text sets A text When the first similarity exceeds the first preset threshold, the two first texts are divided into the same clustering set. The method of claim 7, wherein the method further comprises: Separately calculating a second similarity between the second text in the to-be-processed text set and the first text in the clustering set to determine whether all the second similarities exceed the first preset threshold; The second text is other text than the first text in the text to be processed; When the second similarities all exceed the first preset threshold, obtain a second average value of the second similarity between the second text and the first text in the cluster set, and determine the second similarity Whether the average degree exceeds the second preset threshold; When the average value of the second similarity exceeds the second preset threshold, determine the cluster set as a candidate cluster set; Obtain all candidate cluster sets corresponding to the second text, use the candidate cluster set that satisfies the preset requirements in the candidate cluster set as the target cluster set, and divide the second text into the target cluster set. The method according to any one of claims 3 to 8, wherein the set of text to be processed is the original text set or the first cluster set. An information processing device, wherein the device includes: The first clusterer is used to perform clustering processing on the text in the original text set by using a preset clustering method to obtain multiple first cluster sets; A second clusterer, configured to perform clustering on the text in each first cluster set by using a preset clustering method to obtain multiple second cluster sets; Wherein, the preset clustering mode is the first preset clustering mode or the second preset clustering mode. The device according to claim 10, wherein The first preset clustering mode and the second preset clustering mode are determined based on the efficiency requirements and/or precision requirements of text clustering. The apparatus according to claim 10, wherein the first clusterer or the second clusterer is used to extract each text in the set of text to be processed, and express each text as a signature vector; for each text The signature vectors are segmented, multiple signature vector segments are obtained, and each signature vector segment is clustered. The apparatus according to claim 12, wherein the first clusterer or the second clusterer is used to obtain a word sequence of text, and obtain the weight of each word in the word sequence, based on obtaining the The weight of each word in the word sequence is used to obtain a weighted word sequence; the hashed word sequence is hashed to obtain a weighted hash value sequence; and each weighted hash in the weighted hash value sequence The values are merged to obtain a weighted hash value corresponding to the text; the weighted hash value is binarized to obtain a binary signature vector. The apparatus according to claim 13, wherein the first clusterer or the second clusterer is used to segment the binary signature vector of each text to obtain multiple binary signature vector segments; Each binary signature vector segment is hashed to obtain a hash value corresponding to the binary signature vector segment; the corresponding text with the same hash value is divided into the same clustering set. The apparatus according to claim 10, wherein the first clusterer or the second clusterer is used to calculate the similarity between any two texts in the text set to be processed, based on the calculation result of the similarity Perform clustering processing on the text in the text set to be processed. The apparatus according to claim 15, wherein the first clusterer or the second clusterer is used to calculate a first similarity between two first texts in the set of text to be processed, and determine the first Whether the similarity exceeds a first preset threshold; the first text is any text in the to-be-processed text set; when the first similarity exceeds the first preset threshold, the two The first text is divided into the same cluster set. The apparatus according to claim 16, wherein the first clusterer or the second clusterer is configured to respectively calculate the second text in the to-be-processed text set and the first text in the cluster set The second similarity between the two, to determine whether all the second similarities exceed the first preset threshold; the second text is other text than the first text in the text to be processed; When all the second similarities exceed the first preset threshold, obtain a second average value of the second similarity between the second text and the first text in the cluster set, and determine the second average value of the similarity Whether it exceeds a second preset threshold; when the average value of the second similarity exceeds the second preset threshold, determine the cluster set as a candidate cluster set; obtain all candidate clusters corresponding to the second text Class set, taking the candidate cluster set that satisfies the preset requirements among the candidate cluster sets as the target cluster set, and dividing the second text into the target cluster set. The device according to any one of claims 12 to 17, wherein the set of text to be processed is the original text set or the first cluster set. A computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented. An information processing device, comprising: a processor and a memory for storing a computer program that can be run on the processor, wherein when the processor is used to run the computer program, any one of claims 1 to 9 is executed Item of the method.

Images