TWI486796B - Text filtering method and text filtering system - Google Patents

Description

Text filtering method and text filtering system

The present application relates to the field of Internet application technologies, and in particular, to a text filtering method and a text filtering system.

With the continuous development of the Internet, the amount of information on the Internet has increased dramatically, and the openness of the Internet has led to a lot of bad information in the Internet. Therefore, monitoring and filtering of information on the Internet has become a common demand.

Application content filtering technology can filter bad information on the Internet to ensure the security of the network environment. There are many forms of information on the web, and the text form is the most common one. Text filtering refers to the process of finding specific text from a large amount of text information. At present, common text filtering methods are implemented based on basic keyword matching technology: the system is based on a plurality of pre-set keywords related to bad information. The search is performed in the input text, and if the content matching the keyword is found in the input text, the part or all of the input text is filtered or replaced.

The above text filtering method can only filter out the text that exactly matches the keyword, but can't judge the position or attitude of the whole text. For example, in the e-commerce website, the "snicker" is defined as the filtering keyword, but the existing one. The text filtering method also filters legitimate texts such as "no sales of bugs" as bad information. It can be seen that the existing text filtering method based on the basic keyword matching technology has a low recognition accuracy rate and cannot meet the practical application requirements of information filtering.

To solve the above technical problem, the embodiment of the present application provides a text filtering method and a text filtering system to improve the accuracy of text filtering. The technical solution is as follows: The present application provides a text filtering method, including: predefining semantics in a text filtering system. a keyword, the semantic keyword, consisting of at least a basic keyword and a logical relationship; after the text filtering system obtains the input text, searching for the basic key constituting the semantic keyword in the input text according to the predefined semantic keyword a word; if a text content matching at least one of the basic keywords is found in the input text, further performing semantic matching on the found text content; the semantic matching comprises: according to a logical relationship constituting the semantic keyword And matching the found text content with the semantic keyword; if the semantic matching is successful, filtering the successfully matched text content.

The present application further provides a text filtering system, including: a keyword storage unit, configured to store a predefined semantic keyword, the semantic keyword being composed of at least a basic keyword and a logical relationship; a basic searching unit for After obtaining the input text, the text filtering system searches for the basic keyword constituting the semantic keyword in the input text according to the predefined semantic keyword; the semantic matching unit is configured to find in the input text in the basic search unit And semantically matching the found text content when the text content matches at least one of the basic keywords; the semantic matching unit includes: the text to be found according to the logical relationship constituting the semantic keyword A logical matching sub-unit that matches the content with the semantic keyword; and a filtering processing unit configured to filter the successfully matched text content when the semantic matching unit matches successfully.

The text filtering method and system provided by the present application filter the text content by using a combination of basic keywords and logical relationships, and can effectively combine the semantics of the basic keywords in the entire text compared with the prior art. Improve the accuracy of filtering.

Existing text filtering methods are only filtered based on simple keywords and do not have logic analysis capabilities, so there are many false positives. For example, the "prohibited sales of bugs" text mentioned above, although containing the keyword "snuggle", combined with the negative word "prohibited", makes the piece of text actually legal information and should not be filtered. To solve this problem, the embodiment of the present application provides a text filtering method as follows: a semantic keyword is defined in a text filtering system, and the semantic keyword is composed of at least a basic keyword and a logical relationship; the text filtering system obtains input text. Then, according to the predefined semantic keyword, the basic keyword constituting the semantic keyword is searched in the input text; if the text content matching the at least one basic keyword is found in the input text, further The found text content is semantically matched; the semantic matching includes: matching the found text content with the semantic keyword according to the logical relationship constituting the semantic keyword; if the semantic matching is successful, the matching is successful The text content is filtered.

The above text filtering method uses a combination of basic keywords and logical relationship characters to filter text content, and can effectively combine the semantics of basic keywords in the entire text to reduce the false positives and improve the situation compared with the prior art. The accuracy of the filter.

The technical solutions in the embodiments of the present application are clearly and completely described in the following, in which the technical solutions in the embodiments of the present application are clearly and completely described. The embodiments are only a part of the embodiments of the present application, and not all of them. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present application without departing from the inventive scope shall fall within the scope of the application.

Embodiment 1:

In the embodiment of the present application, the text content is filtered based on the semantic keyword, and the semantic keyword includes two basic components: a basic keyword and a logical relationship. Wherein, the basic keyword is an independent word or phrase, which is equivalent to the simple keyword used in the prior art; and the logical relationship is used to represent the logical relationship, and the basic logical relationship includes: "and", "or "," "Non", etc., in the semantic keyword, can be represented by the symbols "&", "|", "~", respectively. Here are a few simple examples of semantic keywords that apply to text filtering for e-commerce sites:

a) Mobile phone tapping ~ anti

The semantics of the semantic keyword representation is: if the information of the product includes "mobile phone eavesdropping" and does not include "reverse", the product information needs to be filtered.

b) surveillance camera | wireless surveillance camera

The semantics of the semantic keyword representation is that if the information of the product includes a "surveillance camera" or includes a "wireless surveillance camera", the product information needs to be filtered.

c) Military & Cable Ties

The semantics of the semantic keyword representation is: if the information of the product includes "military" and includes "bundle", the product information needs to be filtered.

For a semantic keyword, its simplest form should be: 2 basic keywords + 1 logical relationship, which are the case in the above three examples. For the case where only one basic keyword is included, it is actually the same as the prior art, and the embodiment of the present application does not introduce this case. It can be understood that in a semantic keyword, more basic keywords and logical relationships can be included to represent more complex semantics, such as:

d) mobile phone tapping ~ (reverse | anti)

The semantics of the semantic keyword representation is: if the information of the product includes "mobile phone eavesdropping" and does not include "reverse" or "prevention", the product information needs to be filtered.

In the preferred solution of the present application, the content of the semantic keyword can be further extended, for example:

Filters can be added to semantic keywords. In fact, unlike the basic keywords and logical relationships described above, the filter condition is independent of the specific content of the text, and its role is to further limit the filtering from other aspects of the text. For example, limit the source of text, categories, and so on, for more accurate filtering.

In the semantic keyword, a filtering behavior may be further added to indicate what kind of processing is performed for the content that matches the semantic keyword text portion, such as content masking, content replacement, and the like.

In the following three examples, filter conditions and filtering behaviors are added in the foregoing a), b), and c), and the extended form of the semantic keyword is schematically illustrated. The semicolon is preceded by a basic keyword and a logical relationship. After the semicolon is extended content, the extensions are separated by commas. Of course, this embodiment does not limit the specific format of the semantics.

A1) Mobile phone tapping ~ reverse; product category: 1002, filtering behavior: off the shelf,

The semantics of the semantic keyword representation is: if the information of the product includes "mobile phone eavesdropping" and does not include "reverse", and the product category is 1002, the product information needs to be taken off the shelf.

B1) surveillance camera | wireless surveillance camera; product category: 101, filtering behavior: off the shelf,

The semantics of the semantic keyword representation is that if the information of the product includes a "surveillance camera" or a "wireless surveillance camera" and the product category is 101, the product information needs to be taken off the shelf.

C1) military & cable tie; product category: 50001, filtering behavior: off the shelf,

The semantics of the semantic keyword representation is: if the information of the product includes "military" and includes "ticket", and the product category is 50001, the product information needs to be taken off the shelf.

The following is a description of the present embodiment in conjunction with a specific process. FIG. 1 is a flowchart of a text filtering method according to an embodiment of the present application, including the following steps: S101: After the text filtering system obtains input text, according to a predefined semantic keyword In the input text, the basic keywords constituting the semantic keyword are searched; in this step, after obtaining an input text, the system first searches for the basic keyword in the input text, and records the search result. For example, for b) or b1) above, the system will first look up the contents of the "surveillance camera" and "wireless surveillance camera" in the input text. The specific implementation of this step may be similar to the method for matching based on simple keywords in the prior art, and is not described in detail in this embodiment.

S102. If the text content matching the at least one basic keyword is found in the input text, the searched text content is further semantically matched; in S101, only the content of the basic keyword is searched, if not Find content that matches any of the basic keywords, indicating that you don't need to filter the input text; if you find text content that matches at least one of the basic keywords, you need to further further find the text content you are looking for. Semantic keywords are compared, this step is called semantic matching.

If only the basic keyword and the logical relationship are included in the semantic keyword, the specific content of the semantic matching is: matching the found text content with the semantic keyword according to the logical relationship in the predefined semantic keyword. . For example, for the aforementioned a), the system finds the basic keyword "mobile phone eavesdropping" in the input text, and does not find the basic keyword "reverse", that is, the actual search result of the two basic keywords conforms to the semantic keyword a. The logical relationship between the two basic keywords defined in the "Non", therefore, the found content matches the semantic keyword a) successfully; for the aforementioned c), the system finds the basic keyword in the input text. Tethered, and did not find the basic keyword "military", that is, the actual search results of the two basic keywords do not match the logical relationship "and" of the two basic keywords defined in the semantic keyword c), therefore The found content fails to match the semantic keyword a); if the semantic keyword also includes the extended content "filter condition", then when the semantic matching is performed, the matching between the attribute of the input text and the filter condition is further considered. .

S103. If the semantic matching is successful, filtering the successfully matched text content.

For the text that successfully matches the semantic keyword in S102, the system will perform filtering processing. If the "filtering behavior" is included in the semantic keyword, the system will filter the text according to the specific content of the "filtering behavior". If the "filtering behavior" is not included in the semantic keyword, the system will filter the text content according to the preset preset mode.

Embodiment 2:

In the prior art, each word needs to be searched one by one in the input text. This embodiment provides an improved basic keyword search method for step S101 in the first embodiment to improve the processing efficiency of the keyword search.

In the actual text filtering application, many words that need to be filtered have the same part, such as: "snicker", "eavesdropping device", "eavesdropping software", etc. For such words, tree search can be used. Method to improve the efficiency of searching.

First, each basic keyword is stored in a tree structure in units of characters in the system. The first character of the basic keyword is the root node, and the last character is the leaf node. The basic keywords with the same first character share the same root node. For example, for the three basic keywords "ab", "abc", and "ade", they can be stored in the structure shown in FIG. 2.

In FIG. 2, a circle represents a root node or a general node, and a diamond represents a leaf node. Since the words "ab", "abc", and "ade" have the same first character "a", they share the same root node. 1; The last characters of the three words are "b", "c", and "e", respectively, so the three characters are leaf nodes 2, 3, and 5, respectively. It should be noted that for the character "b", although it is not the last character in the second word, it is the last character in the first word, and therefore remains a leaf node. That is to say, the leaf node is not necessarily the end node of the tree structure, but the end node of the tree structure must be the leaf node.

FIG. 3 is a flow chart of a basic keyword search method based on a tree structure, including the following steps:

S301: Acquire one character in the input text; set the character as a current character, and set a root node of the tree structure as a current node. According to the actual filtering application requirements, the obtained character may be the first character of the input text or a character selected from any part of the input text.

S302: Match the current character with the current node; if the matching is successful, execute S303; otherwise, execute S304.

S303. Determine whether the current node has a child node. If not, end the search; if yes, go to the next character of the current character, the child node of the current node, and then execute S302.

S304. Determine whether the current node has a sibling node. If not, end the search; if yes, keep the current character unchanged, go to the sibling node of the current node, and then execute S302.

After the search is completed, the system connects the current node to the root node to obtain a matching path, and determines the found basic keywords according to the matching matching leaf nodes on the matching path.

The following is a description of the basic keyword search method based on the tree structure in combination with two specific examples:

1) Assuming that the input text is adf, after the system obtains the character "a", it traverses the root node in the keyword library, finds that the matching with node 1 is successful, and node 1 has child nodes, and further the character "d" and the node The child nodes 2 and 4 of 1 match.

The character "d" matches the node 4 successfully, and the node 4 has a child node, then the character "f" is further matched with the child node 5 of the node 4, the character "f" fails to match the node 5, and the node 5 does not have Other sibling nodes, at this point, the search ends. The current matching path is 1-4-5, and the leaf node that matches the success is not included in the path. Therefore, it can be determined that the basic keyword is not found in the input text.

2) Assuming that the input text is abc, after the system obtains the character "a", it traverses the root node in the keyword library and finds that the matching with node 1 is successful, and node 1 has a child node, and further the character "b" is The child nodes 2, 4 of node 1 match.

The character "b" matches the node 2 successfully, and the node 2 has the child node, then the character "c" is further matched with the child node 3 of the node 2, the character "c" matches the node 4 successfully, and the node 3 does not have The child node ends the search at this time. The current matching path is 1-2-3, where nodes 2 and 3 are leaf nodes that match successfully. Therefore, according to the contents of nodes 2 and 3, it is determined that the basic keyword "ab" is found in the input text and "abc".

It can be seen that the basic keyword search method based on the tree structure is applied, and the matching operation of each level is performed only for the node with the last matching success, so that each character and all the keys for the input text are not needed. The word characters are matched one by one, thereby effectively improving the processing efficiency of the keyword search.

In the above example, the first character is taken as the root node, and this method is applicable to the case where a plurality of basic keywords have the same first code. It can be understood that, for a case where multiple basic keywords have the same tail code, for example, “telephone eavesdropping”, “mobile phone eavesdropping”, “mobile phone monitoring”, etc., the last character of the basic keyword may be used as the root node, The first character is a tree structure of leaf nodes to store keywords. Correspondingly, in the matching process, the characters of the input text should be matched in the order from the back to the front, and the specific method implementation is similar to the foregoing, and the description is not repeated here.

In addition, in order to avoid text filtering, many people now use special characters in the published text, such as "stealing-listening", "spying", etc. In this case, the dictionary function can be further combined. Find keywords.

The dictionary defines a set of characters and defines the prototype of the character. The prototype can be the character itself. For example, if the prototype of the character 'a' is 'a' itself, it can be another character, such as a traditional character. The prototype is the corresponding Simplified Chinese. Commonly used dictionaries include: simplified dictionary, traditional dictionary, English dictionary, digital dictionary and so on. In addition, the business personnel can define the dictionary by themselves, for example, by defining the prototype of the character "-" as an empty character.

According to the foregoing step S302, the system may find in the dictionary whether the current character has a prototype character before matching the current character with the current node; if yes, convert it to the corresponding prototype character, and The prototype character is the current character and matches the current node.

In the foregoing example 2) of the present embodiment, if the input text is aBc, the system traverses all the dictionaries and finds that the character "B" has the prototype "b" before matching the character "B" with the node 2. , converts the "B" in the original input text to the prototype "b", and then matches the node with the "b" as the current character.

For text such as "stealing-listening", the system will convert the character "-" to an empty character by querying the dictionary. During the matching process, when the system matches "stealing", the empty character will be skipped and directly matched with "listening".

It can be seen that by querying the dictionary and converting the characters, the system can identify more bad information, thereby achieving better text filtering effect.

Corresponding to the above method embodiment, the present application further provides a text filtering system, as shown in FIG. 4, including: a keyword storage unit 410, configured to store a predefined semantic keyword, the semantic keyword, at least by a basic key a basic search unit 420, configured to: after the text filtering system obtains the input text, search for the basic keyword constituting the semantic keyword in the input text according to the predefined semantic keyword; semantic matching The unit 430 is configured to further perform semantic matching on the found text content when the basic search unit 420 finds the text content matching the at least one basic keyword in the input text; the semantic matching unit 430 includes: a logic matching sub-unit 431 for matching the found text content with the semantic keyword according to a logical relationship constituting the semantic keyword; and a filtering processing unit 440, configured to be successful when the semantic matching unit 430 matches , filter the text content that matches the success.

The keyword storage unit stores the basic keyword in a tree structure in units of characters; wherein, the first character of the basic keyword is a root node, and the last character is a leaf node, and has the same first character. The basic keyword shares the same root node; as shown in FIG. 5, the basic search unit 420 may include: a text acquisition sub-unit 421, configured to acquire one character c1 in the input text; and a character matching sub-unit 422. For the current character of c1, the root node of the tree structure is the current node, and the current character is matched with the current node; if the current character matches the current node successfully, and the current node has a child node, the current node will be current The next character of the character is matched with the child node of the current node; if the current character fails to match the current node, and the current node has a sibling node, the current character is matched with the sibling node of the current node; a determining sub-unit 423, configured to connect the current node to the root node to obtain a matching path, and determine the checked according to the matching leaf node on the matching path. The basic keyword to be obtained; as shown in FIG. 6, the basic search unit 420 may further include: a character conversion sub-unit 424, configured to search the dictionary for the current word before the character matching sub-unit 422 performs matching. Whether the element has a prototype character, and if so, converts it to a corresponding prototype character; then the character matches sub-unit 423, with the prototype character being the current character, matching the current node.

The composition of the semantic keyword may further include: a filtering condition; the semantic matching unit 430 further includes: a category matching sub-unit 432 for matching the attribute of the input text with the filtering condition, as shown in FIG. 7.

The composition of the semantic keyword may further include: a filtering behavior; and the filtering processing unit is configured to perform filtering processing on the found text content according to the filtering behavior.

For the convenience of description, the above system is described as being divided into various units by function. Of course, the functions of each unit can be implemented in the same software or software and/or hardware in the implementation of the present application.

As can be seen from the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of a software plus a necessary universal hardware platform. Based on such understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in the form of a software product, which can be stored in a storage medium such as a ROM/RAM, a disk, A disc or the like includes instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform the methods described in various embodiments of the present application or portions of the embodiments.

The various embodiments in the specification are described in a progressive manner, and the same or similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment. The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, ie may be located in one Places, or they can be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the embodiment. Those of ordinary skill in the art can understand and implement without any creative effort.

This application can be used in a variety of general purpose or special purpose computing system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, small computers, Large computers, decentralized computing environments including any of the above systems or devices, and more.

The application can be described in the general context of computer-executable instructions executed by a computer, such as a program module. Generally, a program module includes routines, programs, objects, components, data structures, and the like that perform particular tasks or implement particular abstract data types. The present application can also be practiced in a distributed computing environment where tasks are performed by remote processing devices that are connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

The above description is only a specific embodiment of the present application, and it should be noted that those skilled in the art can also make several improvements and retouchings without departing from the principles of the present application. It should be considered as the scope of protection of this application.

410. . . Keyword storage unit

420. . . Basic search unit

430. . . Semantic matching unit

431. . . Logical matching subunit

440. . . Filter processing unit

421. . . Text acquisition subunit

422. . . Character matching subunit

423. . . Determining subunit

424. . . Character conversion subunit

432. . . Category matching subunit

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings to be used in the embodiments or the description of the prior art will be briefly described below. Obviously, the drawings in the following description are only Some of the embodiments described in the application can be used to obtain other drawings based on these drawings without departing from the prior art.

FIG. 1 is a flowchart of a text filtering method according to an embodiment of the present application;

2 is a schematic diagram of a basic keyword tree storage structure according to an embodiment of the present application;

3 is a flowchart of a basic keyword searching method according to an embodiment of the present application;

4 is a schematic structural diagram of a text filtering system according to an embodiment of the present application;

FIG. 5 is a schematic structural diagram of a basic search unit according to an embodiment of the present application;

6 is another schematic structural diagram of a basic search unit according to an embodiment of the present application;

FIG. 7 is a schematic structural diagram of a semantic matching unit according to an embodiment of the present application.

Claims (8)

A text filtering method, comprising: predefining a semantic keyword in a text filtering system, the semantic keyword being composed of at least a basic keyword and a logical relationship, the basic keyword being in units of characters, according to The tree structure is stored in the text filtering system, wherein the first character of the basic keyword is the root node, the last character is the leaf node, and the basic keywords having the same first character share the same root node; the text filtering system obtains After inputting the text, searching for the basic keyword constituting the semantic keyword in the input text according to the predefined semantic keyword, comprising the steps of: acquiring one character c1 in the input text; and using c1 as the current character The root node of the tree structure is the current node, and the current character is matched with the current node; if the current character matches the current node successfully, and the current node has a child node, the next character of the current character is Matching with the child node of the current node; if the current character fails to match the current node, and the current node has a sibling node, Then matching the current character with the sibling node of the current node; repeating this step; and connecting the current node to the root node to obtain a matching path, and determining the found basic keyword according to the matching leaf node on the matching path; Searching for the text content matching the at least one of the basic keywords in the input text, and further performing semantic matching on the found text content; the semantic matching comprises: according to the logical relationship constituting the semantic keyword, The found text content is compared to the semantic keyword If the semantic matching is successful, the text content that matches the matching is filtered. The method of claim 1, wherein before the matching the current character with the current node, the method further comprises: searching the dictionary for whether the current character has a prototype character; if yes, converting the current character The corresponding prototype character is matched with the current node by using the prototype character as the current character. The method of any one of claims 1 to 2, wherein the semantic keyword further comprises: a filtering condition; the semantic matching further comprises: matching the attribute of the input text with the filtering condition . The method of any one of claims 1 to 2, wherein the composition of the semantic keyword further comprises: a filtering behavior; and the filtering processing of the found text content comprises: according to the filtering behavior , filter the text content found. A text filtering system, comprising: a keyword storage unit, configured to store a predefined semantic keyword, the semantic keyword being composed of at least a basic keyword and a logical relationship, the keyword storage unit, and the word The element is stored in a tree structure, wherein the first character of the basic keyword is the root node and the last character is a leaf node, and the basic keywords having the same first character share the same root. a basic search unit, configured to: after the text filtering system obtains the input text, search for the basic keyword constituting the semantic keyword in the input text according to the predefined semantic keyword; and the semantic matching unit is configured to The basic search unit further performs semantic matching on the found text content when the text content matching the at least one basic keyword is found in the input text; the semantic matching unit comprises: according to the constituent semantic keyword a logical matching unit, a logical matching subunit matching the found text content with the semantic keyword; and a filtering processing unit, configured to filter the successfully matched text content when the semantic matching unit matches successfully, The basic search unit includes: a text acquisition sub-unit, configured to acquire one character c1 in the input text; and a character matching sub-unit, configured to use c1 as the current character and the root node of the tree structure as The current node matches the current character with the current node; if the current character matches the current node successfully And the current node has a child node, then the next character of the current character is matched with the child node of the current node; if the current character fails to match the current node, and the current node has a sibling node, the current character is Matching with the sibling node of the current node; repeating this step; and determining a subunit for connecting the current node with the root node to obtain a matching path The path, and determining the found basic keyword according to the matching leaf node on the matching path. The system of claim 5, wherein the basic search unit further comprises: a character conversion sub-unit, configured to search the dictionary for the current character before the character matching sub-unit performs matching Has a prototype character, and if so, converts it to a corresponding prototype character; then the character matches the sub-unit, with the prototype character being the current character, matching the current node. The system according to any one of claims 5 to 6, wherein the semantic keyword further comprises: a filtering condition; the semantic matching unit further comprises: an attribute for the input text and the filtering The condition matches the category matching subunit. The system of any one of clauses 5 to 6, wherein the composition of the semantic keyword further comprises: a filtering behavior; the filtering processing unit, configured to search the text according to the filtering behavior The content is filtered.