JP2004280569A - Information monitoring device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To efficiently extract a site containing a large amount of information matching to an investigation object. <P>SOLUTION: An information monitoring device acquires many Web documents from the Internet, acquires site URLs to which the documents belong to, calculates the appearance frequency of rumor expression contained in the document and a site feature expressing the feature of the document contents with respect to each site URL, and stores the appearance frequency and the site feature into a site management table. For the Web document obtained on the basis of retrieval conditions inputted by a system user, the site feature for every URL is calculated by the same method mentioned above, and a site similar to the site feature contained in the feature list of the site specified by the system user is outputted as an objective site. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an information monitoring device for collecting documents on the Internet, extracting desired information, and delivering the desired information.
[0002]
[Prior art]
Many companies intend to extract information useful for business analysis from Web documents existing in large quantities on the Internet and use it for risk management of companies, identification of customer needs, sales promotion, and the like. However, information providing sites such as bulletin boards and e-mail magazines published on the Internet are increasing very rapidly, and the information provided on those sites is constantly updated, so it is impossible to investigate manually. It is.
[0003]
Demand for an information clipping service that receives a survey request from a company, extracts desired information such as product complaints and corporate reputation from the information provided on the Internet, creates a survey report, and distributes it to the requester Are also growing rapidly.
The information clipping service (1) collects web documents related to the contents of the survey, (2) extracts necessary information such as complaints and reputation from the collected web documents, and (3) organizes the extracted information. It is divided into three stages of processing, that is, distribution to the requesting company. Conventionally, there is no substitute system for performing all of the processes (1) to (3) automatically. Web documents are collected using a full-text search engine and other search software that are open to the public and used manually. Usually, the whole work is performed through the work.
[0004]
As a conventional information clipping service support technology, for example, in a Web page search report apparatus disclosed in Patent Document 1, a Web page search is performed by a plurality of search engines using search conditions specified by a research client. The URL (Uniform Resource Locator) of the Web page obtained as a result of the search is stored in the database together with the update date and time of the Web page. A newly obtained Web page or an updated Web page is downloaded as a report target. Further, the sentence before and after including the search keyword is extracted from the downloaded Web page, a search result report is created and provided to the survey client.
[0005]
[Patent Document 1]
JP 2001-325275 A
[0006]
[Problems to be solved by the invention]
As described above, in the conventional information clipping service, a Web page on which a search keyword is posted can be extracted. However, with regard to the information clipping service, it is desired not only to search on a Web page basis but also to find an information providing site itself that includes a lot of contents matching the purpose of the survey and to continuously monitor the information of the entire site. High demand.
The term “site” as used herein refers to a collection of Web pages related to the topic of information provision built on a certain Web server, and generally has a structure including a top page and lower layers below the top page. Have.
With the conventional method, the URL of each Web page can be extracted, but the URL (site URL) of the top page of the site to which the Web page belongs cannot be extracted. Furthermore, since there is no management function for judging the contents of each Web page included in the site, there is a problem that there is no means for finding a site that meets the purpose of the survey.
[0007]
SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems, and has as its object to obtain an information monitoring device that extracts a site that includes a large amount of information that meets a purpose of investigation.
[0008]
[Means for Solving the Problems]
An information monitoring device according to the present invention circulates and collects Web documents from documents on the Internet, outputs a collected document and a document URL, and a preset crawler unit from each document collected by the crawler unit. A first reputation calculating unit that extracts a reputation expression, calculates and outputs a reputation of each document based on an evaluation value corresponding to the extracted reputation expression, and a document URL output by a crawler unit. A first site extraction unit for extracting a site URL to which a document belongs, and a site feature representing the content feature of the site specified by the site URL are output, and the site URL and the site feature are stored in a site management table in association with each other. A first site feature calculation unit, a site selection unit that extracts a site feature B of a site URL specified by the system user from the site management table, A document search unit that searches for documents on the Internet based on search conditions and outputs document information including a document URL and an update date and time as a search result, and a URL management table that stores document information for each document URL. Among the document URLs output by the search unit, a document URL that is not registered in the URL management table and a document URL whose document information is updated are output as a new URL, and the document information of the new URL is registered in the URL management table. A new URL extraction unit, a download unit that obtains a document of the new URL from the Internet, and a pre-set reputation expression extracted from each document obtained by the download unit, and an evaluation value corresponding to the extracted reputation expression A second reputation calculating unit that calculates and outputs the reputation of each document based on the URL, and the attribute of each document from the new URL. A second site extraction unit for extracting a new site URL, a second site feature calculation unit for outputting a site feature A representing the content feature of the site specified by the new site URL, a site feature A and a site feature A similar site extraction unit that calculates the similarity of B, outputs a new site URL having a similarity of a certain value or more, and records the document information of the new site URL in a site management table.
[0009]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, various embodiments of the present invention will be described.
Embodiment 1 FIG.
FIG. 1 is a block diagram showing a configuration of an Internet information monitoring device (information monitoring device) 100 according to Embodiment 1 of the present invention.
As shown in the figure, the Internet information monitoring device 100 includes a crawler unit 1, a first reputation calculating unit 2, a first site extracting unit 3, a first site feature calculating unit 4, a site management table 5, a document searching unit. 6, new URL extraction unit 7, URL management table 8, download unit 9, second reputation calculation unit 10, second site extraction unit 11, second site feature calculation unit 12, site selection unit 13, and the like A site extraction unit 14 is provided.
[0010]
The Internet information monitoring apparatus 100 is a computer having an Internet connection function, a Web page browsing function, and the like. Also, a crawler unit 1, a first reputation calculation unit 2, a first site extraction unit 3, a first site feature calculation unit 4, a document search unit 6, a new URL extraction unit 7, a download unit 9, a second The reputation calculation unit 10, the second site extraction unit 11, the second site feature calculation unit 12, the site selection unit 13, and the similar site extraction unit 14 control the operation of the central processing unit of the Internet information monitoring device 100. The central processing unit is conveniently divided according to the module of the program to be executed.
[0011]
The site management table 5 and the URL management table 8 are sets of data records managed by a relational database or the like, and are stored in a storage device or the like outside the Internet information monitoring device 100. Note that the site management table 5 and the URL management table 8 may be stored in the same storage device.
[0012]
Next, information monitoring processing by the Internet information monitoring apparatus 100 will be described.
The information monitoring process can be divided into a two-stage process of a site registration process and a site information extraction process. First, the site registration process will be described with reference to the flowchart of FIG.
[0013]
First, in the crawler unit 1, documents on the Web are cyclically collected in order to collect candidates for monitoring target sites (step ST201). The crawler unit 1 is provided with a seed URL list that is a preset starting point of the cyclic collection. The crawler unit 1 sequentially retrieves the URL from the top of the seed URL list and accesses the corresponding URL. The crawler unit 1 analyzes the document content of the URL, and sequentially collects the linked document by sequentially following the URL specified as the link destination. At this time, document information such as a document title, a URL, and an update date and time is also acquired. The cyclic collection ends under a predetermined end condition such as the link depth and the number of accesses.
[0014]
Next, the first reputation calculating unit 2 analyzes each document collected in step ST201 and extracts a reputation expression included in each document. The reputation expression is, for example, information such as a reputation or a complaint about a company or a product. The first reputation calculation unit 2 quantifies the degree of reputation included in each document (reputation) by a later-described reputation calculation process and outputs the result as a reputation extraction result (step ST202).
[0015]
Next, the first site extraction unit 3 analyzes the contents of each document acquired in step ST201 and extracts the URL of the site to which each document belongs (step ST203).
For example, if a document is described in HTML (Hyper Text Markup Language), the HTML document is analyzed, and if there is an anchor tag corresponding to a hyperlink to the top page, the URL of the site to which each document belongs is found therefrom. Is extracted. More specifically, the link destination URL (href attribute value) of the anchor tag including the characteristic words (“top”, “TOP”, “HOME”, “home”, etc.) representing the top page is extracted.
Alternatively, the URL of a representative site may be held in advance, and the site URL may be determined by collation with the URL.
[0016]
Next, in the first site feature calculation unit 4, a result of comprehensively determining the contents of the documents belonging to the same site among the documents acquired in step ST201 is output as a site feature for each site (step ST204). .
A method for generating a site feature will be specifically described with reference to FIG. In the figure, W _i (I is a natural number from 1 to n) represents a certain word, and S _j (J is a natural number from 1 to m) represents a site URL identification number. The site URL identification number is a number assigned to one site URL.
[0017]
w _ij Is the word W _i Site URL identification number S _j Is a value indicating a weight indicating the feature of the appearance tendency in the site indicated by the symbol, and is calculated based on a predetermined calculation criterion.
w _ij An example of the calculation method of the will be described. First, the site URL identification number S _j A plurality of documents having a high reputation value obtained in step ST202 are selected from a plurality of documents belonging to the site indicated by. Merge the text information of those documents to create site text information. Next, using the site text information, the word W _i Statistic information on the appearance frequency of _ij And As the statistical information, for example, a TF-IDF value, χ ² Values can be used. In addition, any other method may be used as long as it represents the feature of the appearance tendency of a word.
In the first embodiment, a portion surrounded by a dotted frame in the drawing, that is, a site URL identification number S _j W of each word on the site _ij Is a site feature of the site.
[0018]
Next, the first site feature calculation unit 4 records the information acquired in steps ST202 to ST204 in the site management table 5 in association with the site URL (step ST205).
FIG. 5 is a diagram showing an example of a record in the site management table 5. The site management table 5 holds a site URL identification number for each site URL, and manages a site feature corresponding to the URL identification number. Also, document information such as the update date and time and title related to the site URL, and a representative URL corresponding to a document having a high reputation value calculated in step ST202 and the reputation thereof among a plurality of documents in the site URL are stored. . Further, it manages various attribute information related to the site URL.
[0019]
Next, in the site selecting unit 13, the site features corresponding to the site URL specified in advance by the system user are acquired from the site management table 5, and the designated site feature list is output (step ST206).
The designated site is a site URL that the user of the system wants to monitor, and can be designated by input means (not shown).
The site feature list is created for each user and stored in a table (not shown). Alternatively, it may be stored as a file.
[0020]
Next, the site information extraction processing will be described with reference to the flowchart of FIG.
In the following processing, a Web document is automatically acquired using a search condition set in advance by the system user, and the same or the same contents as the sites included in the designated site feature list created during the site registration processing are obtained from the Web documents. Extract information of similar sites. As a result, the system user can efficiently obtain, from a large number of search result documents, those that include a large amount of information that matches the purpose of the search.
[0021]
The user previously sets a search keyword related to the content to be investigated as a search condition (query) (step ST301). Here, the search condition is a condition in which a set of keywords relating to the content of a document to be collected is registered in advance as customer profile data in order for the client of the information clipping service to specify the content of information monitoring. For example, the requester can freely set a word related to the content of the survey to be monitored as a search keyword used for document collection. Key words include, for example, company names and personal names, product category names such as "TV", "mobile phone", and "car", specific product names, and other fields to be surveyed such as "brand", "stock price", and "management" Words about names etc. can be considered. Further, conditions relating to document information such as update date and time, language, domain name, etc. may be added to the search conditions and used as an auxiliary. The customer profile data is registered in, for example, a storage device (not shown) and supplied to the document search unit 6 when performing a search.
[0022]
Next, in the document search unit 6, a document search on the Web is periodically executed according to the search conditions set in the customer profile data (step ST302). As a result of the search, document information including the result document URL and the update date and time is obtained. The document information includes various information such as the title, URL, update date, and summary of each document output as a search result by the search engine used in the document search unit 6.
[0023]
Next, the URL management table 8 is referred to in the new URL extraction unit 7, and among the document URLs acquired in step ST302, the document URL that is not registered (new) or registered in the URL management table 8 is the document URL. The updated information is extracted as a new URL (step ST303).
The URL management table 8 stores document information including the document URL of the search result document and the update date and time. The new URL extracting unit 7 updates the document URL and document information of the new URL in the URL management table 8.
[0024]
Next, the download unit 9 accesses the new URL extracted in step ST303 and obtains the document by downloading the document (step ST304).
[0025]
Next, the second reputation calculating section 10 outputs the reputation extraction result for each document obtained in step ST304 (step ST305). The reputation calculation process in the second reputation calculation unit 10 is performed in the same manner as the process in the first reputation calculation unit 2 described later.
[0026]
Next, the second site extraction unit 11 analyzes the contents of each document acquired in step ST304 and extracts the site URL to which the document belongs (step ST306). The processing in the second site extraction unit 11 is the same as the processing in the first site extraction unit 3 in step ST203, and a description thereof will be omitted.
[0027]
Next, the second site feature calculation unit 12 comprehensively determines the contents of the documents belonging to the same site among the documents acquired in step ST304, and uses the determination result as a site feature (site feature A) for each site. (Step ST307). The site feature calculation process in the second site feature calculation unit 12 is the same as the process in the first site feature calculation unit 4 in step ST204, and a description thereof will be omitted.
[0028]
Next, the similar site extraction unit 14 acquires the site feature (site feature B) of the designated site from the designated site feature list output by the site selection unit 13 in step ST206 of FIG. Is calculated (step ST308). As the similarity, for example, the inner product value of the feature vectors of the site feature A and the site feature B can be used. In addition, any other information that indicates the similarity between the site features A and B can be used.
[0029]
If the calculated similarity is equal to or more than a certain value, the similarity between the site feature A and the site feature B is high, and the similar site extraction unit 14 outputs the site URL corresponding to the site feature A. In this way, the system user can acquire the site URL of the site that satisfies the input search condition and is specified in advance or a site similar thereto. The output document information of the site URL is registered and updated in the site management table 5 (step ST309).
[0030]
Next, the reputation calculation processing in the first reputation calculation unit 2 and the second reputation calculation unit 10 will be described with reference to FIGS. Since the configuration and the operation of the reputation calculation process of the first reputation calculation unit 2 and the second reputation calculation unit 10 are the same, only the first reputation calculation unit 2 will be described here.
[0031]
FIG. 6 is a block diagram illustrating a configuration of the first rumbling rating calculation unit 2. As shown in the figure, the first reputation calculating section 2 includes a reputation extracting section 15 and a reputation extracting rule table 16. The reputation extraction rule table 16 is stored in a storage unit inside the Internet information monitoring apparatus 100, or is stored in an external storage device or the like. The reputation extraction rule table 16 is a set of data records holding predetermined rules (reputation extraction rules) for extracting descriptions corresponding to reputation expressions from a document. The reputation extraction rule table 16 may be stored in the same storage device as the site management table 5 and the URL management table 8.
[0032]
FIG. 7 is a flowchart of the reputation calculating process in the first reputation calculating unit 2.
First, the document URL and the list of document information of the document obtained in step ST201 of FIG. 2 are input to the first rumbling calculator 2 (step ST601). The document information list includes information such as the text, title, URL, update date, and summary of each document. The document information list may be separately stored in a database. In this case, the relevant database name is input to the reputation extraction unit 15, and the reputation extraction unit 15 acquires document information by referring to the database.
[0033]
Next, reputation extraction section 15 determines whether or not the document information list acquired in step ST601 is empty (step ST602). If it is determined that the document information list is not empty, the process proceeds to step ST603, and if it is determined that the document information list is empty, the process proceeds to step ST611.
[0034]
If it is determined in step ST602 that the document information list is not empty, the reputation extraction unit 15 reads the text content of one document from the document information list (step ST603).
[0035]
Next, in step ST604, the reputation extraction unit 15 determines whether the description unit U can be obtained from the document. The description unit U may be a part of the text in the document or all the text. For example, the reputation extraction unit 15 cuts out a sentence unit from lexical information such as a period, a question mark, and a line feed, and sets it as a description unit U. Alternatively, the description unit U may be obtained by referring to the position information of a word in a document and extracting a description existing in a certain range of positions. Alternatively, a document having a structure such as an HTML document may be separated by tag information. In the case of an Internet bulletin board, the input document may be divided into individual articles and used as sub-documents. In this case, referring to the tag information, only the description including the text information is selected to be the description unit U. The description unit U cut out as described above is stored in the temporary buffer T.
[0036]
As a result of the above determination, if the description unit U can be acquired, the process proceeds to step ST605 to perform morphological analysis on the description unit U. If the description unit U cannot be obtained, the process returns to step ST602, and if an unprocessed document URL remains in the document information list, the process proceeds to step ST603 to read the document and repeat the process of step ST604.
[0037]
Subsequently, in step ST605, the reputation extraction unit 15 performs a morphological analysis on the description unit U stored in the temporary buffer T.
The morphological analysis performed in step ST605 is a method of analyzing linguistic information such as a headline and a part of speech of a word constituting a text, and is a well-known technique, and thus description thereof is omitted. As a result of the morphological analysis, information on the word heading and the part of speech of the description unit U is obtained.
[0038]
Next, the reputation extraction unit 15 collates the word contained in the description unit U and its part of speech information obtained in step ST605 with the contents of the reputation extraction rule table 16 (step ST606).
FIG. 8 is a diagram illustrating an example of the contents of the reputation extraction rule table 16. In the figure, the extraction intention is a classification of the intention of the writer of the document. An intention extraction expression, which is a set of word headings with part of speech information, is associated with the extraction intention. The weight is a value that is added to a score (intention score) given to the document when the intention extraction expression is collated and matched in the document, and is a value determined in advance according to the importance of each rumor extraction rule. .
For example, as shown in the figure, an expression including "correspondence (sa-modification) / evil (adjective)", specifically, "correspondence is poor" is extracted as information relating to some claim, and 1.0 is added to the score. You.
[0039]
The reputation extraction unit 15 determines whether or not the description unit U stored in the buffer T includes a word with part-of-speech information corresponding to the intention extraction expression (step ST607).
When it is determined that the intention extraction expression is included, the process proceeds to step ST608, and when it is determined that the intention extraction expression is not included, the process returns to step ST604.
[0040]
Next, the reputation extraction unit 15 extracts the intention extraction expression included in the description unit U as a characteristic expression. Further, an extraction intention corresponding to the intention extraction expression is acquired from the reputation extraction rule table 16 and is associated with the extracted characteristic expression (step ST608).
[0041]
Next, the reputation extraction unit 15 adds the feature expression acquired in step ST608 to the feature expression list (step ST609).
The feature expression list is a list having information as shown in FIG. 9, for example. As shown in the figure, a characteristic expression, a corresponding extraction intention (label), an extraction position of the characteristic expression in a document, and a weight corresponding to the extraction intention are stored.
[0042]
Next, the process proceeds to step ST610, where the weight of the rumor extraction rule that matches the characteristic expression is added to the intention score of the document. The intention score of a document is, for example, the sum of the weights of the feature expressions extracted in the document, and is managed for each extraction intention. Upon completion of the processing up to step ST610, the process returns to step ST604.
[0043]
When the process up to step ST610 is completed, the process returns to step ST604, and repeats the process until the description unit U cannot be obtained.
[0044]
If it is determined in step ST602 that the document information list is empty, the reputation extracting unit 15 sends the total value of the score of each document and the characteristic expression list to the first site extracting unit 3 together with the document content as the reputation extraction result. Output.
Similarly, the second reputation calculation unit 10 performs the processing of steps ST601 to ST611, and outputs the total score of the document downloaded by the download unit 9 and the feature expression list to the second site extraction unit 11. .
[0045]
As described above, according to the first embodiment, a large amount of Web documents are collected by the crawler unit 1 during the site registration process, and the first reputation calculation unit 2, the first site extraction unit 3, and the first The site characteristic calculation unit 4 acquires information representing the characteristics of the collected Web documents in units of sites, and records the acquired information in the site management table 5. Based on the information in the site management table 5 obtained as described above, the site selection unit 13 creates a feature list of the site that the system user wants to monitor and designates in advance, and outputs it as a designated site feature list.
At the time of site information extraction, the document search unit 6 searches for documents on the Web in accordance with the keyword specified by the system user, and a new URL is extracted by the new URL extraction unit 7 from the search results. get. The second reputation calculation unit 10, the second site extraction unit 11, and the second site feature calculation unit 12 acquire information representing features of the downloaded Web document in units of sites, and are created during the registration process. The same or similar sites are extracted by comparing with information indicating the features of the sites included in the specified site feature list. As a result, the system user can find and provide a new reputation site that matches the content of the client's investigation purpose from a large number of Web documents.
[0046]
The reputation extraction rule table 16 stores reputation extraction rules for a plurality of extraction intentions, and the reputation extraction unit 15 calculates a score of a feature expression included in the document for each extraction intention based on the reputation extraction rule for each document. Then, since it is output as a feature expression list, the user can know the contents of the reputation included in the document in more detail.
[0047]
Furthermore, according to the first embodiment, the first site feature calculation unit 4 and the second site feature calculation unit 12 calculate the feature of the appearance tendency of each word in the document belonging to the site URL to obtain vector information. Since the site feature of the site URL is adopted, there is an effect that a new site that matches the purpose of the search can be easily acquired based on the appearance tendency of the word in the document.
[0048]
Embodiment 2 FIG.
In the first embodiment, the site selection unit 13 outputs a specified site feature list of a site URL specified by a user of the system. In the second embodiment, a designated site feature list of a site URL that matches a condition is output by designating a site search condition without directly designating a site URL to the site selection unit.
[0049]
FIG. 10 is a block diagram illustrating a configuration of the site selection unit 17 according to the second embodiment. The site selection unit 17 is a component that is substituted for the site selection unit 13 in FIG. The components other than the site selection unit 17 are the same as those in FIG.
The site selection unit 17 includes a site search unit 18, an index storage unit 19, and a site feature acquisition unit 20. The index storage unit 19 is held in a storage unit inside the Internet information monitoring device 100 or is stored in an external storage device. The index storage unit 19 is a database that stores the correspondence between site URLs and keywords representing the contents of documents included in the site URLs in an indexed form. The index storage unit 19 may be stored in the same storage device as the site management table 5 and the URL management table 8.
[0050]
The operation of the site selection unit 17 will be described.
The site search unit 18 searches the index storage unit 19 based on a search keyword input by a system user using an input unit (not shown), and obtains a site URL where the search keyword matches a keyword in the index storage unit 19. I do.
The site feature acquiring unit 20 acquires a site feature corresponding to the site URL acquired by the site searching unit 18 from the site management table 5, and outputs a site feature list.
Note that the search method of the index storage unit 19 is not limited to the search using the keyword, but may be any method for searching the index. For example, the search condition to be input and the keyword stored in the index storage unit 19 are text, and a weight vector representing the characteristic of the tendency of appearance of the word included therein is created. A method of calculating the similarity between the search condition vector and the vector in the index storage unit 19 and outputting a site URL having a high similarity may be used.
[0051]
As described above, according to the second embodiment, the site selection unit 17 acquires the site URL that matches the search condition input by the site search unit 18 from the index storage unit 19, and acquires the site feature acquisition unit No. 20 creates a designated site feature list by extracting the site features of the acquired site URL from the site management table 5. As a result, the system user can efficiently obtain the specified site feature list of the site URL that matches the purpose of the survey without directly specifying the site.
[0052]
Embodiment 3 FIG.
In the third embodiment, information (attribute information) that further characterizes the content of a site is registered in the site management table, in addition to the site characteristics and the reputation. Here, as an example of such information, a specific product name or company name appearing in a document, or the presence or absence of a link to a bulletin board is registered.
[0053]
FIG. 11 is a block diagram showing a configuration of the Internet information monitoring apparatus 101 according to Embodiment 3 of the present invention. As shown in the figure, the Internet information monitoring apparatus 101 has an attribute information extracting unit 21. The attribute information extraction knowledge storage unit 22 is stored in a storage device outside the Internet information monitoring device 100. The attribute information extraction knowledge storage unit 22 may be stored in the same storage device as the site management table 5 and the URL management table 8. Other configurations are the same as those of the second embodiment.
[0054]
The attribute information extraction unit 21 includes a target name extraction unit 23 and a bulletin board presence / absence extraction unit 24 as shown in FIG. Further, the attribute information extraction knowledge storage unit 22 includes a target name table (reflection evaluation target table) 25 and a bulletin board extraction knowledge table 26. The target name table 25 stores a list of specific product names and company names. The bulletin board extraction knowledge table 26 stores a list of characteristic words representing bulletin boards. Note that the attribute information extraction unit 21 may include only one of the target name extraction unit 23 and the bulletin board presence / absence extraction unit 24. Also, if the attribute information extraction unit 21 has only the target name extraction unit 23, the attribute information extraction knowledge storage unit 22 only needs to have only the target name table 25. When only the bulletin board presence / absence extraction unit 24 is provided, it is sufficient to have only the bulletin board extraction knowledge table 26.
[0055]
Next, the operation of the attribute information extracting unit 21 will be described.
The target name extracting unit 23 is configured to select a company name or a product name that matches the product name or the company name registered in the target name table 25 from among the characteristic expressions of the respective documents extracted by the first reputation calculating unit 2. Is extracted.
[0056]
The target name table 25 stores a company name and a product name designated in advance by the system user, so that the system user can know the name of the product or the company that is the subject of the reputation. The target name extracting unit 23 outputs the obtained company name or product name as an attribute value of the attribute type “reputation target”.
[0057]
Next, the bulletin board presence / absence extracting unit 24 searches whether or not each document collected by the crawler unit 1 includes a word that matches a characteristic word representing a bulletin board registered in the bulletin board extracted knowledge table 26. In the bulletin board extraction knowledge table 26, words representing bulletin boards such as "bulletin board" and "BBS" and a site URL of a bulletin board designated in advance by a system user are registered.
When the document is described in a format language such as HTML, the bulletin board presence / absence extracting unit 24 reads the HTML document and analyzes whether or not there is an anchor tag indicating a hyperlink. If there is an anchor tag, it is analyzed whether or not the tag includes a characteristic word representing a bulletin board registered in the bulletin board extraction knowledge table 26. If the characteristic word is included, “related” is output as the attribute value of the attribute type “bulletin board”, and if the characteristic word is not included, the attribute value of the attribute type “bulletin board” is “ Not relevant "is output.
[0058]
With respect to the attribute value of the attribute type “reputation target” obtained by the attribute information extraction unit 21, the first site extraction unit 3 calculates the appearance frequency of each product name and company name for each site, and In the feature calculation unit 4, the product name and the company name and their appearance frequency are stored in the site management table 5 corresponding to the corresponding site URL.
[0059]
As for the obtained attribute value of the attribute type “bulletin board”, the first site extraction unit 3 outputs the presence or absence of a link to the bulletin board of each site, and the first site feature calculation unit 4 outputs the site information. The presence or absence of a link to the bulletin board is stored in the site management table 5 in correspondence with the URL.
[0060]
As described above, the attribute information extracting unit 21 is provided, and the attribute information of the site, such as the name of the product or company that is the subject of the reputation and the presence or absence of a link to the bulletin board, is registered in the site management table 5. Thus, when the site selection unit 17 creates the site feature list, there is an effect that such attribute information can be used as a site designation condition.
Note that the attribute information registered in the site management table 5 is not limited to the above-mentioned product name, company name, or presence / absence of association with the bulletin board, but may be any information that indicates the characteristics of the site. In this case, the attribute information extraction knowledge storage unit 22 stores a table used for the purpose of extracting other attribute information, in addition to the target name table 25 and the bulletin board extraction knowledge table 26.
[0061]
Embodiment 4 FIG.
In the fourth embodiment, in the site information extraction processing, information on the new site extracted by the similar site extraction unit 14 is provided as a report.
[0062]
FIG. 13 is a block diagram illustrating a configuration of the Internet information monitoring apparatus 102 according to the fourth embodiment. 1 denote the same components. As shown in the figure, the Internet information monitoring apparatus 102 has a report creation unit 27. Other configurations are the same as those of the first embodiment.
[0063]
The report creation processing by the report creation unit 27 will be described.
The report creation unit 27 creates and outputs a report based on the site URL output by the similar site extraction unit 14 and the document information on the site URL.
[0064]
The output report is created for each customer who has requested the survey. For example, as shown in FIG. 14, the search keyword registered by the client, the designated site set as the survey target, and the URL of the new site extracted this time are displayed. The summary is reported. Further, a document URL with a high reputation in the extracted new site may be presented together.
[0065]
As described above, since the report creation unit 27 is provided, the extraction result of the new site can be made into a report, so that the survey requester can continuously use the report to obtain information on the site that matches the purpose of the survey. Can be monitored.
[0066]
【The invention's effect】
As described above, according to the present invention, it is possible to obtain an information monitoring device that extracts a site that includes a large amount of information that matches the purpose of a survey.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of an Internet information monitoring device according to a first embodiment of the present invention.
FIG. 2 is a flowchart of a site registration process according to the first embodiment of the present invention.
FIG. 3 is a flowchart of a site information extracting process according to the first embodiment of the present invention.
FIG. 4 is a diagram for explaining a site feature vector according to the first embodiment of the present invention.
FIG. 5 is a diagram showing an example of a site management table according to the first embodiment of the present invention.
FIG. 6 is a block diagram showing a configuration of a first valuation calculating unit according to the first embodiment of the present invention.
FIG. 7 is a flowchart of a reputation calculating process in a first reputation calculating unit according to the first embodiment of the present invention.
FIG. 8 is a diagram showing an example of a reputation extraction rule table according to the first embodiment of the present invention.
FIG. 9 is a diagram showing an example of a feature expression list according to the first embodiment of the present invention.
FIG. 10 is a block diagram showing a configuration of a site selection unit according to Embodiment 2 of the present invention.
FIG. 11 is a block diagram showing a configuration of an Internet information monitoring device according to a third embodiment of the present invention.
FIG. 12 is a block diagram showing a configuration of an attribute information extraction unit and an attribute information extraction knowledge storage unit according to Embodiment 3 of the present invention.
FIG. 13 is a block diagram showing a configuration of an Internet information monitoring device according to a fourth embodiment of the present invention.
FIG. 14 is a diagram showing an example of the contents of a report according to Embodiment 4 of the present invention.
[Explanation of symbols]
Reference Signs List 1 crawler section, 2 first reputation calculation section, 3 first site extraction section, 4 first site feature calculation section, 5 site management table, 6 document search section, 7 new URL extraction section, 8 URL management table , 9 download unit, 10 second reputation calculation unit, 11 second site extraction unit, 12 second site feature calculation unit, 13 site selection unit, 14 similar site extraction unit, 15 reputation extraction unit, 16 reputation extraction Rule table, 17 site selection section, 18 site search section, 19 index storage section, 20 site feature acquisition section, 21 attribute information extraction section, 22 attribute information extraction knowledge storage section, 23 target name extraction section, 24 bulletin board presence / absence extraction section, 25 Subject name table (reputation target table), 26 Bulletin board extraction knowledge table, 27 Report creation unit, 100, 101, 102 Internet information monitoring device.

Claims (10)

A crawler unit that circulates and collects Web documents from documents on the Internet, and outputs the collected documents and document URLs;
A first reputation for extracting a preset reputation from each document collected by the crawler unit, calculating and outputting a reputation for each document based on an evaluation value corresponding to the extracted reputation; A calculating unit;
A first site extraction unit that extracts a site URL to which each document belongs from the document URL output by the crawler unit;
A first site feature calculation unit that outputs a site feature representing the content feature of the site specified by the site URL, stores the site URL in association with the site feature in a site management table,
A site selection unit for extracting a site feature B of a site URL specified by the system user from the site management table;
A document search unit for searching a document on the Internet based on the input search condition and outputting document information including a document URL and an update date and time as a search result;
With reference to a URL management table that stores document information for each document URL, a document URL that is not registered in the URL management table and a document URL whose document information is updated among the document URLs output by the document search unit. As a new URL, and registering the document information of the new URL in the URL management table;
A download unit for obtaining the document of the new URL from the Internet;
A second reputation for extracting a preset reputation from each document obtained by the download unit, calculating and outputting a reputation for each document based on an evaluation value corresponding to the extracted reputation; A calculating unit;
A second site extraction unit for extracting a new site URL to which each document belongs from the new URL;
A second site feature calculation unit that outputs a site feature A representing the content feature of the site specified by the new site URL;
A similar site extraction unit that calculates a similarity between the site feature A and the site feature B, outputs a new site URL having a similarity equal to or more than a certain value, and records document information of the new site URL in a site management table; Information monitoring device equipped with. The first reputation calculator and the second reputation calculator are:
With reference to a pre-registered extraction intention and a reputation extraction rule table that stores an intention extraction expression associated with the extraction intention, a word that matches the intention extraction expression is extracted from each document collected by the crawler unit. 2. The information monitoring apparatus according to claim 1, further comprising: a reputation extraction unit that totals scores preset for the intention extraction expression for each extraction intention and outputs the score as the reputation of each document. The first site feature calculation unit and the second site feature calculation unit quantify the appearance tendency of a plurality of words included in a document belonging to each site, and use the numbers as site features of the site. Or the information monitoring device according to claim 2. The site selection section
The keyword registered in the index storage unit that indexes and holds the correspondence between the site URL and the keyword representing the content of the document included in the site URL matches the site URL that matches the search condition keyword specified by the system user. A site search unit for extracting and outputting as a designated site URL;
4. A site feature acquiring unit for acquiring a site feature of the designated site URL from a site management table, and generating and outputting a designated site feature list. 9. The information monitoring device according to claim 1. The site selection section
The vector information stored in an index storage unit that indexes and holds the correspondence between the site URL and the vector information representing the characteristics of the appearance tendency of a plurality of words in the document included in the site URL is specified by the system user. A site search unit that extracts a site URL having a high degree of similarity with vector information indicating a tendency of appearance of a plurality of words in the search condition, and outputs the extracted site URL as a designated site URL;
4. A site feature acquiring unit for acquiring a site feature of the designated site URL from a site management table, and generating and outputting a designated site feature list. 9. The information monitoring device according to claim 1. An attribute information extraction unit that analyzes the content of the document collected by the crawler unit and extracts site attribute information characterizing the content of the site,
The information monitoring apparatus according to claim 1, wherein the first site feature calculation unit stores the site attribute information in a site management table. The attribute information extraction unit, from the documents collected by the crawler unit, the words in the table registered in the attribute information extraction knowledge storage unit composed of a plurality of tables that store words representing the attributes registered in advance 7. The information monitoring device according to claim 6, wherein the information is extracted. The attribute information extraction unit extracts, from the documents collected by the crawler unit, the product names and company names registered in the reputation target table storing the specific product names and company names,
The first site extraction unit outputs, for each site, the extracted product name or company name and their appearance frequency,
The first site feature calculation unit stores the product name and the company name and their appearance frequency in a site management table as information on a specific product or company to be reputed in a document. The information monitoring device according to claim 7, wherein The attribute information extracting unit extracts, from the documents collected by the crawler unit, the characteristic word representing the bulletin board registered in the bulletin board extraction knowledge table storing the characteristic word representing the bulletin board,
The first site extraction unit outputs, for each site, the presence or absence of the characteristic word representing the bulletin board,
8. The information monitoring system according to claim 7, wherein the first site feature calculation unit stores the presence / absence of the characteristic word representing the bulletin board in the site management table as information on the presence / absence of the site with the bulletin board. apparatus. 10. The system according to claim 1, further comprising a report creation unit that creates a report that presents a new site URL output by the similar site extraction unit and document information of the new site URL. Information monitoring device.

Images