WO2015012720A1 - Method for checking websites for the presence of real-time multimedia streams on said websites and computer-implementable system for performing said method - Google Patents

Abstract

The method for checking websites for the presence of real-time multimedia streams on said websites comprises downloading a website for checking the schedule, and analyzing the downloaded websites in respect of the presence of references to multimedia streams on said websites followed by storage of the references found in a database of streams, wherein information relating to the fact of a multimedia stream belonging to a website is stored in a database of interrelations. A check is then performed on the given streams for determining the type of streams, namely whether the multimedia stream is a real-time stream, and for identifying changes in the type of stream and/or the state of the stream and/or in the characteristics of the stream, with information relating to the changes subsequently being stored in the database of streams. Furthermore, a record of the fact of changes is made in the database of interrelations, new references to websites on which references to real-time multimedia streams have been identified are added to the schedule and/or the check period for references to websites for which changes in the stream have been identified which are already in the schedule is changed and/or the start time of a subsequent check is changed. The computer-implementable system comprises modules and databases which reproduce the algorithm of the method.

Description

 METHOD FOR checking WEB PAGES FOR AVAILABILITY IN THEM

 REAL-TIME MULTIMEDIA STREAMS AND COMPUTER-IMPLEMENTABLE SYSTEM FOR IMPLEMENTATION

 METHOD

 1. The technical field to which the invention relates.

 The present invention relates to computer and network technologies, namely, to search engines on the Internet, the purpose of which is to download, analyze, save and index Web pages containing targeted content, which is, for example, a real-time multimedia stream or, so called "live" stream or "live" content.

 More specifically, the invention relates to a technology for rechecking web pages previously found by search robots on the Internet that host real-time multimedia streams. In this case, rechecking is carried out according to a schedule that determines the order (using the coefficient of significance for the page) and the period (frequency) of rechecking. All web pages containing real-time multimedia streams should be double-checked for a certain period in order to store up-to-date data in the search system in order to further provide the user with the ability to search.

 The invention can be applied both to search for specific media objects (on-line radio streams, streams from webcams, video streams, etc.), and to search for objects in the form of links to external sources of a certain type, signaling the presence of target audio and / or real-time video content, for example, data transfer schemes - rtmp, rtsp, mms, etc.

 2. The prior art

 Currently, search engines are widely known and massively used in the world, which provide users with the ability to search the Internet for web pages containing the information they need for the search queries they enter. Popular search engines are, in particular, Yahoo!, Google, Yandex, Rambler.

The general principle of operation of well-known search engines is based on the collection of information on web pages on the Internet, its processing and indexing to ^• further provide the user with the opportunity to search for the necessary information to the extent that has been processed by the search engine. Part Each search engine includes search robots, the purpose of which is to scan web pages on the Internet and load them. After the search robot contacts the specified web page address, it scans, for example, http-headers, checking when the page was last modified. If the search robot has already viewed this web page, and the date of the last modification of the page has changed, then it will load it again for processing, if the web page it browses has not been viewed at all, then it will be immediately loaded for processing.

 Web pages loaded by the search robot are processed by the corresponding software and hardware components of the search engine. The purpose of this processing is to analyze the page: as a rule, the title is first extracted from the web page, since it carries general information about the web page. Next, all the text is extracted and processed, which is somehow highlighted, for example, in italics, underline or size (in particular, its font size is larger than the font size of the main text), since the search engine assumes that these are key places in the text and they are emphasized .

 Some search engines look at the meta tags of web pages, suggesting that they contain keywords or phrases on the page. At the same time, since the content of meta tags is often given inaccurate information, some search engines do not use them to determine the keywords of the page.

 Also, the entire text of the web page is fully processed. For example, those search engines that do not use meta tags to determine the keywords of a web page search for keywords by checking for how often a particular word appears in the text, and for this all “stop words” are removed from the text such as <a>, <he>, <you>, <b>, as well as all the characters and numbers, as they create noise when searching for keywords.

 Finally, the processed text of the web page is properly indexed by the search engine so as to provide the user with a web interface (for example, a browser) the ability to conveniently search the search engine database based on the input of search queries.

The structure and basic principles of the functioning of search engines are described in more detail, in particular, in "The anatomy of a large-scale hypertextual Web search engine ", Brin, S., Page, L., Computer Networks and ISDN Systems, 30 (1 -7): 107-1 17, 1998;" Effective Web Crawling ", Castillo, C, PhD thesis, University of Chile, 2004; "Crawling the Web." Web Dynamics: Adapting to Change in Content, Size, Topology and Use, ed. by M. Levene, A. Poulovassilis, 153-178, Pant, G., Srinivasan, P. , Menczer, F., 2004.

 However, search engines constructed in this way cease to meet today's requirements due to the ever-increasing volume and variety of information presented on the Internet. In particular, when conducting such a text-based search of web pages that contain pictures or videos of interest to the user, an extensive resulting list of web pages is produced in which the proportion of pages that truly meet the requirements and requirements of the user is small, since this list , due to the specifics of the search engines described, those web pages that contain mentions, discussions, advertisements, reviews about pictures or videos, but do not directly contain the search pictures or and video. As the aggregate volume and variety of web content on the Internet grows, the proportion of relevant web pages for such specific searches will only decrease, and as a result, users will be forced to build repeated search queries and spend time sifting through large arrays of search results.

 This problem determines the relevance of creating specialized (so-called, vertical) search engines that are strictly focused on searching on thematic Internet resources, including a system for re-checking found objects according to a special schedule.

In particular, technical solutions are known from the prior art that provide scheduling of re-checking web pages in order to store and index relevant data posted on the Internet (patent for invention US7886042B2). This solution relates to search engines capable of loading web pages, their further storage in the database and indexing in order to provide the end user with the ability to search the generated database. Re-checking web pages keeps the current data in the index. Since the amount of data on the Internet is huge, the task of re-checking requires search for more optimal solutions in order to save resources and identify landing pages for double-checking.

 Closest to the claimed technical solution is a method and system for re-checking scheduled web documents presented in the patent for invention US8386459В 1 - “Scheduling a recrawb. The invention provides a method for constructing a re-check schedule for web documents based on information about the document being checked. According to the present invention, for constructing a web document rechecking schedule, the frequency of change of the web document itself is determined based on the history of its previous check, which allows one to determine the time interval within which the web document has changed, and based on this information, calculate the optimal time interval for double-checking it.

 However, the known solution is based on double-checking all possible web documents on the Internet. This decision does not take into account the possibility of excluding non-target data from verification. Also, such a solution requires storing the history of a previous check of web documents, which is a highly costly solution, given the amount of data available on the Internet. Moreover, in the well-known technical solution, there is no possibility of making a decision on rechecking operatively, since the construction of a rechecking schedule is based on the history of the previous page check.

 3. Disclosure of invention

 The objective of the present invention is to provide a method and system focused on identifying web pages with link (s) to the real-time multimedia stream based on the results of checking an array of web pages according to a specific schedule.

The technical result of the invention is to increase the efficiency (including performance) of detecting multimedia streams in real time, the links to which are contained in the checked web pages, as well as reducing the number of resource-intensive operations (optimization of the used computing and network resources) used in checking the web pages that do not contain such content, as well as reducing the time to save and maintain data up to date, while increasing the reliability of the results. Thus, the inventive method can be implemented with significantly lower costs, including temporary (in comparison with well-known counterparts) required to search for web pages with target content while increasing the relevance of the detected web pages to the presence of the target content.

 When using the proposed method of double-checking web pages, the results produced by the search program, practically do not contain information noise. Thus, the results obtained correspond to the search criteria of the real-time multimedia content set by the user and contain only reliable content, so the user spends less time filtering the search results.

 The problem is solved in that the method of checking web pages for the presence of multimedia streams in real time, according to the invention, is as follows:

 • load web pages (for example, in the data loading module) for checking according to a schedule that determines the period and order (sequence) of checking links to web pages,

• analyze downloaded web pages for the presence of links to multimedia streams in them and then save the found links to multimedia streams in the stream database, while information about the membership of the multimedia stream to the web page is stored in the relationship database (or, in other words, maintain the link relationship to web pages and their associated links to multimedia streams), · when identifying links to multimedia streams that are not in the stream database, add to the stream database, and then check the data streams to determine their type, whether the multimedia stream is a real-time stream or not, and also periodically check links to real-time streams located in the stream database to detect changes in the type of stream and / or state of the stream and / or in the characteristics flow, with the subsequent storage of information about changes in the database flows, while in the database of relationships make a note about the fact of changes, • at the same time, new links to web pages on which links to real-time multimedia streams are identified are added to the schedule, and / or the check period for existing links to web pages for which changes in the stream are detected are changed and / or are changed the start time of the next check, while changing the characteristics of the stream, the start time of the next check of the corresponding web page is changed to the current one, if changes in the type and / or status of the stream are detected due to an increase in the number of links on the web page lock on real-time multimedia streams, the verification period is reduced, and if the number of links to real-time multimedia streams on a web page decreases, the verification period is increased, and in the case when the number of links to real-time multimedia streams becomes zero, this link to the web - The page is excluded from the scan schedule.

 In the claimed invention, a web page is a file directly containing the text of the web page and / or a script file associated with this web page.

 Downloading a web page by reference can be done by emulating the operation of a web browser by building a model of a web document and creating all objects that potentially contain links to multimedia streams.

 The period of checking links to streams from the database of streams having the status of a multimedia stream of real time to detect changes in the type of stream, and / or the state of the stream, and / or in the characteristics of the stream, can be selected from the interval of values 2-5 minutes.

 The state of the stream is determined based on whether it is on or off.

 The relationship database has a structure showing that a link to a web page belongs to one or more links to multimedia streams, while in the relationship database, streams that are of the type of real-time multimedia streams are noted.

As characteristics of the stream, a description of the multimedia stream and technical characteristics of the stream can be used. As a description of a multimedia stream, use the text description of the stream, the title of the stream, an indication of the owner of the stream, a link to the site of the stream, or any other data transmitted within the stream and displaying its essence. As the technical characteristics of the stream, bitrate, format, information about audio or video codecs or any other technical characteristics of the stream are used.

 Based on the analysis of the loaded web pages, the availability of web pages is additionally determined, and if inaccessible web pages are detected, an appropriate check mark is made in the check schedule. If an unavailable web page is in this state for a week, the link to this web page is excluded from the scan schedule.

где T_mj_n - минимально допустимый период проверки, например, 1 час, с последующим сохранением полученных результатов для (К) и (Т) в расписании проверки. При этом загрузку веб-страниц и их анализ осуществляют начиная со ссылки на веб-страницу из расписания, имеющей наибольшее значение коэффициента (К). In one embodiment, for each link to a web page, a significance coefficient (K) can be determined, which can be chosen equal to the number of links to real-time multimedia streams on this page, and the verification period can be calculated based on the relation T = T _con st / K, where T _con st is the specified check period, for example, 24 hours, and if T <T _m j _n , then

where T _m j _n is the minimum validation period, for example, 1 hour, followed by storing the results for (K) and (T) in the scan schedule. In this case, the loading of web pages and their analysis is carried out starting with a link to a web page from the schedule that has the highest coefficient (K).

 The problem is also solved by the fact that a computer-implemented system for checking web pages for the presence of multimedia streams in real time includes:

 Schedule database, including a list of links to web pages with a period for each link and the procedure for checking it,

 Stream database, including a list of links to multimedia streams, as well as information about the type, status and characteristics of streams,

 Relationship database storing information about the affiliation of the multimedia stream to the corresponding web page, as well as the type of multimedia stream,

 Data loading module, configured to download web pages via a link from the schedule database and analyze downloaded web pages for links to multimedia streams in them,

A data management module configured to save multimedia streams of links found by the data loading module to the database streams, as well as saving and / or changing information about the relationship between the multimedia stream and the web page in the relationship database,

 A module for checking streams from a database of streams, configured to determine the type of a multimedia stream, as well as periodically check references to multimedia streams in real time to detect changes in the type of stream and / or in the state and / or characteristics of the stream, followed by storing the received information in the stream database

 A flow control module, configured to detect changes made to the flows database, followed by recording information about changes in the relationships database,

 Schedule management module, configured to change the schedule in the schedule database by adding new links to web pages into it, on which links to real-time multimedia streams are revealed, and / or by changing the check period for existing links to web pages in the schedule, for which there are changes in the stream, and / or changes in the start time of the next check, while changing the characteristics of the stream, the start time of the next check of the corresponding web page is changed to The present, while maintaining the verification period; when detecting changes in the type and / or status of the stream associated with an increase in the number of links to real-time multimedia streams on a web page, the verification period is reduced, and when the number of links to real-time multimedia streams on a web page is reduced, the verification period is increased, and in the case when the number of links to real-time multimedia streams becomes equal to zero, this link to a web page is excluded from the scan schedule.

The data loading module is also configured to emulate the operation of a web browser by building a model of a web document and creating all objects potentially containing links to multimedia streams. Additionally, the data loading module is configured to determine the availability of a web page, and in case of unavailable web pages, information about this is recorded in the schedule database. If an unavailable web page is in this state for a week, the schedule management module is implemented with the ability to exclude links to this web page from the schedule database. The module for checking streams is also configured to change the period for checking links to streams from the database of streams having the status of a multimedia stream of real time from 2 to 5 minutes. The module for checking flows when checking the status of a real-time stream determines whether a given stream is on or off.

 In the stream database, as a stream characteristic, a description of the multimedia stream and technical characteristics of the stream are present; as a description of a multimedia stream, there is a text description of the stream, the title of the stream, an indication of the owner of the stream, a link to the site of the stream, or any other data transmitted within the stream and showing its essence; as the technical characteristics of the stream there is bitrate, format, information about audio or video codecs or any other technical characteristics of the stream.

The schedule management module is configured to determine a significance coefficient (K) for each link to a web page, which is chosen equal to the number of links on this page to real-time multimedia streams, and to calculate the verification period based on the relation T = T _CO nst /, where T _CO nst is the specified verification period, for example, 24 hours, and if T <T _m j _n , then T = T _m j _n , where T _m j _n is the minimum validation period, for example, 1 hour, s subsequent storage of the results for (K) and (T) in the schedule database. At the same time, the data loading module is configured to download web pages and analyze them starting from a link to a web page from the schedule database that has the highest coefficient K.

A distinctive feature of the claimed technical solution is that the criterion for rechecking the web page (or web document) is not the web page itself, but the connection of this web page with a link to the real-time multimedia stream (or information about the multimedia stream belonging to the web -page). The criterion for checking a web page is the presence in this page of a link to a multimedia server that broadcasts the multimedia stream in real time. Moreover, the claimed solution allows you to dynamically exclude or include from the check web pages that link to multimedia streams in case of data changes in the type, condition or characteristics of the stream, for example, if the multimedia stream Since real-time is disabled, then all web pages that link to this stream are excluded from re-checking. This decision is based not on the dynamics of changes in these web pages, but on an independent resource, such as a multimedia server, changes in which lead to a double-check of the web page that refers to this multimedia resource.

 4. Brief Description of the Drawings

 The invention is illustrated by drawings, where in FIG. 1 presents a block diagram of the inventive system for checking web pages for the presence of a multimedia stream of real time; in FIG. 2 is a flowchart showing a flowchart of calculating web page verification parameters; Fig. 3 shows a mapping of a link to a web page with links to multimedia streams; figure 4 presents an example of linking links to web pages and links to streams; figure 5 is an example of data communication after re-verification; figure 6 presents the algorithm for storing threads in the database of threads; figure 7 presents the display of changes in the structure of relations;

 With reference to FIG. 1 marked: 1 - a schematic representation of the movement of data between the modules of the system; 2 - data loading module, receiving links to web pages from the database of schedule 9; 3 - a data management module that creates and modifies associative links between links to web pages and links to multimedia streams; 4 - stream database, which contains links to multimedia streams and all information about them; 5 is a module for checking streams, which determines the type, status and characteristics of the multimedia stream; 6 is a database of relationships, which stores relevant associative links between links to web pages and links to multimedia streams; 7 is a flow control module that detects changes in the type, condition, or characteristics of flows in a stream database 4, followed by a record of changes in the relationship database 6; 8 - schedule management module, which makes changes to the schedule database 9 by adding new records to it or by modifying existing records; 9 - schedule database, which contains a list of links to web pages with a period for checking it for each link and the verification procedure.

5. The best embodiment of the invention The claimed invention allows to optimize the scan schedule of web pages that contain links to real-time multimedia streams by calculating the optimal scan period. Changing the validation period of a web page is based on the change:

 - type of multimedia stream - is the link to the multimedia stream a real-time stream; and / or

 - state of the multimedia stream - the stream is on or off; and / or

 - Characteristics or descriptions of the real-time multimedia stream. The basis for calculating the period of checking the web page are multimedia streams of real time, located on the web page.

 Search robots find web pages on the Internet that host multimedia streams. All web pages containing real-time multimedia streams should be re-checked for a certain period in order to store up-to-date data in the search engine related to the checked web page to further enable the user to search. To double-check the web pages, the schedule is used, which is located in the database of schedule 9, in which the coefficient of significance for the checked link to the web page is set, as well as the period of verification and the time of the end of the last check and the start of the next check. All links to web pages containing multimedia streams are stored in the system in the schedule database 9. The streams located on the page are checked to determine their membership in real-time multimedia streams. The purpose of re-checking web pages is to search on web pages for new links to multimedia streams and confirm the presence on the web page of links to multimedia streams found during the previous check of the web page, as well as updating the availability of the page and the information contained on it that displays it essence.

Data loading module 2 receives a list of links to web pages, which must be checked in accordance with the schedule from schedule database 9. The loaded pages of data loading module 2 are analyzed, which searches for multimedia links to streams. In addition to links from the page, information related to the multimedia stream is extracted in the form of a text description, which in the future used as part of the media stream description. The obtained information of the downloaded and analyzed web page and the links to the multimedia streams found in it is transmitted to the data management module 3. The data management module stores the found links to the multimedia streams in the stream database 4. For all the transmitted links to the multimedia streams, the data management module receives information from the database of streams 4 about the current type of multimedia stream to mark in the database of relationships 6. After receiving information for the found links to multimedia streams, The module data management 3 begins to check and make changes to the database linkages 6:

 • Checks the previous associative link of this link to the web page and links to multimedia streams in the database of relationships 6;

 • Changes the associative relations between the link to the web page and the links to the multimedia streams, based on the new associative links between the link to the web page and the links to the multimedia streams found on the web page to the relationship database 6;

 • Saves changes or, if there are no changes, confirms the previous associative relationship in the relationship database 6.

 All multimedia streams located in the database of streams 4 are checked by the streaming check module 5. All new streams are checked to identify real-time multimedia streams, as well as all multimedia streams that are defined as real-time streams and are in working (on) state, or were in the on state, but currently turned off. A regular check of multimedia streams in real-time status is carried out in order to store relevant information about streams, in the framework of which it is determined:

 • Type of multimedia stream, in order to confirm that the link to the stream belongs to real-time multimedia streams;

• The status of the multimedia stream (for example, the server transmitting the multimedia stream is on or off); • Specifications and a description of the multimedia stream, as well as track their changes.

 The flow control module 7 receives from the stream database 4 a list of real-time multimedia streams that have changes in type, and / or status, and / or technical characteristics, and / or description. Next, the flow control module 7 makes a note of the changes in the relationship database 6 for all links to web pages that have associative links with the resulting list of multimedia streams. It should be noted that with a single link to a multimedia stream can be associated with many links to web pages. Schedule management module 8 retrieves a list of links to web pages from the relationship database 6, for which there is a mark on changes in real-time multimedia streams. For the received links to web pages, the degree of changes in the flow is determined, the significance coefficient is calculated, and the verification period is also calculated. Based on the received data, the calculation of the start time of the next check is performed. Schedule management module 8 saves the results to the database of schedule 9 for the received list of links to web pages, changing the start time of the next check for links to web pages, and also adds or excludes links to web pages from re-checking. Links available for download on schedule 9 are sent for download and analysis to data loading module 2.

 Below is a more detailed description of the structure of the module for checking threads 6 and the algorithm of its operation.

 The algorithm for checking links to streams from the database of streams for the presence of multimedia streams in them in real time includes the following steps:

 - connection to the media server via the link,

 - receiving (downloading) from the media server information about the media stream, including the characteristics of the stream in a given format and / or part of the stream, intended for playback on the client side,

- analysis of the information received about the media stream, consisting in the search for signs indicating that the analyzed stream is a source of multimedia broadcasting which is carried out in real time, however, any sequence of characters and / or bytes in the media stream is used as signs, on the basis of which it is concluded that the media stream meets the criterion of a real-time multimedia stream (“live” stream).

 As information about the media stream, protocol headers can be additionally used.

 When a sequence of bytes is received from the server, their analysis is carried out in continuous mode until the data intended for direct reproduction is received, and upon receipt of a message with information that the stream is recorded, it is concluded that the stream being tested is not a multimedia source, broadcasting which is carried out in real time.

 As signs can be used:

 - a parameter characterizing the duration of the stream (Duration), and / or - a parameter characterizing the position at which playback starts in the stream (Start Time) and / or

 - parameter of the ability to rewind within the transmitted stream (Seekable).

 If the value of the parameter characterizing the duration of the stream (Duration) is negative or zero, or more than a specified limit, an analysis of the value of the parameter of the possibility of rewinding within the transmitted stream (Seekable) is carried out, if it indicates the prohibition of rewinding in the stream, conclude that the stream being analyzed is a multimedia source, the broadcast of which is carried out in real time.

If the value of the parameter characterizing the duration of the stream (Duration) is in the range from zero to the specified limit, reconnect to the media server and determine the values of this parameter and the parameter characterizing the position from which playback starts (Start Time), which compared with the values of similar parameters obtained during the initial connection, and if at least one of the parameter values does not match, it is concluded that the analyzed stream is the source multimedia broadcasting in real time; in case the parameter values coincide, they search for signs of the multimedia stream in the server response headers, upon detection of which they conclude that the stream being checked is a multimedia source broadcast in real time.

 The established limit of the values of the parameter characterizing the duration of the flow is selected experimentally and can be in the range of values from 5 to 9 hours.

 If the server has not received the values of the stream duration and / or playback position parameters, it is concluded that the stream being checked is a multimedia source broadcast in real time.

 The thread verification module for implementing the above algorithm contains:

 - a multimedia client, configured to connect to a media server via a link and download information about the media stream, including the characteristics of the stream in a given format and / or a specific part of the stream, intended for playback on the client side and / or information about the protocol headers received from the server ,

 - a unit for analyzing information about the media stream, which is configured to check the received information about the media stream, which consists in searching for signs indicating that the analyzed stream is a multimedia source broadcast in real time, where any sequence is used as the signs characters or bytes in the media stream, based on which they conclude that the media stream meets the criterion of "live" stream.

 As a multimedia client, such applications as MPlayer or VLC media player can be used, as well as any other product, including a self-developed multimedia client, configured to communicate, process and provide the necessary information.

Thus, the technology for determining the type of stream, whether it is a real-time stream or a static file of a fixed duration, consists in analyzing meta-information obtained from the media stream itself. The media client connects to the media server, after which receives from him meta-information about the stream in a given format, as well as a certain part of the stream intended for playback on the client side. The received meta-information, as well as the transmitted media stream buffer, pass the verification stage in order to determine the type of stream. The main purpose of the check is to analyze the data and search for signs that indicate that the analyzed stream is a multimedia source, the broadcast of which is carried out in real time. At the same time, a characteristic feature of a “live” stream (content) is the inability to perform “fast forward” with respect to it using the means of a client playback application.

 Typical examples of “live” AV content on the Internet are television (TV) and radio broadcasting on-air studios, special Internet broadcasting by professional and amateur studios, and images from a webcam for streaming broadcasting.

 The principle of building a schedule 9 is presented in more detail below.

To link to a web page, Schedule Management Module 8 determines the changes in the stream. If there are changes in the characteristics of the stream, which may include data such as a description of the stream, the title of the stream, an indication of the owner of the stream, a link to the site of the stream, or any other data transmitted within the stream and showing its essence, as well as changes in the technical characteristics of the stream, such as bit rate, format, information about audio or video codecs, or any other technical characteristics of the stream, then for the checked link to the stream, the start time of the next check is set equal to the current th time. Setting the start time of the next link check to a web page equal to the current time causes the link to be checked immediately. If a link or a state that can take two values changes for a link to a real-time multimedia stream, the stream is turned on or the stream is turned off, then Schedule Control Module 8 changes the significance factor, period and time of the next check of the link to web pages. If the multimedia stream is on, then the significance coefficient of the link to the web page increases by one; if the stream was turned on and now it is off, the significance coefficient decreases by one. For example, for one link to a web page there may be more than one “working” stream, if there are three, then the coefficient of significance will accordingly be equal to three. In the particular case, the significance coefficient fully corresponds to the number of links to real-time multimedia streams located on a web page. The conditions for determining the rules for choosing a significance factor for checking links on a web page may not depend on the number of links to real-time multimedia streams located at the specified address of the web page and, as a result, can be determined on the basis of other conditions. If the checked link to the web page has, for example, two links to real-time multimedia streams, and both links to the streams have state changes, for example, the streams have stopped working (are off), in this case, the link check value is per web page will be zero, which will exclude the page from scanning. Based on the significance coefficient, the period for checking the link to the web page is calculated, and the sequence of loading the web pages in the loading module 2, for which the time has come to check, is established. The rule for calculating the validation period may vary, depending on the events that are the reason for the double-checking of the web page. In a particular case, for each link to a web page, a significance coefficient (K) is determined, which is chosen equal to the number of links to real-time multimedia streams on this web page, and the verification period is calculated based on the ratio T = Tconst / K, where Tconst is a predetermined verification period, for example, 24 hours, and if T <Tmin, then T = Tmin, where Tmin is the minimum validation period, for example, 1 hour. All changes are recorded in the database of schedule 9, from which later the data goes to the data loading module 2.

The proposed system can be implemented on one or more server computers, combined to jointly implement the prescribed functionality, while the above modules can be implemented in software and hardware components of these server computers, known to specialists and widely used in technology. In particular, the above databases can be implemented on one or more commonly known computer-readable media, for example, hard disk drives, RAID arrays, solid state memory, etc. The data download module can be connected and can interact with the Internet based on well-known wired and / or wireless network technologies and equipment, in particular, based on the protocol stack http / tcp / ip. To interact with database modules, the operator can use any known terminal equipment that supports the ability to execute commands of the database interaction language (for example, SQL). Such equipment may be, for example, a suitably configured personal / laptop / handheld computer.

 The following are specific examples of the schedule for rechecking links to web pages with real-time multimedia streams placed on them.

Example JV ^" 1:

 The first example shows the appearance of new links on a web page in the schedule database 9.

 The search engine found a new web page using the link l link, after analyzing which it was revealed that the page has two links to the multimedia streams Stream l and Stream_2. Data management module 3, the information about the web page is transmitted, which contains various meta-information about the page itself and the detected links to multimedia streams (see Fig. 3).

The data management module sends links to the stream to the database of streams 4, where along with it it requests status for the transferred streams. If this link to the multimedia stream has already been transferred to the database of streams 4, then the data management module will receive information about it, if the link to the stream is new, then the stream information will remain unknown until it is verified by the streaming check module 5. Next data management module 3 checks the information about the link Reference l in the relationship database 6 to determine the associative relationship of this link to a web page with links to multimedia streams obtained from a previous check of this page. If this link to a web page is not in the database of relationships (that is, new), then it will be added to it, where the link of this link to the web page and links to multimedia streams will be indicated (see Fig. 3). Information about this link to a web page will remain unchanged in the database of relationships until links to the multimedia streams that were found on it are checked. Database changes interrelations 6 and further work with this link to the web page will be carried out only if after checking the links to the streams it will be determined that at least one of them refers to links to multimedia streams in real time. The module for checking streams 5 takes references to the verification from the database of streams 4, after which it determines that one of the links refers to multimedia streams in real time (see Table N ° 1).

 Table N ° l

The thread control module 7 requests information from the thread database 4 about real-time streams that have changes in type, state or characteristics. After the request, the flow control module 7 will receive a link to the stream Stream l, which will indicate that the stream has switched to the type of streams that belong to live streams (see table N ° 2).

Talits N ° 2

The flow control module 7 makes a mark in the database of relationships 6 for all links to web pages that have a link to this link to a multimedia stream that this stream has switched to the status of a "live" stream. Next, the schedule management module 8 takes from the relationship database 6 all links to web pages that have changes in the type, state or characteristics of links to real-time streams. Schedule management module 8 will receive a Reference l link for which the number of real-time streams and a mark on changes in the streams will be indicated (see table H).

Поскольку для данной ссылки на веб-страницу приходится только одна ссылка на поток реального времени, ей будет установлен коэффициент значимости равный единице, а также период проверки страницы, например равный 24 часам, после чего данные будут сохранены в базе данных расписаний 9 (см. таблицу N°4). Поскольку ссылка Reference l ещё не участвовала в расписании повторной проверки, то для неё будет создана отдельная запись, в которой будет указано время после которого она будет перепроверена. Время следующей проверки для новой ссылки на веб-страницу определяется как текущее время плюс период. Ν ° 3 table

Since for this link to a web page there is only one link to a real-time stream, it will be set to a significance factor of one, as well as a page verification period, for example, equal to 24 hours, after which the data will be stored in the schedule database 9 (see table N ° 4). Since the Reference l link has not yet participated in the rechecking schedule, a separate record will be created for it, in which the time after which it will be re-checked will be indicated. The next check time for a new link to a web page is defined as the current time plus period.

 Table N ° 4

 JVs example 2

 Consider the situation of re-checking web pages that are already on the schedule. Data loading module 2 receives from the database of schedule 9 (see table N ° 5) three links to web pages for which it is time to start the scan.

 Table N ° 5

In the relationship database 6, until verification, links to web pages have the following relationships with multimedia links to real-time streams (see FIG. N ° 4).

The reference_5 web page contains two links to real-time streams, while Reference_6 has a common stream with it, and the web page link Reference_7 contains an independent link to a stream that has no intersections with other web pages. Data Download Module 2 downloads web pages from the specified links. After analyzing the content, links to streams that have the connection shown in FIG. 5 were found in the downloaded documents.

 The found data is transmitted to the data management module 3. Next, the data management module sends the found links to the streams to the database of streams 4 in order to save new links to the streams and obtain information for already known streams (see Fig. 6).

 The received information about streams from the stream database 4 (see table N ° 6) indicates that the links to the stream Stream_10 and Stream_50 are checked and are real-time streams, the link Stream ll pointed to the real-time stream, is in the off state, a Stream_51 is a new link to a multimedia stream and requires verification by the stream verification module 5.

 Table N ° 6

Next, the data management module starts checking the previous association for the data of links to web pages in the relationship database 6. Based on the check, it is determined that some web pages have changed links to multimedia streams, which leads to changes in the associations in the relationship database 6 for these links to web pages (see FIG. 7).

The data management module notes that the web page at Reference_7 no longer contains a link to the Stream_12 stream and assigns new links to it, indicating that it contains Stream_50 and Stream_51 streams. Along with the changes in the associative relations, information on streams, which was obtained from the stream database 4. It is noted that the status of the stream ll link has changed, where it is indicated that the stream is turned off, and the links to Stream lO and Stream_50 streams are operational and are real-time multimedia streams. Since Stream_51 is a new link to a stream, there is no information for it that can affect changes in the schedule 9. Schedule management module 8 requests links to web pages that have changes in real-time streams from the relationship database 6 (see table N ° 7)

 Table> G ° 7

For reference_5, a coefficient of significance equal to one will be calculated, since one of the two available streams has stopped working, for reference_6, the coefficient of significance will be zero, because it does not currently have real-time streams in the on state, for reference_7 the coefficient significance will remain equal to one. After calculating the significance coefficient and calculating the validation period for current links to web pages, the schedule management module makes changes to the schedule 9 database (see table N ° 8).

Table N ° 8

The changes concern only two records, for Reference_5 the significance coefficient decreases and, as a result, the period increases. Reference_6 is excluded from the scan because it does not currently have real-time streams working. There are currently no changes for Reference_7, since before the scheduled check this link to a web page had an associative connection with Stream_12, after checking it began to point to two streams, but the type of stream was known only for one link to the stream, the second link to the stream was new, therefore the significance coefficient for this link currently remains equal to 1. Data loading module 2 revealed a new link to the Stream_51 stream for which the type was not determined, after it was checked by the thread verification module 5, Roedel that the link points to a live stream. The flow control module 7 requests data from the database of streams 4 and receives information (see table N ° 9) that Stream_51 refers to multimedia links to real-time streams.

 Taliza Ν ° 9

Then, the flow control module makes changes to the relationship database 6, where for all links to web pages that have a link to Stream 51, a mark is made about the change in the type of stream. Module Schedule Management 8 again requests from the relationship database 6 information about links to web pages that have changes in the type, state or characteristics of real-time streams and receives data (see table ° 10) about the changes for Reference_7.

 Table N ° 10

For reference_7, the significance coefficient, the verification period and the time of the next verification start are calculated, after which changes are made to the database of schedule 9 (see table Ν ° 11).

 Table .N l 1

Since Reference_7 now has 2 links to real-time streams, accordingly, it increases the significance coefficient and the recheck period changes, which leads to a change for the next time that the link to the web page is checked.

Claims

CLAIM 1. A method of checking web pages for the presence of multimedia streams in real time, according to which  load web pages for checking according to a schedule that includes links to web pages and determines the period and procedure for checking links to web pages,  analyze downloaded web pages for the presence of links to multimedia streams in them, followed by saving the found links to multimedia streams in the stream database, while information about the membership of the multimedia stream to the web page is stored in the relationship database,  then check the data streams to determine their type, whether the multimedia stream is a real-time stream or not, and also periodically check links to real-time streams in the stream database to detect changes in the type of stream and / or stream status, and / or in the characteristics of the stream, with subsequent storage of information about changes in the database of flows, while in the database of relationships make a note about the fact of changes,  at the same time, new links to web pages on which links to real-time multimedia streams are detected are added to the schedule, and / or the check period for existing links to web pages for which changes in the stream are detected are changed and / or the time is changed the beginning of the next check, while changing the characteristics of the stream, the start time of the next check of the corresponding web page is changed to the current one, when changes in the type and / or status of the stream are detected due to an increase in the number of links on the web page to real-time multimedia streams, the verification period is reduced, and if the number of links to real-time multimedia streams on a web page decreases, the verification period is increased, and in the case when the number of links to real-time multimedia streams becomes zero, this link to the web - The page is excluded from the scan schedule. 2. The method according to claim 1, characterized in that the web page is a file directly containing the text of the web page, and / or a script file associated with this web page. 3. The method according to claim 1, characterized in that the download of the web page by reference is carried out by emulating the operation of the web browser by building a model of the web document and creating all objects that potentially contain links to multimedia streams.  4. The method according to claim 1, characterized in that the period of checking links to streams from the database of streams having the status of a multimedia stream of real time, to detect changes in the type of stream, and / or the state of the stream, and / or in the characteristics of the stream, is selected from an interval of values of 2-5 minutes.  5. The method according to claim 1, characterized in that the state of the stream is determined based on whether it is in the on state or off.  6. The method according to claim 1, characterized in that the relationship database has a structure that displays the link of a web page to one or more links to multimedia streams, while in the relationship database mark streams that are of the type of multimedia streams in real time.  7. The method according to claim 1, characterized in that the characteristics of the stream use the description of the multimedia stream and the technical characteristics of the stream.  8. The method according to claim 1, characterized in that as a description of the multimedia stream using a text description of the stream, the title of the stream, an indication of the owner of the stream, a link to the site of the stream, or any other data transmitted within the stream and displaying its essence.  9. The method according to claim 1, characterized in that as the technical characteristics of the stream using bitrate, format, information about audio or video codecs or any other technical characteristics of the stream.  10. The method according to claim 1, characterized in that according to the results of the analysis of the loaded web pages, the availability of the web pages is determined, and in the case of unavailable web pages, an appropriate check mark is made in the check schedule.  11. The method according to claim 10, characterized in that if the unavailable web page is in this state for a week, the link to this web page is excluded from the check schedule. 12. The method according to claim 1, characterized in that for each link to a web page determine the significance coefficient (K), which is chosen equal the number of links on this page to real-time multimedia streams, and the verification period is calculated based on the relation T = Tconst / K, where Tconst is the specified verification period, for example, 24 hours, and if T <T _m jn, then

where T _m in is the minimum validation period, for example, 1 hour, followed by storing the results for (K) and (T) in the scan schedule.  13. The method according to p. 12, characterized in that the loading of web pages and their analysis is carried out starting with a link to a web page from the schedule that has the highest coefficient K.  14. A computer-implemented system for checking web pages for the presence of real-time multimedia streams, including:  Schedule database, including a list of links to web pages with a period for each link and the procedure for checking it;  Database of streams, including a list of links to multimedia streams, as well as information about the type, status and characteristics of streams;  Relationship database, including information on the affiliation of the multimedia stream to the corresponding web page, as well as the type of multimedia stream;  Data loading module, configured to download web pages via a link from the schedule database and analyze downloaded web pages for links to multimedia streams in them;  A data management module, configured to save the links found to the multimedia streams found by the download module to the streaming database, as well as to save and / or change the information about the relationship between the multimedia stream and the web page in the interconnection database;  A module for checking streams from a database of streams, configured to determine the type of a multimedia stream, as well as periodically check references to multimedia streams in real time to detect changes in the type of stream and / or in the state and / or characteristics of the stream, followed by storing the received information in the stream database; A flow control module configured to detect changes made to the database of streams with subsequent recording of information about the presence of changes in the database of relationships; Schedule management module, configured to change the schedule in the schedule database by adding new links to web pages into it, on which links to real-time multimedia streams are revealed, and / or by changing the check period for existing links to web pages in the schedule, for which there are changes in the stream, and / or changes in the start time of the next check, while changing the characteristics of the stream, the start time of the next check of the corresponding web page is changed to The present while preserving the verification period; when detecting changes in the type and / or status of the stream associated with an increase in the number of links to real-time multimedia streams on a web page, the verification period is reduced, and when the number of links to real-time multimedia streams on a web page is reduced, the verification period is increased, and in the case when the number of links to real-time multimedia streams becomes equal to zero, this link to a web page is excluded from the scan schedule.  15. The system of claim 14, characterized in that the data loading module is configured to emulate a web browser by building a model of a web document and creating all objects potentially containing links to multimedia streams.  16. The system of claim 14, characterized in that the module for checking streams is configured to change the period of checking links to streams from the database of streams having the status of a multimedia stream of real time from 2 to 5 minutes.  17. The system according to 14, characterized in that the flow check module, when checking the status of the real-time stream, determines whether a given stream is on or off.  18. The system according to 14, characterized in that in the stream database as a stream characteristics there is a description of the multimedia stream and technical characteristics of the stream. 19. The system according to 14, characterized in that in the stream database as a description of the multimedia stream there is a text description of the stream, the stream title, an indication of the owner of the stream, a link to the stream website, or any other data transmitted within the stream and displaying it essence. 20. The system of claim 14, characterized in that the stream database contains bitstream, format, information about audio or video codecs or any other technical characteristics of the stream as technical characteristics of the stream.  21. The system according to 14, characterized in that the data loading module is configured to determine the availability of a web page, and in the event that inaccessible web pages are detected, information about this is recorded in the schedule database.  22. The system according to item 21, characterized in that if the unavailable web page is in this state for a week, the schedule management module is configured to exclude links to this web page from the schedule database. 23. The system of claim 14, characterized in that the schedule management module is configured to determine a significance coefficient (K) for each link to a web page that is selected equal to the number of links on this page to real-time multimedia streams, and calculating a period checks based on the relation T = T _con st K, where const is the given verification period, for example, 24 hours, and if T <T _m i _n , then T = T _m in, where T _m in is the minimum allowable period checks, for example, 1 hour, followed by saving the received cut Ltats for (K) and (T) in the schedule database.  24. The system according to claim 23, characterized in that the data loading module is configured to download web pages and analyze them starting from a link to a web page from the schedule database having the highest coefficient K.