RU2829699C1 - Method and system for checking availability of information resource and content on it in computer networks - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to a method and a system for checking availability of an information resource and content on it in computer networks. Method comprises checking the possibility of establishing a connection with the final address of the resource in the network; sending an HTTP request to the address of the inspected information resource; checking the HTTP status returned by the checked information resource; decoding the text content of the HTTP response from the inspected information resource, followed by establishing the presence or absence of text signatures, which are a text sequence, from which it is possible to unambiguously determine that information on the selected resource is blocked or available; presence or absence of text signatures established at the previous step is used to determine accessibility to the placed content on the checked information resource.

EFFECT: providing simultaneous check of availability of an information resource and content placed on it.

6 cl, 13 dwg

Description

AREA OF TECHNOLOGY

[1] This technical solution generally relates to methods of checking and monitoring resources on the network, namely to a method of checking the availability of an information resource and the content posted on it in computer networks.

LEVEL OF TECHNOLOGY

[2] The availability of resources on the Internet is one of the most important indicators of their effectiveness, which affects their traffic. Accordingly, there is a need for the ability to one-time/constantly monitor the availability of resources from Internet provider access points.

[3] The prior art discloses a solution described in RU 2453916 "METHOD FOR SEARCHING FOR INFORMATION RESOURCES USING READDRESSES", Lebedev Igor Viktorovich (RU), published 20.06.2012: "The invention relates to methods for searching for information resources using redirections from the original addresses of information resources to the resulting addresses of information resources. The technical result is the expansion of the functional capabilities of access to resources in the network by organizing a redirection method. The essence of the claimed technical solution is that a client module is introduced into the client application and, upon determining the fact of lack of access to the requested information resource, the client application processes the initial address of the information resource, as a result of which, upon interpretation of the initial address of the information resource as an alias or address of an accessible information resource, the client application accesses the information resource by the resulting addresses corresponding to the initial address of the information resource, otherwise, if the initial address is not interpreted as an alias or address of an accessible information resource, then the client application generates and transmits a request containing the initial address of the information resource to the client module DB and the server of the information system, which processes the initial address of the information resource with the determination and/or search for the resulting addresses of the information resource corresponding to the initial address of the information resource."

[4] Known monitoring methods and systems do not allow, together with determining the availability of an information resource, to determine the availability of the content posted on it.

DISCLOSURE OF INVENTION

[5] This technical solution is aimed at eliminating the shortcomings inherent in existing solutions known from the state of the art.

[6] The technical problem solved in this technical solution is the complexity of unambiguous and simultaneous verification of the availability of an information resource and the content posted on it in computer networks.

[7] The main technical result that emerges from solving the above problem is ensuring accurate and simultaneous verification of the availability of an information resource and the content posted on it in computer networks.

[8] An additional technical result that appears when solving the above problem is the expansion of functional capabilities when checking the availability of an information resource and the content posted on it in computer networks.

[9] The specified technical results are achieved by implementing a method for checking the availability of an information resource and the content posted on it in computer networks, implemented using a processor and a data storage device, including the following steps:

• check the possibility of establishing a connection with the final address of the resource in the network;

• send an HTTP request to the address of the information resource being checked;

• check the HTTP status returned by the information resource being checked;

• decode the text content of the HTTP response from the information resource being checked, followed by determining the presence or absence of text signatures;

• based on the presence or absence of text signatures established in the previous step, the accessibility of the content posted on the information resource being checked is determined.

[10] In one of the specific examples of implementing the method, the HTTP request can be executed using one of the methods: GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, TRACE, PATCH.

[11] In another particular example of implementing the method, additionally, the route of the data packet is traced to the network node, with the determination of all intermediate nodes through which the packet passes to the final address of the resource in the network.

[12] In addition, the stated technical result is achieved through a system for checking the availability of an information resource and the content posted on it in computer networks, containing:

- at least one data processing device;

- at least one data storage device;

- at least one program, where one or more programs are stored on one or more data storage devices and executed on one or more data processing devices, wherein the one or more programs ensure the execution of the following steps:

• check the possibility of establishing a connection with the final address of the resource in the network;

• send an HTTP request to the address of the information resource being checked;

• check the HTTP status returned by the information resource being checked;

• decode the text content of the HTTP response from the information resource being checked, followed by determining the presence or absence of text signatures;

• based on the presence or absence of text signatures established in the previous step, the accessibility of the content posted on the information resource being checked is determined.

[13] In one of the specific examples of the system implementation, an HTTP request can be executed using one of the methods: GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, TRACE, PATCH.

[14] In another particular example of the implementation of the system, additional tracing of the data packet route to the network node is carried out, with the determination of all intermediate nodes through which the packet passes to the final address of the resource in the network.

BRIEF DESCRIPTION OF DRAWINGS

[15] The features and advantages of the present technical solution will become apparent from the following detailed description and the accompanying drawings, in which:

[16] Fig. 1 illustrates a block diagram of the implementation of the claimed method.

[17] Fig. 2 illustrates the resource verification algorithm using the TCP method.

[18] Fig. 3 illustrates an example of displaying in the system packets received by the TCP method from the resource being checked.

[19] Fig. 4 illustrates an example of packets received by the system using the TCP method.

[20] Fig. 5 illustrates the algorithm for checking a resource using the HTTP method (without checking the text content of the response).

[21] Fig. 6 illustrates an example of displaying in the system packets received via the HTTP method from the resource being checked.

[22] Fig. 7 illustrates an example of packets received by the system using the HTTP method.

[23] Fig. 8 illustrates the algorithm for checking a resource using the HTTP method (with checking the text content of the response).

[24] Fig. 9 illustrates an example of displaying received packets in the system using the HTTP method with verification of the text content of the response.

[25] Fig. 10 illustrates an example of packets received by the system using the HTTP method with verification of the text content of the response.

[26] Fig. 11 illustrates the contents of the received packets in a situation where the test bench is checking the target 10.14.0.91:80/ with a check for the presence of the text “Hello, world!”.

[27] Fig. 12 illustrates the contents of packets in the case where the test bench is testing the target 10.14.0.91:80/ with a check for the absence of the text "Hello, world!".

[28] Fig. 13 illustrates a system for implementing the claimed method.IMPLEMENTATION OF THE INVENTION

[29] Below we will describe the terms and concepts necessary for the implementation of this technical solution.

[30] Client - a technical device connected to the network of a communications operator providing services for providing access to the Internet information and telecommunications network, for the automated collection of data and determining the availability of information resources.

[31] Signature is a text sequence by which one can clearly determine whether information on a selected resource is blocked or, conversely, accessible.

[32] Layer L7 (OSI model) is the application layer of the basic hierarchical model of open systems interaction. Provides a user interface for interaction with network components.

[33] Layer L4 (OSI model) is the transport layer of the basic hierarchical model of open systems interaction. Provides data flow control and performs error monitoring.

[34] HTTP request - a message sent by a client to initiate a response from the resource being checked.

[35] HTTP response - a message sent by the server of the resource being checked to the client in response to its request.

[36] HTTP status - HTTP status code. Delivered by the server of the resource being checked as a response to any HTTP request

[37] An IP address is a unique address that identifies a resource on the Internet or a local network.

[38] SBC Single-Board Compute - single-board electronic computing device.

[39] TCP request - checks whether all data has been delivered to the recipient and checks whether there were any errors in transmitting the information.

[40] URL - the address of a resource on the Internet.

[41] The presented method for checking the availability of an information resource and the content posted on it in computer networks (Fig. 1 shows a diagram of the method) solves the problems of ensuring accurate and simultaneous checking of the availability of an information resource and the content posted on it in computer networks and expanding the functional capabilities when checking the availability of an information resource and the content posted on it in computer networks by sequentially performing the following steps:

• check the possibility of establishing a connection with the final address of the resource in the network (101 in Fig. 1);

• send an HTTP request to the address of the information resource being checked (102 in Fig. 1);

• check the HTTP status returned by the information resource being checked (103 in Fig. 1);

• decode the text content of the HTTP response from the information resource being checked, followed by determining the presence or absence of text signatures (104 in Fig. 1);

• based on the presence or absence of text signatures established in the previous step, the accessibility to the content posted on the information resource being checked is determined (105 in Fig. 1).

[42] The described method and system use several methods for checking the availability of resources on the Internet and local area networks, which represent a unique technology for automatic monitoring of social networks, instant messengers, portals and other data sources when receiving content.

[43] For one-time/constant monitoring of resource availability from Internet provider access points, a number of clients are used, implemented on the basis of single-board computers (SBC) located at various points of the network infrastructure, separated both geographically and logically. The client is a technical device connected either to the network of a communications operator providing services for access to the Internet information and telecommunications network (hereinafter referred to as the communications operator, provider), or to an internal private network (Intranet). The client ensures the automated collection of data to determine the availability of information resources.

The achievement of the main and additional technical results is ensured by the consistent application of the following solutions:

1) Checking the possibility of establishing a connection with the final address of the resource hosting provider on the Internet/Intranet (network host).

2) Tracing the route of a data packet to a network host, identifying all intermediate nodes through which the packet passes to the specified address.

3) Sending an HTTP request using any of the specified methods (GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, TRACE, PATCH).

4) Checking the HTTP status returned by the resource (according to RFC 7231).

5) Decoding the text content of the HTTP response from the resource, followed by establishing the presence or absence of text signatures specified by the system operator.

In this case, text signatures are understood as such text sequences by which it is possible to unambiguously determine whether the information on the selected resource is blocked or, on the contrary, accessible. In this case, the system assumes the implementation of the following types of requests.

1) Level L7 (OSI model)

1. HTTP request with GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, TRACE and PATCH methods and response code evaluation according to RFC 7231.

2. Checking the HTTP status returned by the resource.

3. Working with text signatures (hereinafter referred to as signature) for returned content:

a. the signature is specified as a substring;

b. The signature is specified as one or more substrings with the verification rules specified.

2) Level L4 (OSI model)

1. TCP requests with port specification (example tcp://8.8.8.8:80).

Thus, the client performs the following sequence of actions. If the address of the resource being checked is specified using a domain name, the IP address is determined using the operating system. In the event of receiving an HTTP redirect from the site being checked, the URL to which the redirection is performed is checked. This is done using the urllib3 library, which is a tool for working with HTTP requests and managing network connections in the Python programming language. Urllib3 provides a high-level interface for sending HTTP requests, processing responses, supporting proxy servers, and checking certificates. In the event of a successful HTTP request, the client checks the received response from the site being checked using the HTTP response code. The correctness of the response itself is also checked at the urllib3 library level.

The client performs a cyclical set of actions, namely:

1. requests a list of resources to be checked from the system’s server equipment;

2. processes the list of resources to be checked (targets) with the timeout for waiting for a response from the target specified in the configuration file, using the number of threads specified in the configuration file;

3. sends the survey results to the system server;

4. goes into standby mode with the interval specified in the configuration file;

5. proceeds to the implementation of point 1.

Below are more detailed descriptions of the methods used.

3) Level L3 (OSI model)

By specifying the MAC address of the receiving device and the loaded RAW sequence of both the sent request and the received response, it is possible to check the availability of the device at the network level.

[44] TCP targets are processed as follows: the client sends a request to establish a connection to the endpoint (specified by an IP address or domain name, as well as a port number). When sending packets, nodes sequentially number them and calculate a checksum. Since all packets have sequential numbers, it becomes clear if some of them are missing. In this case, a request is sent to resend the packet. If the checksum does not match for some packet, a request is sent to resend the packet. The TCP mechanism provides a data stream with preliminary connection establishment, performs a repeated request for data in the event of data loss, and eliminates duplication when receiving two copies of a packet, thereby guaranteeing the integrity of the transmitted data and notifying the sender of the results of the transmission.

If the connection is established successfully, the resource being checked is considered available.

If it is impossible to establish a connection (including due to various errors occurring during the attempt to establish a connection), the resource being checked is considered unavailable.

Fig. 2 shows the algorithm for checking a resource using the TCP method.

Fig. 3 shows an example of displaying in the system packets received by the TCP method from the resource being checked, which is a test stand with the IP address 10.14.0.91. An HTTP server based on Nginx with a start page on the standard port used for HTTP servers (port 80) is deployed on the test stand. The example illustrates the situation when the resource being checked is available.

An example of packets received by the system in the case where the above test bench is not available (the port used is 81) is shown in Fig.4.

[45] Processing of targets using the HTTP method (without checking the text content of the response) is carried out as follows: the client sends an HTTP request to the address specified by the system operator. Data transfer within the framework of the functioning of the protocol in question is carried out using HTTP messages. The structure of HTTP messages includes the following components:

• a status line, which is used to describe the version of the protocol being used and to indicate other data;

• HTTP headers describing the message body;

• an empty string, indicating that all available metadata for the response or request has been sent to the "recipient";

• optional message body. This component includes data associated with the request or data transmitted in the response.

If an HTTP response with status code 200 is received, the resource being checked is considered available.

In other cases (including due to various errors occurring during the process of sending an HTTP request and receiving/decoding an HTTP response), the target is considered unavailable.

Fig. 5 shows the algorithm for checking a resource using the HTTP method (without checking the text content of the response).

Fig. 6 shows an example of displaying in the system the packets received by the HTTP method from the resource being checked, which is a test stand with the IP address 10.14.0.91. An HTTP server based on Nginx with a start page on the standard port used for HTTP servers (port 80) is deployed on the test stand. The example illustrates the situation when the resource being checked is available (status code 200).

An example of packets received by the system via the HTTP method, in the case where the above test stand is unavailable (the port used is 80), is shown in Fig. 7.

[46] Processing targets using the HTTP method (with checking the text content of the response):

• The client sends an HTTP request to the address of the resource being checked (target) specified by the operator.

• If errors occur during the HTTP request/HTTP response decoding process, the target is considered unavailable;

• If an HTTP response with a status code other than 200 is received, the target is considered unavailable.

• If the HTTP response content cannot be decoded as text, the target is considered unreachable.

• If the decoded text meets the specified conditions, the target is considered accessible.

• In other cases, the target is considered unavailable.

Fig. 8 shows the algorithm for checking a resource using the HTTP method (with checking the text content of the response).

Fig. 9 shows an example of displaying received packets in the system using the HTTP method with verification of the text content of the response (the presence of the text "Welcome to nginx!") from the resource being checked, which is a test stand with the IP address 10.14.0.91. An HTTP server based on Nginx with a start page on the standard port used for HTTP servers (port 80) is deployed on the test stand. The example illustrates a situation when the resource being checked is available (status code 200, the specified text is present).

An example of packets received by the system using the HTTP method with verification of the text content of the response, in the case where the absence of the specified text “Welcome to nginx!” (port 80 used) is checked, is shown in Fig. 10.

The example in Fig. 11 illustrates the contents of the received packets in a situation where the test bench is checking the target 10.14.0.91:80/ with a check for the presence of the text “Hello, world!”.

The contents of the packets in the case where the test bench is testing the target 10.14.0.91:80/ with a check for the absence of the text “Hello, world!” is shown in Fig. 12.

The combination of methods used and developed verification algorithms allows for high-quality and reliable automated data collection to determine the availability of information resources and content on them.

[47] The claimed technical solution may be implemented, for example, by a system, a machine-readable medium, a server, etc. In this technical solution, a system means, among other things, a computer system, a computer (electronic computer), a CNC (numerical control), a PLC (programmable logic controller), computerized control systems and any other devices capable of performing a given, clearly defined sequence of operations (actions, instructions).

[48] A command processing unit is an electronic unit or integrated circuit (microprocessor) that executes machine instructions (programs).

[49] The command processing unit reads and executes machine instructions (programs) from one or more data storage devices, such as random access memory (RAM) and/or read-only memory (ROM). ROM may include, but is not limited to, hard disk drives (HDD), flash memory, solid-state drives (SSD), optical storage media (CD, DVD, BD, MD, etc.), etc.

[50] A program is a sequence of instructions intended for execution by a computer control device or a command processing device.

[51] The term "instructions" as used in this application may refer generally to software instructions or software commands that are written in a given programming language to perform a specific function, such as, for example, receiving and processing data, generating a user profile, receiving and transmitting signals, analyzing received data, identifying a user, etc. The instructions may be implemented in a variety of ways, including, for example, object-oriented methods. For example, the instructions may be implemented using the C++ programming language, Java, Python, various libraries (e.g., Microsoft Foundation Classes), etc. The instructions that perform the processes described in this solution may be transmitted via both wired and wireless data transmission channels, such as Wi-Fi, Bluetooth, USB, WLAN, LAN, etc.

[52] In general terms (see Fig. 13), the system for combining data from information systems for their subsequent visualization (1300) contains one or more processors (1301) connected by a common information exchange bus, memory means such as RAM (1302) and ROM (1303), and input/output interfaces (1304).

[53] The processor (1301) (or several processors, a multi-core processor, etc.) can be selected from a range of devices that are widely used at present, for example, from manufacturers such as: Intel™, AMD™, Apple™, Samsung Exynos™, MediaTEK™, Qualcomm Snapdragon™, etc. The processor or one of the processors used in the system (1300) must also include a graphics processor, for example, an NVIDIA GPU with a software model compatible with CUDA, or Graphcore, the type of which is also suitable for the full or partial implementation of the method, and can also be used for training and applying machine learning models in various information systems.

[54] RAM (1302) is a random access memory and is intended for storing machine-readable instructions executed by the processor (1301) for performing the necessary operations for logical data processing. RAM (1302), as a rule, contains executable instructions of the operating system and the corresponding software components (applications, software modules, etc.). In this case, the available memory capacity of the graphic card or graphic processor can act as RAM (1302).

[55] ROM (1303) is one or more permanent storage devices, such as a hard disk drive (HDD), a solid-state drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media (CD-R/RW, DVD-R/RW, BlueRay Disc, MD), etc.

[56] Various types of I/O interfaces (1304) are used to organize the operation of the device components (1300) and to organize the operation of external connected devices. The choice of appropriate interfaces depends on the specific design of the computing device, which may include, but are not limited to: PCI, AGP, PS/2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS/Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[57] To ensure user interaction with the device (1300), various I/O information means (1305) are used, for example, a keyboard, display (monitor), touch display, touchpad, joystick, mouse, light pen, stylus, touch panel, trackball, speakers, microphone, augmented reality means, optical sensors, tablet, light indicators, projector, camera, biometric identification means (retina scanner, fingerprint scanner, voice recognition module), etc.

[58] The network interaction means (1306) provides data transmission via an internal or external computer network, such as an Intranet, the Internet, a LAN, etc. One or more means (506) may be, but are not limited to: an Ethernet card, a GSM modem, a GPRS modem, an LTE modem, a 5G modem, a satellite communication module, an NFC module, a Bluetooth and/or BLE module, a Wi-Fi module, etc.

[59] The specific choice of device elements (1300) for the implementation of various software and hardware architectural solutions may vary while maintaining the required functionality. In particular, such an implementation may be performed using electronic components used to create digital integrated circuits. I am not limited to, but may use microcircuits whose operating logic is determined during manufacture, or programmable logic integrated circuits (FPGAs), whose operating logic is specified by programming. For programming, programmers and debugging environments are used that allow you to specify the desired structure of a digital device in the form of a basic electrical circuit or a program in special hardware description languages: Verilog, VHDL, AHDL, etc. An alternative to FPGAs are: programmable logic controllers (PLCs), basic matrix crystals (BMCs), which require a factory production process for programming; ASICs - specialized custom large-scale integrated circuits (LSI), which are significantly more expensive for small-scale and individual production. Thus, the implementation can be achieved by standard means based on classical principles of implementing the basics of computing technology.

[60] In a particular embodiment, the described solution is a hardware and software complex for monitoring the availability of resources on the Internet, where the hardware consists of two servers: a server with a virtual machine with a VPN server, a database server, and a number of clients implemented on the basis of single-board computers (SBC (Single-Board Computer)), which can be located at various sites.

[61] The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be interpreted as limiting other, particular examples of its implementation that do not go beyond the scope of the requested legal protection, which are obvious to specialists in the relevant field of technology.

Claims (19)

1. A method for checking the availability of an information resource and the content posted on it in computer networks, which includes the following steps: • check the possibility of establishing a connection with the final address of the resource in the network; • send an HTTP request to the address of the information resource being checked; • check the HTTP status returned by the information resource being checked; • decode the text content of the HTTP response from the information resource being checked, followed by establishing the presence or absence of text signatures, which are a text sequence by which it can be unambiguously determined that the information on the selected resource is blocked or, conversely, accessible; • based on the presence or absence of text signatures established in the previous step, the accessibility of the content posted on the information resource being checked is determined. 2. A method for checking the availability of an information resource and the content posted on it in computer networks according to paragraph 1, characterized in that the HTTP request can be executed using one of the methods: GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, TRACE, PATCH. 3. A method for checking the availability of an information resource and the content placed on it in computer networks according to paragraph 1, characterized in that the route of the data packet to the network node is additionally traced, with the determination of all intermediate nodes through which the packet passes to the final address of the resource in the network. 4. A system for checking the availability of an information resource and the content posted on it in computer networks, containing: at least one data processing device; at least one data storage device; at least one program, where one or more programs are stored on one or more data storage devices and executed on one or more data processing devices, wherein the one or more programs ensure the execution of the following steps: • check the possibility of establishing a connection with the final address of the resource in the network; • send an HTTP request to the address of the information resource being checked; • check the HTTP status returned by the information resource being checked; • decode the text content of the HTTP response from the information resource being checked, followed by establishing the presence or absence of text signatures, which are a text sequence by which it can be unambiguously determined that the information on the selected resource is blocked or, conversely, accessible; • based on the presence or absence of text signatures established in the previous step, the accessibility of the content posted on the information resource being checked is determined. 5. A system for checking the availability of an information resource and the content posted on it in computer networks according to paragraph 4, characterized in that the HTTP request can be executed using one of the methods: GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, TRACE, PATCH. 6. A system for checking the availability of an information resource and the content placed on it in computer networks according to paragraph 4, characterized in that it additionally carries out tracing of the route of a data packet to a network node with the determination of all intermediate nodes through which the packet passes to the final address of the resource in the network.