WO2021118390A1 - Method and system for identifying images from video stream - Google Patents

Abstract

The present technical solution relates to methods and systems for identifying images in vending machines and self-serve kiosks. A method for identifying images from a video stream, which can be implemented by at least one computation device and in which at least one frame is obtained from at least one camera, the at least one frame including at least one product from a kiosk, wherein each frame is divided by a virtual cash register line; the at least one frame with the product obtained in the previous step is transferred to the input of a trained artificial neural network, which identifies the at least one product in the frame by obtaining product coordinates and a class to which the given product belongs; the computation device determines whether or not the center of the product is located behind or beyond the virtual cash register line, wherein if the at least one product appears in the cash register zone and crosses the line, the given product is added to a virtual basket; and the content of the virtual basket formed in the previous step is sent to a server for carrying out payment settlement with the user for the at least one removed product.

Description

METHOD AND SYSTEM FOR RECOGNIZING IMAGES FROM VIDEO STREAM

FIELD OF TECHNOLOGY

[001] This technical solution generally relates to the field of computing, and in particular to methods and systems for pattern recognition in vending machines and self-service kiosks.

LEVEL OF TECHNOLOGY

[002] Currently, vending machines are popular because of their convenience, because they are ready to work at any time of the day or night. In addition, vending machines can be placed in remote locations such as train stations, hotel corridors, and the like, which are convenient for customers.

[003] Recognition of images in a video image, for example, in the same vending machines, requires significant computing power of the equipment used. The existing solutions in the prior art do not provide efficient and fast pattern recognition equipment with low computing power.

[004] Prior art patent application CN 110231837 (A) “A kind of multiple target directive property intelligent monitor system and its method emulating human eye vision”, (Applicant: Kunming University of Science and Technology, published: 2019-09 -13). The present invention relates to a type of intelligent surveillance systems with multiple target properties and to a method for imitating human eye vision related to the technical field of video monitoring. The solution contains a field of view image capturing module, a processing module and a drive module. The image capturing module includes a panoramic camera and a small field-of-view camera, the processing module includes an NVIDIA Jetson TX2 component, and the drive module includes a controlled rotation motor. The present invention has low algorithm complexity, high stability, fast response time, and advantages towards multiple targets. [005] However, this solution is not adapted to solve problems with image recognition from a video stream, when it is necessary to track the trajectory of a product taken by a user from a kiosk.

ESSENCE OF THE TECHNICAL SOLUTION

[006] This technical solution is aimed at eliminating the disadvantages known from the prior art.

[007] The technical problem or technical problem solved in this technical solution is the implementation of the method and system for recognizing images from a video stream, as well as a device for their implementation.

[008] The technical result achieved when solving the above technical problem is to reduce transaction costs by simplifying the mechanism for determining the entity retrieved by the customer, subsequent interaction with the payment system for mutual settlements between the company and the user, and increasing the rate of feedback of the kiosk to the user.

[009] The specified technical result is achieved by implementing a method for recognizing images from a video stream, which is performed by at least one computing device and in which at least one frame is received from at least one camera containing at least one product from a kiosk, each the frame is separated by a virtual cashier line; at least one frame with the product obtained in the previous step is transmitted to the input of the trained artificial neural network, which identifies at least one product on the frame by obtaining the coordinates of the product and the class to which this product belongs; determining by means of a computing device whether the center of the goods is behind the virtual checkout line or outside, and if at least one item appears in the checkout area and crosses the line, this item is added to the virtual cart; sending the contents of the virtual basket, formed in the previous step, to the server for performing settlements with the user for at least one retrieved item. [0010] In some embodiments of the technical solution, each n-th frame with the product obtained at the previous step is transmitted to the input of the trained artificial neural network.

[0011] In some embodiments of the technical solution, on frames other than n-th, the item is tracked using a correlation tracker based on the frame difference.

[0012] In some embodiments of the technical solution, when recognizing a product from a video stream, the code of this product is sent to the server via the Internet via HTTPS, and the order of actions is received from the server using the entity code, which is laid down when recognizing this product.

[0013] In some embodiments of the technical solution, a trained neural artificial network recognizes an expiration date on a product.

[0014] In some embodiments of the technical solution, the trained neural artificial network recognizes the place of the first identification of the product and / or the frame of the last identification, and / or the direction of movement of the product, and / or the absolute coordinates of the product. [0015] Also, the technical result is achieved by implementing a system for recognizing images from a video stream, which contains a vending kiosk, in the body of which there are structurally installed: at least one camera installed on the kiosk, configured to obtain at least one frame containing at least one product from a kiosk, and each frame is separated by a virtual checkout line; at least one computing device containing a trained artificial neural network and configured to receive at least one frame with a product at the input of a trained artificial neural network, which identifies at least one product on the frame by obtaining the coordinates of the product and the class to which it belongs this product; determining whether the center of the item is behind the virtual checkout line or outside, and if at least one item appears in the checkout area and crosses the line, this item is added to the virtual cart; at least one door locking unit, configured to send to at least one server a signal to unlock the door lock; at least one information input unit configured to receive commands from the user; at least one system interaction unit 5 configured to transmit signals between system units. [0016] In some embodiments of the technical solution, the door locking unit is an electromagnetic lock, which consists of a metal plate and a magnetic sensor, and / or an electromechanical lock. [0017] In some embodiments, the computing device is an NVIDIA Jetson or GPU computing device.

[0018] In some embodiments of the technical solution, the system further comprises a temperature readout unit 15, which is a set of temperature sensors for generating data on temperature readings inside the kiosk.

[0019] In some embodiments of the technical solution, the information input unit is a contactless card reader. [0020] In some embodiments of the technical solution, the system interaction unit 20 is a Raspberry Pi.

BRIEF DESCRIPTION OF DRAWINGS

[0021] The features and advantages of the present technical solution will become apparent from the following detailed description and the 25 accompanying drawings, in which:

[0022] FIG. 1 shows an example of the implementation of the architecture of a system for recognizing images from a video stream.

[0023] FIG. 2 shows an example of realizing product recognition using a video camera and forming a virtual basket zo [0024] FIG. 3 shows an example of a kiosk implementation, which implements a system for recognizing images from a video stream.

[0025] FIG. 4 shows an example of the implementation of product recognition using a video camera and the formation of a virtual basket, where the product is tracked. DETAILED DESCRIPTION OF THE INVENTION

[0026] Below will be discussed in detail the terms and their definitions used in the description of this technical solution.

[0027] In this invention, the system means a computer system, a computer (electronic computer), CNC (numerical control), PLC (programmable logic controller), computerized control systems and any other devices capable of performing a given, well-defined sequence of operations. (actions, instructions), centralized and distributed databases, smart contracts.

[0028] By a command processor is meant an electronic unit or an integrated circuit (microprocessor) executing machine instructions (programs), or the like. A command processor reads and executes machine instructions (programs) from one or more storage devices. The role of data storage devices can be, but are not limited to, hard disks (HDD), flash memory, ROM (read only memory), solid state drives (SSD), optical drives.

[0029] A program is a sequence of instructions for execution by a computer control device or command processing device.

[0030] Server (English server) - an electronic device that performs service functions at the request of the client, providing him with access to certain resources. For the purposes of this description, a server is contemplated that has a persistent connection to the internetwork that can transmit data to a client device. The server can process this data and transmit the processing result back to the client device.

[0031] A data exchange module is a server module that can represent a receiver of incoming signals, and a converter for further processing, and a translator for further sending. [0032] Computing module is a server module that is a microprocessor specially adapted for complex signal processing.

[0033] A database storage module is a server module that stores data, which can be configured as a hard disk drive or flash memory, which is a semiconductor of electrically programmable memory.

[0034] An electronic lock is an electromechanical lock with a remote control system (RC) and a power supply. An electromechanical lock consists of three parts: a lock, a deadbolt and mounting brackets.

[0035] Video analytics is a technology that uses computer vision methods to automatically obtain various data based on the analysis of a sequence of images received from video cameras in real time or from archive records.

[0036] In this technical solution, the concepts of "vending machine", "self-service kiosk", "kiosk" are used, however, it is obvious to any person skilled in the art that any vending equipment can be used, for example, vending machines for selling drinks, vending machines food, vending machines for the sale of non-food products, amusement machines, attractions, etc.

[0037] The proposed device can be used to work with any commercially available vending machines, including those previously installed. At the same time, for the functioning of the proposed system, the presence in the machine of such nodes as a product selection panel, a display, a bill acceptor, a coin acceptor, a change dispenser, a bank card reader, and a receipt printer is insignificant. In fact, a vending machine or a kiosk in the proposed solution is an automatic point of delivery of goods, which can only contain means for storing goods and a mechanism for their issuance, controlled by a controller, which is connected through a telecommunication network with a server via a modem and payment converter - a device that is required, if the machine has blocks for accepting coins or bills, to confirm the payment and authorize the issuance of goods.

[0038] In some embodiments, the presence of a transparent display case, which occurs in a conventional kiosk design, is also unimportant, since the full range of goods can be transferred directly to the user's mobile device upon purchase. The absence of a display case in the vending machine contributes to the increase of its vandal resistance.

[0039] This technical solution works as follows. [0040] Having expressed a desire to buy a particular product from a particular kiosk, the user opens (or installs) the corresponding mobile application on the mobile device or immediately opens the door of the kiosk directly if he does not use the mobile application.

[0041] After the user has taken out the goods, this technical solution recognizes the movement of the goods on the video from the cameras, and thus counts the amount of the goods that the user bought. [0042] A sequence of actions refers to a set of programming language instructions chained together to call simple operations that result in an atomic operation (eg, buying, returning, removing an item from a kiosk, returning an item to a kiosk, etc.).

[0043] In some embodiments, at least one frame from the video stream is received, which is transmitted to the input of an artificial neural network (hereinafter ANN). The ANN identifies the goods on the frame and returns the coordinates of the goods (for example, in pixels) and the class to which this product belongs. By analyzing the sequence of changes in the coordinates of objects, the algorithm concludes which product was taken from the refrigerator. In some implementations, the class is a product name or a product brand, such as snickers, mars, caesar salad, and so on. In some implementations, the ANN recognizes the expiration date of the item. A label is attached to the product, which allows the INS to recognize the expiration date. When recognizing goods, an ANN is used, a system installed on a computing device that contains a set of all images of goods that are sold in a kiosk.

[0044] Parameters such as the place of the first identification of the object, the frame of the last identification, the direction of movement of the object, the absolute coordinates of the object, and the like can be analyzed. The place of the first identification of the object is determined as follows. The camera looks at the user who is going to buy groceries, the buyer puts his hand into the fridge of the kiosk, but the camera still does not see the product. When the product enters the field of vision of the camera, the ANN finds the product, which is the moment of the first identification of the product.

[0045] A video stream from a camera or a set of cameras can be processed on an embedded modular NVIDIA ™ Jetson ™ supercomputer, with every nth frame being processed by an ANN to reduce the computational load on the system. In a specific implementation, an ANN of the Yolo architecture (Convolutional Artificial Neural Network) can be used. Such ANNs provide a fairly high accuracy and speed of detecting objects in images. In some implementations, the machine learning libraries TensorFlow, Keras, Darknet, Deepstream, etc. can be used, but not limited to. Each module of the system, both software and hardware, can work on any modern computer with a graphics processor from Nvidia or on mobile platforms Jetson ™ TX2, Jetson Nano ™, etc. [0046] The camera (s) may (s) be provided as a digital camera configured to capture still images and moving images (eg, video). The camera may include a lens and one or more image sensors configured to capture and convert light into electrical signals. By way of example only, an image sensor may include a CMOS image sensor (eg, a CMOS active pixel (APS) sensor) or a CCD (charge coupled device) sensor. In general, an image sensor in a camera includes an integrated circuit having an array of pixels, in which each pixel includes a photo detector that collects light. It will be apparent to those skilled in the art that Photodetectors in imaging pixels generally record the intensity of light captured through a camera lens. However, photodetectors themselves are generally incapable of detecting the wavelength of captured light and thus incapable of detecting color information.

[0047] Accordingly, the image sensor may further include a color filter array (CFA) that may overlap or overlap the pixel array of the image sensor to capture color information. The array of color filters can include an array of small color filters, each of which can overlap a corresponding pixel of the image sensor and filter the captured light by wavelength. Thus, when used together, the color filter array and photodetectors can provide wavelength and intensity information with respect to the light captured by the camera, which can represent the captured image.

[0048] In one embodiment, the color filter array may include a Bayesian color filter array that provides a filter pattern that is 50% green, 25% red, and 25% blue. Thus, an image sensor that uses an array of Bayesian color filters can provide information regarding the intensity of light received by the camera at green, red, and blue wavelengths, so that each pixel in the image registers only one of the three colors (RGB). This information, which may be referred to as "primary image data" or data in the "primary region", may then be processed using one or more demosaicing techniques to transform the primary image data into a full color image, generally by interpolating a set of red, green, and blue for each pixel.

[0049] Cameras in some embodiments may be mounted on top of the kiosk, built into the kiosk itself, mounted at the bottom of the kiosk, and so on, but not limited to. Alternatively the implementation of the camera can be installed not on the kiosk itself, but above the kiosk or an assembly for the correct recognition of goods with the field of view of the camera or cameras. In this particular implementation example, processing means the ANN's prediction of the position of the product and the type of product in at least one frame. In order for the ANN to predict the position and type of goods on the frame, it must be pre-trained, which will be described below. The information processing method consists in continuous analysis of the video stream from a set of cameras installed inside or outside the kiosk. At least one camera must be installed on top of the kiosk so that the camera can see the entire shopping area. Additionally, it is necessary for the camera to slightly capture the shelf in order to see how the user picks up the goods.

[0050] Image processing can be understood as various stages of image processing, for example, operations for detecting / correcting defective pixels, lens shading correction, demosaicing and image sharpening, noise reduction, gamma correction, image enhancement, color space conversion, image compression, color subsampling and scaling the image, etc. In some embodiments, an image processing "pipeline" may be used to perform each of the various steps in image processing. A given pipeline can be composed of a set of components, according to each desired step, and the components can be implemented using hardware (eg, digital signal processors or ASICs) or software, or through a combination of hardware and software components.

[0051] Before proceeding, it should be noted that although various embodiments of the various image processing techniques discussed below may use Bayesian CFA, the invention described herein is not subject to limitation in this regard. Indeed, those skilled in the art will appreciate that the image processing techniques presented herein can be applied to any type of color filter array, including RGBW filters, CYGM filters, etc. [0052] By analyzing the coordinates of the item and the type of item on frames from a sequence of frames (for example, a video stream), the Jetson ™ modular supercomputer generates a virtual shopping cart. [0053] In some embodiments, the video stream is split into chunks. A fragment can be a sequence of frames, a single frame, or a part of a frame. The video stream is divided into fragments in such a way that each fragment corresponds to one object or event observed by the camera and of interest for recognition. It is possible that several products are present in one fragment if they appear simultaneously (for example, the user immediately took milk and bread). It is advisable to limit the size of a fragment to reduce the transmission time of the highest priority fragments over communication channels with limited bandwidth and more efficient use of video storage.

[0054] Video processing can be: a) embedded in a video transmitter (network camera or network video server); b) be on the video receiver (on the server); c) distributed between the transmitter and the video receiver.

[0055] The object recognition method can be implemented in the form of software modules and integrated with such components of a video surveillance system as a network camera, network video server, network video recorder (NVR), video management system (VMS), and online video surveillance system. In addition, these software modules can be run on workstations and dedicated servers, in a data processing center (DPC) or hosted by a cloud provider.

[0056] Fragments of the video stream can also be transmitted between system components or to third-party servers or data storage in the form of files, for example, in popular containers such as TS, M2TS, MKV, OGV, MP4, AVI, etc.

[0057] For example, when recognizing an entity from a video stream, the code of this entity is sent to the server via the Internet via HTTPS, the order of actions is received from the server using the entity code (for example, the product is removed from the kiosk, payment), which is set when this entities, then sequentially begin to perform the resulting order of actions to achieve the set goal. An example of a procedure would be to take a product out of a kiosk and pay. The final goal of the entire chain of actions is to receive the product from the kiosk by the user after paying for it.

[0058] In an exemplary embodiment, each frame from the video stream is separated by a virtual ticket line, as shown in FIG. 2, for example, crossing the Uz frame from below, without limitation. When a user takes an item from a kiosk and when a product crosses this line, the item is added to the virtual cart. If a customer, for example, returns an item to the fridge of a kiosk, the item is removed from the virtual cart. Payment takes place after the lock of the kiosk refrigerator is closed. At the time of closing, the system interaction unit 104 initiates payment through the terminal or through the server, depending on what the client pays with: by card or through a mobile application. The algorithm works as follows: when the ANN detects a new object (for example, “milk” product), the computing device determines whether the center of the object is behind the checkout line or outside, and stores the data in the database. To determine the center of the object, the ANN "traces" the product selected by the user with a rectangle or any other geometric figure, not limited to, as shown in FIG. 4. In some embodiments, the ANN generates four digits: the coordinates of the upper left corner (x, y), and the width and height of the rectangle - (w, h). Thus, the center of the object is a point with coordinates (x_c; y_c) x_c = x + w / 2, yc = y + h / 2. On the next frame, it is checked where this object is located, whether the goods have crossed the checkout line. If crossed, the product is added to the virtual cart or removed, depending on the original position of the object.

[0059] If the item appears at the checkout area and crosses the line, then the item is added to the virtual cart. In some implementations, the line is a polygonal line and is formed empirically based on the characteristics of the kiosk (eg, the size of the entire kiosk and / or the size of the refrigerator only, and / or view areas of cameras for recognition, etc.). If the goods appeared outside the checkout and crossed the line, in this case the item is removed from the virtual basket. FIG. 2 on the left shows the name of the types of goods and their quantity. In a specific embodiment in FIG. 2, the product has not yet crossed the checkout line, therefore, everywhere near the names there are zeros in quantity.

[0060] A person skilled in the art will understand that a virtual shopping cart can be implemented in any number of ways within the scope of the present invention.

[0061] In general, a virtual shopping cart is an array / list of objects that are located in a kiosk and that a user wishes to purchase, and goods that the user took out or returned to the refrigerator are added / removed to this array. The virtual basket is changed when the product crosses the virtual checkout line. Upon completion of the purchase, the virtual cart array is transmitted over HTTP to block 104. In some implementations, UDP, RS-232, etc. data transmission protocols may also be used, but are not limited to. Ethernet, Wi-Fi, Bluetooh, Zig-Bee, LoRA, etc. can be used as the transmission medium, but not limited to.

[0062] To speed up the calculations, the ANN processes every n-th frame, and on the remaining frames track the goods using a correlation tracker based on the frame difference. Jetson analyzes the difference between the two frames and determines the coordinates where the object has moved. For example, the line dividing the cashier line is defined by two points t1 (0; 100) and t2 (1280; 100). At the time of the first identification of the product, the coordinates of the center of the product were (740; 30), which means that the product is in the refrigerator area. The user moves the product, for example, takes the product out of the kiosk refrigerator and picks it up. On the next frame, the trained artificial neural network determined the new coordinates of the object (540; 200), i.e. center of item crossed the checkout line. The item is added to the cart. Based on tracking the coordinates of the position on the frame of the goods, its trajectory is formed outside the kiosk, which can be an array of data that is saved to the database. [0063] Which frame will be selected is preselected empirically by analyzing the processing speed and the availability of computing power. In a specific embodiment, the camera captures 30 frames per second, every 4th frame is sent to the ANN, and the intermediate frames with goods are tracked using a correlation tracker. The correlation tracker works based on the difference between adjacent frames. For example, on frame 325, the ANN has found the required object. Then this object is transferred to the library, i.e. the frame (image) itself and the coordinates of the object in pixels. When the 326th frame is received, this frame is sent to the library, and the command is to update the coordinates of the object. Since the frame has not changed much, the library uses algorithms known from the prior art to find where the object has moved and returns the new coordinates of the object.

[0064] As shown in FIG. 1 unit of interaction of systems 104 (for example, in a particular implementation of the Raspberry Pi) through the router 103 sends a request to start the purchase to the computing device 100 (for example, a single board computer Jatson). The interaction of the block of interaction of systems 104 104 and the computing device 100] occurs using GET and POST requests. A signal request is sent when a customer opens a door (or, in some embodiments, another event) to computing device 100, after which computing device 100 turns on cameras. In some implementations, in addition to the Raspberry Pi, any ARM, x86, x64 computer of architecture capable of running Linux can be used.

[0065] Computing device 100 turns on the camera, video recording of the purchase and item counting begins. Upon receipt of the purchase completion signal, the computing device 100 sends the contents of the virtual cart to the systems interaction unit 104, as well as via HTTP using a POST request. Cameras are activated only when the user opens the kiosk door. A purchase completion signal is sent when the user closes the kiosk door. The systems interaction unit 104 receives a signal from the door-in-lock position sensor, and the systems interaction unit 104 signals the completion of the purchase.

[0066] Several cameras can be installed on the kiosk to cover the entire shopping area. Each camera will be connected to a separate computing device 100. In this case, data on the virtual basket from all computing devices 100 will be summarized and sent to the system interaction unit 104.

[0067] Training I NS is carried out by recording test videos, where the user takes the goods from the kiosk, and then breaks the video data into frames (a sequence of frames). Then the coordinates and the class of the goods are marked on the frames. They add noise, distortion, etc. to process the resulting images: stretching, compression, rotation, etc. Then this set of frames is sent to the ANN for training. The ANN learning process is well known from the prior art and is obvious to any specialist. After training, the trained ANN is loaded into the kiosk, after which the kiosk can determine the coordinates of the goods and the type of goods. [0068] The hardware part of the system consists of a set of components, described in detail below, which are functionally all interconnected

[0069] Cameras, (for example, in a specific RBG implementation of a video camera) are connected to a single board computer 100. A Jetson 100 single board computer, or in other words, a computing device based on camera frames determines the product that the user has bought. A kiosk can be presented as a complex of components, or as one specific component. Possible system components may include, but are not limited to: counter, refrigerator, vending machine, coffee machine, barcode scanner, temperature sensor, card reader, monitor, etc. The devices are built into the kiosk by connecting them to a second independent computer responsible for interaction between the kiosk components.

[0070] The door locking unit 107 (or electrical locking system) may be an electromagnetic lock consisting of a metal plate and a magnetic sensor, or for example an electromechanical lock. This unit sends a 12V signal to the server to unlock the lock. When the signal is stopped, the lock is locked. The principle of operation of an electromagnetic lock is based on the interaction of a magnet (built into the lock body) and a counterpart (a metal plate attached to the door). Unlike an electromechanical lock, in this case, the coil is constantly powered, when the power is turned off, the contact between the magnet and the contact plate opens and the door opens. The power of the magnet must be large enough to prevent the door from being forced open. In some embodiments, electromagnetic locks of magnetic locking or magnetic-mechanical locking can be used.

[0071] An information input unit 108, which may be a contactless card reader. A credit card reader is a special device that recognizes data from various types of plastic cards in the following ways:

[0072] 1. Magnetic tape on the card;

[0073] 2. Chip embedded in the card;

[0074] 2. Contactless payment method.

[0075] A credit card reader, when reading from a magnetic tape or chip, has a special analysis slot through which the card is passed. The required information is then scanned automatically. [0076] The contactless method works according to the following principle: the card is brought to the module at a distance of 30 to 250 mm. Its identifier falls under the electromagnetic radiation generated by the device and scans the individual code. The information is redirected to the controller, which happens within a fraction of a second.

[0077] The user brings the plastic card to the reader, and the card and the reader interact with the EMV protocol. The information input unit 108 can be a payment and information universal terminal, made in the form of a computer and equipped with external control and monitoring ports that are used to connect to the central control unit via interface communication lines of an information display device, an information input device, a control unit, a reader for plastic electronic or magnetic cards, sound device, characterized in that it is equipped with a USB port.

[0078] The block for reading temperature information 106, which is a block of N temperature sensors, where N> = 0. Block polls temperature sensors at a specified frequency and generates data on temperature readings. The sensor for measuring temperature is based on the dependence of electrical resistance on temperature. In an exemplary embodiment, the sensor may be a low power temperature sensor such as an LM20 from Texas Instruments ™ or a Ds18b20 from Dallas Semiconductor ™.

[0079] A networking unit 103, for example, implemented as a router for connecting all components via an Ethernet network and accessing the Internet. The router is equipped with N slots for sim cards, where N = 2 and M slots for Ethernet, where M> = 2. Duplication of sim cards is necessary for uninterrupted operation of the kiosk. The unit serves as a connecting function between the kiosk and its internal components, as well as between the kiosk and the server. The router also has a firewall to protect the kiosk from cyberattacks. In general, the aforementioned networking unit 103 may comprise a mobile radio network communication interface such as, for example, a GPRS (General Purpose Packet Radio) modem, a UMTS (Universal Mobile Communications System) modem, an HSPA (High Speed Packet Data) modem, or an LTE modem. (Long Term Development Standard), in which the above interface is connected via a mobile communication protocol to a base station to communicate with the Internet. In this case, the system may include a router, such as an ADSL router or a fiber optic router that connects the LAN to the Internet and communicates with a laptop that communicates over the Internet.

[0080] The system interaction unit 104 is a computing device that polls the temperature information reading unit 106 at regular intervals to obtain data on the temperature inside the kiosk.

[0081] An information processing unit (processor), which is a computer for pattern recognition in a video stream.

[0082] An information input / output unit 105, implemented as a touchscreen display for interaction with a user. The display receives a signal from the System Interconnect 104, for example via an HDMI cable and displays this signal. By means of a touch screen, the display sends a signal to the system interaction unit 104, which is a computer, data on the coordinates of the user's touch to the display. As further shown, display 105 can be configured to generate various images that can be viewed by a user. For example, during operation of a kiosk to purchase goods, display 105 may display a graphical user interface ("GUI") that allows a user to interact with an operating system and / or application running on the kiosk. A GUI can include various layers, windows, screens, templates, or other graphical elements that can be displayed all over, or a portion, of a display device. For example, in the illustrated embodiment, the operating system GUI may include various graphical icons, each of which may correspond to different applications that may be opened or executed upon detection of a user's selection (eg, via keyboard or touch input). Thumbnails can be displayed in a dock or in one or more graphical window elements displayed on the screen. In some embodiments, selecting an icon may result in a hierarchical navigation process whereby selecting an icon leads to a screen or opens another graphics window that includes one or more additional icons or other GUI elements.

[0083] System Interconnection Unit 104, which is a computing device containing a set of modules, which can be either software or hardware, which are disclosed below.

[0084] In an exemplary implementation, the system comprises the following modules:

• a module for pattern recognition in a sequence of frames and counting the goods that the user has bought;

• a module for performing interaction between kiosk components;

• cash module for performing settlements with the user; • operating system;

• database;

• web server.

[0085] The interaction of the above modules and components is carried out as follows:

[0086] When the user opens the kiosk door, a signal is sent to the computing device 100. The cameras begin to record the user's purchase. After the kiosk door is closed, computing device 100 sends a list of items (in the form of a generated virtual cart) that the user has purchased to the module for interaction between the kiosk components.

[0087] The sequence of actions is as follows:

[0088] The user physically approaches the kiosk or any other vending machine a sufficient distance to initiate interaction, for example, one meter.

[0089] The user then unlocks the kiosk through his mobile application 200 or by touching a plastic card to the plastic card reader 108. The plastic card reader 108 sends a signal to the kiosk system interaction unit 104 to the kiosk object interaction module. This module sends a signal to the remote server 109 to check the plastic card and identify the client on it.

[0090] If the client is successfully identified, the kiosk door is unlocked, and if the determination is unsuccessful, an error is displayed, for example, on the kiosk display and asked to repeat the previous step on the kiosk display or in the user's mobile application.

[0091] The user opens the kiosk door with his hand on the kiosk door handle.

[0092] When the door is opened, the signal from the door lock 107 is sent to the kiosk systems interaction unit 104 to the module for interaction between the kiosk objects.

[0093] The module for interaction between the objects of the kiosk sends a signal to the computational module for pattern recognition. [0094] Then, the pattern recognition module sends a signal to the camera set to start video recording.

[0095] The user retrieves the objects of interest from the kiosk. [0096] The user closes the kiosk door, the door is locked. [0097] The door lock 107 signals the kiosk systems interaction unit 104 to the module for interaction between the kiosk objects.

[0098] The module for interaction between the objects of the kiosk sends a signal to the computational module for pattern recognition.

[0099] The pattern recognition module signals the cameras to complete the recording, then receives a sequence of frames from cameras installed in or outside the kiosk.

[00100] Next, the pattern recognition module starts processing video from cameras, determines the product that the user has bought.

[00101] Upon completion of video processing, this module sends a list of goods that the user has bought to the block for interaction between kiosk objects.

[00102] A module for interaction between kiosk objects sends a signal to a server over the Internet to obtain a list of actions to be taken when receiving a signal.

[00103] Next, this module for interaction between the objects of the kiosk sends a signal to the cash register module to perform settlements with the user for the extracted object / s.

[00104] The module for interaction between kiosk objects sends a signal to the server based on the results of the settlement operation with the client for the retrieved object / s.

[00105] The module for interaction between the objects of the kiosk sends a signal to the display device of the kiosk about the result of the operation.

[00106] All components of this technical solution can be connected by a bus. In some examples, a data bus includes one or more data buses. The data bus can be implemented in accordance with the Controller Area Network (CAN) bus protocol, which is defined by the International Organization for Standardization (ISO) 11898-1, the Multimedia System Data Bus (MOST) protocol, the flexible information CAN-bus (CAN-FD) (ISO 11898-7) and / or K-line bus protocol (ISO 9141 and ISO 14230-1) and / or Ethernet ™ -mnHbi IEEE 802.3 protocol (2002 onwards), etc.

[00107] The elements of the proposed technical solution are in a functional relationship, and their joint use leads to the creation of a new and unique technical solution. Thus, all blocks are functionally linked.

[00108] All blocks used in the system can be implemented with electronic components used to create digital integrated circuits, which is obvious to a person skilled in the art. Not limited to, microcircuits can be used, the logic of which is determined during manufacture, or programmable logic integrated circuits (FPGA), the logic of which is set through programming. For programming, programmers and debugging environments are used that allow you to set the desired structure of a digital device in the form of a circuit diagram or a program in special hardware description languages: Verilog, VHDL, AHDL, etc. An alternative to FPGAs can be programmable logic controllers (PLCs), basic matrix crystals ( BMK) requiring a factory production process for programming; ASICs are specialized custom large integrated circuits (LSIs), which are significantly more expensive for small-scale and single-piece production.

[00109] Typically, the FPGA itself consists of the following components:

• configurable logic blocks that implement the required logic function;

• programmable electronic links between configurable logic blocks;

• programmable input / output blocks providing connection of the external output of the microcircuit with the internal logic.

[00110] Blocks can also be implemented using read-only memory devices. [00111] Thus, the implementation of all the blocks used is achieved by standard means based on the classical principles of the implementation of the foundations of computing.

[00112] As will be understood by one skilled in the art, aspects of the present technical solution may be implemented as a system, method, or computer program product. Accordingly, various aspects of the present technical solution may be implemented solely as hardware, as software (including application software, and so on), or as an embodiment combining software and hardware aspects, which may generally be referred to as a "module" , "System" or "architecture". In addition, aspects of the present technical solution may take the form of a computer program product implemented on one or more computer-readable media having computer-readable program code that is implemented thereon.

[00113] Any combination of one or more computer readable media can also be used. The computer-readable storage medium can be, without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or any suitable combination thereof. More specifically, examples (non-exhaustive list) of a computer-readable storage medium include: an electrical connection using one or more wires, a portable computer diskette; hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), fiber optic connection, compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any combination of the above. As used herein, a computer-readable storage medium can be any flexible storage medium that can contain or store a program for use by the system itself, device, apparatus, or in connection therewith. [00114] The program code embedded in a computer-readable medium can be transmitted using any medium, including, without limitation, wireless, wired, fiber optic, infrared, and any other suitable network or a combination of the above. [00115] Computer program code for performing operations for the steps of the present technical solution may be written in any programming language or combinations of programming languages, including an object-oriented programming language such as Java, Smalltalk, C ++ and so on, and conventional procedural programming languages. for example, the "C" programming language or similar programming languages. The program code can be executed on the user's computer in whole, in part, or as a separate software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN), a wide area network (WAN), or a connection to an external computer (for example, via the Internet using Internet service providers).

[00116] Aspects of the present technical solution have been described in detail with reference to block diagrams, circuit diagrams, and / or diagrams of methods, devices (systems), and computer program products in accordance with embodiments of the present technical solution. It will be appreciated that each block from the block diagram and / or diagrams, as well as combinations of blocks from the block diagram and / or diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other data processing device to create a procedure, such that instructions executed by a computer processor or other programmable data processing device create means to implement the functions / actions specified in block or blocks of flowchart and / or diagram. [00117] These computer program instructions may also be stored on a computer-readable medium that can control a computer other than a programmable data processing device or devices that function in a particular way, such that the instructions stored on the computer-readable medium create a device including instructions that perform the functions / actions specified in the block diagram and / or diagram.

Claims

FORMULA 1. A method for recognizing images from a video stream, performed by at least one computing device, and includes the following steps: • receive from at least one camera at least one frame containing at least one product from the kiosk, and each frame is separated by a virtual checkout line; • at least one frame with the product obtained at the previous step is transmitted to the input of a trained artificial neural network, which identifies at least one product on the frame by obtaining the coordinates of the product and the class to which this product belongs; • determining by means of a computing device whether the center of the goods is behind the virtual checkout line or outside, and if at least one item has appeared in the checkout area and crossed the line, this item is added to the virtual cart; • send the contents of the virtual basket, formed in the previous step, to the server to perform settlements with the user for at least one retrieved item. 2. The method according to claim 1, characterized in that each n-th frame with the product obtained at the previous step is transmitted to the input of the trained artificial neural network. 3. The method according to claim 2, characterized in that on the remaining frames, except for the nth one, the goods are tracked using a correlation tracker based on the frame difference. 4. The method according to claim 1, characterized in that when recognizing a product from a video stream, the code of this product is sent to the server via the Internet via HTTPS, and the order of actions is received from the server using the entity code, which is incorporated in the recognition of this product. 5. The method according to claim 1, characterized in that the trained artificial neural network recognizes the expiration date on the product. 6. The method according to claim 1, characterized in that the trained neural artificial network recognizes the place of the first identification of the product and / or the frame of the last identification, and / or the direction of movement of the product, and / or the absolute coordinates of the product. 7. A system for recognizing images from a video stream, containing a vending kiosk, in the body of which the following are structurally installed: • at least one camera installed on the kiosk, configured to receive at least one frame containing at least one product from the kiosk, each frame being separated by a virtual checkout line; • at least one computing device containing a trained artificial neural network and configured to: o receiving at least one frame with a product at the input of a trained artificial neural network, which identifies at least one product on the frame by obtaining the coordinates of the product and the class, to which this product belongs; on determining whether the center of the item is behind the virtual checkout line or outside, and if at least one item appears in the checkout area and crosses the line, this item is added to the virtual cart; • at least one door locking unit configured to send a signal to at least one server to unlock the door lock; • at least one block of information input, configured to receive commands from the user; • at least one system interaction unit capable of transmitting signals between system units. 8. The system according to claim 7, characterized in that the door locking unit is an electromagnetic lock, which consists of a metal plate and a magnetic sensor, and / or an electromechanical lock. 9. The system of claim. 7, characterized in that the computing device is NVIDIA Jetson or a computing device with a GPU. 10. The system according to claim 7, characterized in that it further comprises a temperature information reading unit, which is a set of temperature sensors for generating data on temperature readings inside the kiosk. 11. The system according to claim 7, characterized in that the information input unit is a contactless card reader. 12. The system according to claim 7, characterized in that the system interaction unit is a Raspberry Pi.