WO2020111961A1 - Intelligent refrigerator - Google Patents

Abstract

The invention relates to the field of computer engineering, and in particular to smart devices in the Internet of things. An intelligent refrigerator comprises a set of side shelves arranged in a door. A strain gauge is arranged under each side shelf and is capable of determining the weight of the shelves and of directing the measurement results to a commutation controller. A set of video cameras is incorporated into a side panel of the housing from inside and into a top panel. The video cameras are capable of producing video images and of directing the latter to a master controller. The commutation controller is incorporated on the reverse side of an interior front panel of the door and is capable of receiving data from the strain gauges and of relaying said data to the master controller. A set of rotary shelves is arranged in the main compartment of the housing. The rotary shelves are capable of rotating upon receiving a signal from the commutation controller. The commutation controller is capable of making a video recording of the contents of the refrigerator by means of the installed cameras when the door is closed to a specified angle of rotation. The technical result is an increase in the accuracy of determining the physical properties of products and goods situated in the refrigerator.

Description

INTELLIGENT REFRIGERATOR

FIELD OF TECHNOLOGY

[001] This technical solution generally relates to the field of computer technology, and in particular to “smart” devices in the concept of the Internet of things.

BACKGROUND

[002] Currently, an increasing number of devices are connected to the Internet - from computers and smartphones of users to more ordinary things, such as televisions or refrigerators. In the case of connecting new types of devices to the Internet, they “acquire” the prefix “smart” (English “smart”), which serves marketing purposes. When connecting Smart devices to the Internet, users get the opportunity to update the devices themselves, monitor the status of the device (for example, the refrigerator) and integrate the device itself into the so-called “smart house” concept. A similar concept allows you to manage such “smart” things (devices) from one point, checking the status of the work of such things, and customizing them to your personal needs. The concept of “smart home” also touches on another concept called the Internet of Things (IoT), which implies the interaction of the above things without the direct involvement of a person.

[003] In many refrigerators, not even smart models, cameras are installed that allow the wearer to see the products inside. In smart refrigerators, this technology has developed logically - the processor uses the camera image to determine which products are in storage. In addition, it is possible to determine their expiration date with automatic ordering. In South Korea, a platform is being prepared for launch, supporting automatic orders for missing products that form smart refrigerators, focusing on the preferences of the owners. Therefore, now you have to enter the name of the products and their expiration dates manually. When the expiration date is over, or some products come to an end, the display information about this appears on the refrigerator and the phone / smartphone of the owner.

[004] US9920979B2, Refrigerator and control method for the same, is known from the prior art (patent holder: LG Electronics Inc., publication date: 2016-08-04), which discloses a refrigerator structure and a method for controlling it. The refrigerator contains a storage compartment, a drawer, a camera that photographs products inside the drawer, and a controller that recognizes the position of the marker on pictures taken with the camera at time intervals, and which determines information about the status of the drawer based on a change in the position of the marker on photos. However, characterization of products using video cameras may not be entirely accurate, even with the use of machine learning algorithms.

[005] The disadvantage of this technical solution is that the method for determining the contents of the product boxes is not accurate, as a result of which errors may occur in the determination of the contents.

ESSENCE OF TECHNICAL SOLUTION

[006] The technical solution aims to eliminate the disadvantages inherent in existing solutions from the prior art.

[007] The technical problem to be solved with the help of the claimed technical solution is the elimination of the disadvantages inherent in known analogues.

[008] A technical problem or task to be solved in this technical solution is the determination of the products contained in the refrigerator by measuring their physical characteristics, including by analyzing images stored in the refrigerator, obtained using one or more cameras.

[009] The technical result achieved in solving the above problem is to increase the accuracy of determining the physical properties of products and goods in the refrigerator through their remote monitoring.

[0010] Also an additional technical result is the expansion of functionality. [0011] The specified technical result is achieved due to the device, which is developed in the concept of the Internet of things and is a device connected to the network for collecting data and their subsequent processing on the server. Data from the server can be viewed through a web service or mobile application. The online refrigerator user gets access to information about the products in the refrigerator, their weight and length of stay.

BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The features and advantages of this technical solution will become apparent from the following detailed description and the accompanying drawings, in which:

[0013] FIG. 1 illustrates the shelves of the main department of a smart refrigerator.

[0014] FIG. 2 shows the circuit mounting of the shelf on the door and the impact on the strain gauge.

[0015] In FIG. Figure 3 shows an example of a video camera capturing products in a refrigerator and displaying weight through the operation of video analytics.

[0016] In FIG. 4 shows an embodiment of the architecture of a technical solution.

DETAILED DESCRIPTION

[0017] Below will be described the concepts and terms necessary for understanding this technical solution.

[0018] In this technical solution, a system is meant, including a computer system, a computer (electronic computer), CNC (numerical control), PLC (programmable logic controller), computerized control systems, and any other devices that can perform a given , clearly defined sequence of operations (actions, instructions).

[0019] An instruction processing device is understood to mean an electronic unit or an integrated circuit (microprocessor) executing machine instructions (programs). [0020] An instruction processing device reads and executes machine instructions (programs) from one or more data storage devices. Hard drives (HDD), flash memory, ROM (read-only memory), solid state drives (SSD), optical drives can act as storage devices.

[0021] A program is a sequence of instructions for execution by a computer control device or an instruction processing device.

[0022] This technical solution has the following design.

[0023] The door of the smart refrigerator and part of the shelves of the main section, as shown in FIG. 1, equipped with built-in sensors (strain gauges or size / volume sensors), data from which show:

a. weight of shelves (balconies),

B. the size of the filled row (for example, eggs or other goods in a storage format - rows);

c. container filling volume.

[0024] The indicated data may be displayed in a graphical user interface on a mobile user communication device.

[0025] In some embodiments, the container is part of a shelf or balcony, for example, open, mounted on a door.

[0026] A strain gauge sensor can be integrated on the back of the inner front panel of the door, so when removing the shelves you do not need to open the contact groups, you can move the shelves and change the size. A strain gauge sensor is a sensor that converts the measured deformation of solids (shelves) into an electrical signal. Consider the principle of operation and the construction of a strain gauge resistance sensor that converts deformation (tension or compression) into a change in the electrical resistance of conductors or semiconductors. Structurally, the device is a strain gauge with a contact element. It is mounted on the top panel of the device, which is in contact with the measured body. To connect the sensor to the network, special electric taps are used that are connected to the sensitive plate. [0027] The change in the resistance of the strain gauge conductor under the action of deformation is explained by 2 reasons: a change in the geometric dimensions (length, wire diameter) and a change in the resistivity of the material of the strain gauge. The operation of the strain gauge is characterized by a strain sensitivity coefficient (S), which is equal to the quotient of dividing the relative change in the resistance of the strain gauge by its relative deformation. The following types of sensors can be used in this technical solution: foil or film, or wire.

[0028] A set of sensors sends information through a switching controller via a data bus to the head controller. In some embodiments, gas sensors can be integrated into the door panel to detect the presence of spoiled products and also direct this information to the head controller. The data bus can be based on USART, Ethernet, or Arcnet, or Token Ring, or FDDI.

[0029] The head controller may be a microprocessor that implements all the computational actions that make the smart refrigerator as a whole, which is enclosed in the housing of the smart refrigerator or located on the body.

[0030] The switching controller receives information from the strain gauge sensors and redirects it to the head controller. A switching controller provides control of four components: a RAM bus, an AGP, a system bus, and an I / O controller. The switching controller can include controllers for drives, keyboards, mice, and other external interfaces, which provides control of the external interfaces of an intelligent refrigerator (USB, COM, LPT, etc.). A single set of transport and logical protocols may include TCP / IP, UDP / IP, MODBUS, BACnet.

[0031] The shelves (balconies) of the door of the smart refrigerator have the ability to be configured in the likeness of an organizer to the user's requirements. Thus, several different shelves are provided, both in size and purpose (for example, egg, butter, cheese, bottle, under plastic trays, etc.). Shelves are a predefined set that can be interchanged and in height. It is possible to purchase additional additional shelves separately and make, for example, A - three classic balconies in full width, or B - 15 different shelves with a width of 10 cm. 2. If the product is in the tray, it presses on the strain gauge with a certain pressure, which is used to determine the weight.

[0032] When the door controller closes the door at a predetermined optimal angle of rotation, it photographs the contents of the refrigerator using installed cameras (from 2 pcs or more). In some embodiments, the camera or camcorder is located close to the door (otherwise it will be covered by products in the main section), and therefore photographing is not carried out in the closed position. If you open the door completely, then the goods are not visible due to the fact that the camera is installed on the door. Cameras are built into the side panel from the inside, and into the top panel. In some embodiments, cameras are partially immersed in a panel to save space. The number of cameras or camcorders can vary from 2, because one camera in a particular implementation is not enough to obtain a viewing angle, and therefore the quality of the resulting image decreases. Obtaining images from cameras installed on the side door is carried out automatically in case of opening the side door of the smart refrigerator at a predetermined opening angle of this door. This angle is selected empirically and can lie in the range from 1 to 90 degrees. Implementation when opening the door to a predetermined angle is carried out using a door opening angle sensor. The operation of this device is based on the principle of operation of all optical sensors: the emitting element directs the light flux, the electronic data processing unit determines the rate of return of the beam.

[0033] This sensor has extensive configuration options:

• setting the time for holding the output in the active state after operation: 2, 7 or 15 seconds, • you can select the sensitivity level of the sensor - high or low,

• protect against erroneous operation of doors, it is recommended to do this with the close proximity of several automatic doors with various sensors,

• set the viewing angle (angle between the plane of the door and the sensor beam): 7.5, 14.5, 21.5 or 28.5 degrees (step 7 degrees),

• adjust the width of the working area: turn off the right and / or left “wing” of the light beam, thus narrowing the working area of the sensor.

[0034] The main department of the smart refrigerator has rotary shelves and cameras installed on the side panels (set), and when analyzing the contents of the main department, the shelves are scrolled 360 degrees, data from the cameras are transmitted to the head controller, as shown in FIG. 3. In some embodiments, the user scrolls the shelves to any angle by using a mobile application and sending control commands to the switching controller, which redirects them to the rotary shelves (their microcontroller). Cameras can be installed in at least one far corner above each shelf or in several corners, without limitation.

[0035] The head controller preprocesses data and photographs. The head controller sends a control signal to the cameras (its controllers) to take a picture (image acquisition), and then transfers it to the server via a wireless data transfer protocol. In some embodiments, the head controller performs visual editing of the received frames, for example, gluing several, working with white balance, straightening, etc. Segmented information is sent to the server via wireless data transfer technology, for example, Wi-Fi, BLE, GSM or another, at the user's choice. Each strain gauge has its own unique identification number. To connect the area of the received frame from the set of cameras and the sensor number, the sensor is calibrated and the head controller. For example, if 5 containers in one section are installed in an intelligent refrigerator, then it is necessary to calibrate the sensor and controller, for example, when a user clicks on a shelf in the interface of a mobile application and confirms that it is one or a triple shelf. If you use one wide balcony for 5 sections, then the data should be summarized with 5 sensors located under the balcony and equidistant from each other. The implementation of calibration improves the accuracy of determining the physical characteristics of products.

[0036] Depending on the implementation example, the images obtained from the head controller are processed either on the server or locally on the smart refrigerator, and using the computer vision / artificial intelligence methods, the name of the product is determined from the existing database of product images. This database of images can be either borrowed from third manufacturers or its own. At this step, an artificial neural network, previously trained on a test data set, classifies the received image from the head controller and determines its physical characteristics.

[0037] A user in a mobile application or web interface can configure:

<± lower critical limit for these sensors, for visualization on a nugget (frame image in the picture);

e. display all possible additional masks

(product name, sensor data, etc.)

f. other user settings.

[0038] The lower critical limit (for example, by weight) for the sensor is pre-configured by the user according to his requirements. Suppose a user consumes 200 grams per day. milk, on the basis of which a limit of 250 gr. With a decrease in mass below 250 g. notification, display or automatic milk ordering occurs in the user's mobile application (in its graphical user interface).

[0039] The impression is a photograph of the door / main compartment of the smart refrigerator, on which a mask with weight data is applied. If the user chose a threshold of 250 gr. , and the weight is less than 250 gr, then the mask lights up in red, and if the weight of the product is in the required interval - green. In other embodiments, highlighting a mask with a color may indicate expiration of the product.

[0040] In some embodiments, the shelf life of the product, physical data (weight, size, volume) can be used as a mask.

[0041] Custom settings may include SMS alerts, auto-order, purge activation in the event of a souring product, etc.

[0042] Next, through a mobile application or Web interface, an authorized user can see a nugget (photo) of the contents of the refrigerator with a mask applied, reflecting the information:

d. Weight, number in a row, volume of goods - data from sensors;

I. Name of the recognized product;

i. Shelf boundaries (if applicable);

j. Highlighting the critical balance of the goods.

[0043] Based on this data, the user can make decisions about ordering, purchasing goods, or setting up their order automatically.

[0044] The general scheme of the technical solution is presented in FIG. 4.

[0045] The communication component of the technical solution is configured to maintain communication, wired or wireless, between the smart refrigerator and other devices. The device may provide access to a wireless network based on a communication standard such as WiFi, 2G or 3G, or a combination thereof. In one exemplary embodiment, the communication component receives a broadcast signal or information related to a broadcast transmission from an external broadcast control system via a broadcast channel. In one exemplary embodiment, the communication component further includes a short-range (NFC) communication module for maintaining near-field communication. For example, the NFC module can be implemented based on the technology of radio frequency identification (RFID), technology Association data transmission in the infrared range (IrDA), Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0046] In exemplary embodiments, the switching or head controller can be implemented using one or more specialized integrated circuits (ASICs), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), programmable logic arrays type FPGA (FPGA), microcontrollers, microprocessors or other electronic components designed to perform the above methods.

[0047] The head controller provides the following interfaces and functions:

• VHF channel (duplex) for two-way communication with readers through a radiating cable.

• Serial interface support for standard PC connectivity.

• Non-volatile memory for storing configuration parameters.

• Built-in microcontroller.

[0048] The above functional elements of an intelligent refrigerator form a structurally functional unity and are connected through assembly operations to the refrigerator body and to each other, for example, by means of threaded or bolted joints, soldering, etc., depending on configuration requirements when creating the claimed device.

[0049] Thus, all the features of the claimed combination of essential features expressing the essence of the utility model as a technical solution are causally related to the specified technical result (improving the accuracy of determining the physical properties of products and goods in the refrigerator through their remote monitoring) influencing its receipt.

[0050] Modifications and improvements to the above described embodiments of the present technical solution will be apparent to those skilled in the art. areas of technology. The preceding description is provided as an example only and is not subject to any restrictions. Thus, the scope of this technical solution is limited only by the volume of the attached formula of the utility model.

Claims

FORMULA 1. Intelligent refrigerator, made with the ability to remotely monitor the physical characteristics of the products, containing: • a set of side shelves located in the door, under each of which there is a strain gauge made with the possibility of determining the weight of the shelves and sending the measurement results to the switching controller; • a set of video cameras embedded in the side panel of the case from the inside and in the upper panel, configured to receive video images and direct them to the head controller; • a switching controller integrated on the back side of the inner front panel of the door and configured to about receiving data from strain gauges and redirecting them to the head controller; about the direction of the control command of rotation to the set of rotary shelves. • a set of rotary shelves located in the main department of the housing, made with the possibility of rotation upon receipt of a control command from the switching controller; • a head controller located in the housing and made with the possibility of video recording the contents of the refrigerator through the installed cameras when closing the door at a given angle of rotation. 2. The smart refrigerator according to claim 1, characterized in that under the side shelf there is additionally a set of size sensors configured to determine the size of the filled row. 3. The smart refrigerator according to claim 1, characterized in that under the side shelf there is additionally a set of volume sensors configured to determine the filling volume of the shelves. 4. The intelligent refrigerator according to claim 1, characterized in that the head controller performs editing of the received frames. 5. The smart refrigerator according to claim 1, characterized in that each sensor has a unique identifier. 6. The smart refrigerator according to claim 1, characterized in that the sensor and the head controller are calibrated to communicate the area of the received frame from the camera and the sensor identifier.