RU2809641C2 - Control method for smart home appliances - Google Patents

Abstract

FIELD: household appliances; home automation methods.

SUBSTANCE: invention relates to methods for controlling smart household appliances and can be used in the development of home automation systems. The method for controlling smart household appliances is that the control and monitoring of the state of household devices is carried out by sending corresponding commands by voice, gestures, by wireless transmission from an application installed on a portable device or by making calls to a virtual phone number. And this device receives, processes and performs the required actions or transmits the specified commands over a wireless network to one or more other household devices. It also receives and processes data transmitted by these devices and transmits it to the user.

EFFECT: simplified remote control of a centralized system of household devices while simultaneously expanding the list of available functions.

15 cl, 1 dwg

Description

Field of technology

The invention relates to methods for controlling smart household appliances and can be used in the development of home automation systems.

State of the art

At the moment, centralized methods for controlling household appliances are known, where the exchange of information between the user and all smart household devices located in the house is carried out through one given household device. There are also known methods for controlling the operation of household appliances by making calls to virtual phone numbers of devices.

For example, from document GB 2295067 A, a method for controlling a smart home system by making calls to a virtual telephone number is known. Using calls, you can find out the current status of household devices and issue commands to change this status.

However, this method does not involve control from the application, as well as gesture control.

An analogue of the proposed method is the method for controlling smart household appliances, described in document CN 110716463 A. When implementing this method, a smart kettle with voice control is used as a central device through which other household appliances are controlled. Using voice commands, the user can control both the kettle, setting tea preparation parameters, and other electronic devices with a wireless remote control function. In particular, the kettle can transmit the command to turn off the TV, turn on music to the audio system, or maintain the set temperature to the air conditioner. The kettle is connected to the Internet and can inform the user about the weather, tell the time and date, and contact an online translator. The kettle also gives confirmation to voice commands. In particular, it repeats the user's command and asks what actions should be performed upon completion, and after receiving a response from the user, it begins work. The operation of the kettle and, accordingly, other associated household devices can also be controlled from an application installed on a portable device.

The advantage of the proposed method over an analogue method is the ability to control using calls to the virtual telephone number of the control device and the presence of a gesture interface on the control device. Having a virtual phone number allows you to control the operation of household appliances while away from home, and does not require time spent searching for buttons to launch the necessary commands in the application. The gesture interface allows you to set the required settings and launch selected programs with a few gestures, which speeds up and simplifies the control process. In addition, the analogue method does not involve synchronizing the control device with other applications installed on the portable remote control device, such as, for example, a calendar.

The proposed control method also includes additional functions not provided by the analogue method. In particular, the function of repeating a received voice command by the control device: the device repeats the command received from the user and starts or cancels the execution of the required program only after receiving a response from the user. This function allows you to avoid situations of launching programs that were erroneously specified by the user or were heard accidentally, for example, in a conversation or on TV. Another additional function is the function of providing recommendations when choosing the name of the control device: the device informs the user of the five most popular names selected by other users in the application, which simplifies the choice of device name, allowing you to assign an existing name or, conversely, select a name that is not in the list, to avoid confusion.

Thus, the objective of the present invention is to create a centralized method for controlling smart household appliances, where the exchange of information between the control device and the user would be carried out both using voice commands and a portable device with an installed application, and through calls to the virtual number of the control device, as well as control using gestures. This method also assumes the ability to synchronize the control device with applications installed on the portable device.

The technical result of the claimed invention is to simplify remote control of a centralized system of household devices while simultaneously expanding the list of available functions.

Disclosure of the Invention

The stated goal, the technical result required and obtained when using the invention is achieved by the fact that the control and monitoring of the state of household devices is carried out by sending corresponding commands to a device equipped with speech recognition, gestures, receiving calls to a virtual telephone number and a data receiving and transmitting unit over a wireless network voice, gestures, wireless transmission from an application installed on a portable device, or by making calls to a virtual phone number. And this device receives, processes and performs the required actions or transmits the specified commands over a wireless network to one or more other household devices. It also receives and processes data transmitted by these devices and transmits it to the user.

In this case, the device receiving commands is equipped with means of generating voice responses.

The device receiving the commands can be synced with the Calendar application installed on the portable device. In this case, upon the user's request, it reproduces the information recorded in this application.

The device receiving the commands reproduces the received request and starts or cancels the corresponding procedure after receiving confirmation or cancellation from the user.

At the user's request, the device receiving commands reproduces information about the current statuses of all smart household devices transmitted by these devices via a wireless network.

At the user's request, the device receiving the commands temporarily stops the operation of all active devices on the network, generates and provides the user with a complete report on the status of the system, including at least information about the current status of each device on the network, as well as information about all errors in the operation of the system, occurred during a given time interval.

The device receiving commands, upon user request, reproduces the five most popular names assigned to similar devices by other users and stored in the application.

In one embodiment, the gesture recognition means may be a gesture recognition sensor that detects the position of the user's hands by sensing an image captured, for example, by a camera.

At the same time, the gesture recognition sensor can recognize nine types of hand movements: up, down, left, right, forward, backward, circular movement, clockwise and counterclockwise, wave-like movement.

In another embodiment, the gesture recognition circuit may be implemented on a chip with an electric potential sensor that detects disturbances in the electric field caused by the movement of the user's hands.

In a third embodiment, an assistive device placed on the user's body, such as a ring or bracelet, may be used to control the device system using gestures.

In this case, the auxiliary device contains a source of high-frequency radiation with a frequency of 60 GHz, operating in both pulsed and continuous modes, a receiver of radiation reflected from the user’s hand, a sensor that determines the angle of inclination of the user’s hand, a microprocessor and a signal receiving and transmitting unit via a wireless communication protocol .

Pulsed radiation is used to determine the position of the hand, and continuous radiation is used to determine the speed of its movement. The microprocessor receives and processes data coming from the reflected radiation receiver and sensor, determines which command is triggered by the recognized gesture, and transmits the corresponding data to the information receiving and transmitting unit via a wireless communication protocol, which, in turn, transmits it to the device, receiving commands.

Reception and transmission of data between a portable device with an installed application and a device receiving commands, as well as between smart household devices, is carried out using any type of communication/connection, such as Wi-Fi, and/or Bluetooth, and/or GSM mobile cellular communication, 3G, 4G, LTE, and/or via ultra-low power data links forming complex wireless mesh networks, such as ZigBee and/or LoRa and/or LoRaWAN and/or LPWAN and/or 6L0WPAN, and /or Z-wave.

Calls to a virtual number can be made using mobile cellular communications GSM, 3G, 4G, LTE.

Brief description of drawings

The essence of the invention is illustrated by the drawing.

In Fig. 1 is a block diagram showing how data is exchanged between the control device, other household devices on the network and the user. The following designations are used in the drawing.

1 - smart control device, through which the status of all system devices is controlled and monitored;

2 - block for performing the functions of the control device;

3 - block for monitoring the state of the control device;

4 - control unit of control device 1.

Function execution block 2 includes all the means necessary to implement the functions of the device. For example, if a smart kettle is selected as the device 2, the unit 2 includes at least a water volume, heating means, and heating element power means.

The monitoring block 3 of the state of the device 1 includes means for monitoring the state of the elements of the block 2. In particular, continuing the consideration of the previous example, these can be a sensor for the residual volume of water in the kettle, means for measuring the storage time of water, means for monitoring the temperature of the heating element.

Block 4 includes a block for receiving and transmitting data via a wireless network 5, a block for receiving calls to a virtual number 6, a gesture recognition block 7 and a block for receiving voice commands 8. Blocks 6, 7 and 8 receive commands sent by the user, process them and, if necessary, transmit via a wireless network to other smart devices 13-1, 13-2, ..., 13-n (here n is the number of controlled household devices).

To exchange data via a wireless network, devices 1 are equipped with wireless data reception and transmission units 5, 17-1, 17-2, ..., 17-n.

Each device 13-1, 13-2, …, 13-n is also characterized by the presence of a function execution unit, 14-1, 14-2, … or 14-n, respectively, and a monitoring unit 15-1, 15-2, … or 15-n states of each device.

Function blocks 14-1, 14-2, …, 14-n devices 13-1, 13-2, …, 13-n include all the means necessary for these devices to perform their functions. In particular, if the device 13-1, for example, is a robot vacuum cleaner, its function block 14-1 will include at least a reservoir for collecting dust and debris, cleaning means - brushes and a roller, a pump and means autonomous power supply of the system. And the condition monitoring unit 15-1 may include, in particular, means for monitoring pump power, the speed of movement of the vacuum cleaner and a sensor for residual battery charge. Upon a user request received and processed by any of the blocks 5, 6, 7 or 8 of the main device 1, the data received by the monitoring means 15-1 is transmitted to the user. For example, by sending the specified data to the application, by playing it back during a call to a virtual number, or by playing it using voice response generation tools 9.

In Fig. 1 also indicated:

10 - router;

12 - cloud server;

16-1, 16-2, …, 16-n-control units of devices 13-1, 13-2, …, 13-n.

Control units 16-1, 16-2, ..., 16-n devices 13-1, 13-2, ..., 13-n may also contain any other means of controlling the operation of these devices. In particular, they may include a control panel, means of receiving voice commands, a gesture interface, etc.

The control unit 4 of the main device 1 may also include a control panel or other control means located directly on the body of the device 1.

Carrying out the invention

Currently, there are a large number of smart household devices with control functions via an application over a wireless network. Control via the application allows you to simply and quickly send various commands to the device and monitor its status. However, if there are several such devices in the house, managing them requires opening several applications, which creates certain inconveniences.

The proposed method for controlling smart household appliances allows you to control all smart household devices located in the house, sending commands to only one of them. In Fig. 1, this device is indicated by the number 1. In this case, commands can be sent over a wireless network, from an application installed on a portable device 11, and can also be transmitted by voice - directly or by calling the virtual number of the device: device 1 is equipped with means of recognizing voice commands 8 and means of receiving virtual calls 6, the processing of which is carried out on the cloud server 12.

The ability to transmit commands by voice allows you to control the operation of household devices 13-1, 13-2, ..., 13-n without directly interacting with them and without the need to open the application, which is convenient in situations when the user is busy, for example, cooking or cleaning and not can quickly open the application. Voice control also makes smart home appliances easier to use for older people, who often have difficulty using apps, and for people with disabilities who are unable to use appliances as usual, such as vision problems or impaired motor skills.

The ability to transmit commands by calling a virtual device number allows for remote control and monitoring of the status of household appliances without the need to interact with the application, which can be convenient in situations where the user is away from home and is not able to devote the required amount of time and attention to working with the application. In particular, if he is walking down the street or driving a car. This control method is simplified as much as possible when using wireless communication devices, such as headsets or wireless headphones with a microphone connected to a smartphone via Bluetooth. In this case, the user’s hands remain free and the transmission of commands to home appliances does not interfere, for example, with driving a car.

Calls to a virtual number can be made using GSM, 3G, 4G, LTE mobile cellular communications to the number assigned to the user in the application. Calls are processed on cloud server 12 and then transmitted to block 6. Call processing can be carried out either in tone mode, when in order to receive information from a certain household device or initiate a certain action, the voice informant prompts the user to press the appropriate button on the phone/smartphone screen; and in voice recognition mode. In the latter case, to request information about the status of the device, the user can ask a simple question, for example: “What’s wrong with the humidifier?”, after receiving which the system will generate a voice response about the status of the device, and also offer options for possible actions, for example: “The humidifier is on, target humidity 55 %, actual humidity - 52%. Should I leave the humidifier on? If the user confirms the proposed action by answering “Yes,” the system will leave the humidifier turned on; if a negative answer is received (“No”), the system will turn off the humidifier.

Remote control commands using a call to a virtual number are equivalent to regular voice commands. In particular, the user can start or interrupt the operation of the device 1 to which the call is made, or any other smart device 13-1, 13-2, ..., 13-n associated with it; set the required operating parameters, request information about the status of one or more devices of interest. The user also has the option of repeating the received command by the device. If this option is enabled, the device, upon receiving a command, repeats it and starts or cancels the required operations only after receiving confirmation or cancellation from the user. To reproduce commands, information about the status of connected devices, notifications, and the like, the device is equipped with means for reproducing voice responses 9, such as speakers. The user has access to the "System Analysis" function, when activated, device 1 stops the operation of all active devices on the network and provides the user with a report containing information about the current status of each device 13-1, 13-2, ..., 13-n, as well as information about all errors that occurred during the operation of the system over a certain period.

For ease of reference, device 1 may be given a user-selected name. When choosing a name, the user can use the hint function, when activated, the device plays the five most popular names specified by other users. Information about popular titles is taken from the application.

Reception and transmission of data between the portable device with the application 11 and the control device 1, as well as between the control device 1 and other associated household devices 13-1, 13-2, ..., 13-n, is carried out using any type of communication/connection, such as Wi-Fi and/or Bluetooth and/or mobile cellular GSM, 3G, 4G, LTE, and/or via ultra-low power data links forming complex wireless mesh networks such as ZigBee, and/ or LoRa, and/or LoRaWAN, and/or LPWAN, and/or 6L0WPAN, and/or Z-wave.

Data exchange can be carried out according to the scheme shown in Fig. 1. The control device 1 and remote control devices 13-1, 13-2, ..., 13-n can be connected to one router 10 connected to a cloud server 12 on which user data is stored.

The described data exchange scheme is given as an example and does not limit the possibility of implementing other data exchange schemes between network devices 13-1, 13-2, ..., 13-n and between the control device 1 and a portable device with the application 11 installed.

The smart device 1, through which control is carried out, also has a gesture interface 7: the user can launch various programs using gestures, without the need to directly interact with the device itself. If the control is a kitchen appliance, the presence of a gesture interface makes it possible to easily control this device, for example, in the process of cooking when your hands are wet or dirty. The gesture interface is also convenient in cases where using voice commands is not desired. For example, at night or other times when silence is required. The ability to control household appliances using gestures makes the proposed control method suitable for mute people and people with speech problems.

Gesture interface 7 can be implemented using any known gesture recognition sensors and programmable integrated circuits that allow receiving and processing information about received gestures and launching the execution of the corresponding program. PAJ7620U2 or any sensor similar in operating principle can be used as a gesture sensor.

The PAJ7620U2 sensor used as an example detects the position of the user's hands by reading the image captured by the camera. The specified sensor detects the user's approach and in real time distinguishes nine movement options: movement up, down, left, right, forward, backward, hand rotation clockwise and counterclockwise, wave-like movement.

Gesture recognition algorithms are implemented by the controller of the sensor itself and do not require the use of additional resources.

The information received by the sensor about the gestures made by the user is transmitted to the elements of the control unit (not shown in the drawing), which trigger the execution of the corresponding commands on the control device 1, or, if the commands are addressed to network devices 13-1, 13-2, ..., 13 -n, -transmitting relevant information to block 5 for receiving and transmitting data via a wireless communication protocol.

To understand the essence of the operation of the gesture sensor, the applicant provides information about its possible implementations. So, for example, to implement gesture control, a control system can be used that contains a module for collecting, processing and transmitting information about hand gestures and a control module, in which the module for collecting, processing and transmitting information about hand gestures is used to communicate with the control module for collecting hand gesture data in real time, performing processing on the collected gesture data to obtain corresponding control information, sending control information wirelessly.

Gesture control can also be implemented using an assistive device placed on the user's body, such as a ring or bracelet. The operating principle of an auxiliary device of this type is based on the inclusion in its design of a source of high-frequency radiation, a sensor unit, a microprocessor and a module for receiving and transmitting data via a wireless communication protocol. The microwave source contains two generators of electromagnetic radiation, one of which operates in a pulsed mode, and the second in a continuous mode. The device works like a radar: radiation with a frequency of 60 GHz is directed to the user's hand, and the reflected radiation hits the receiver. Pulsed radiation is used to determine the position of the user's hand, and continuous radiation is used to determine the speed of its movement. High frequency, 60 GHz, provides high data throughput.

The touch unit includes a set of sensors that allow receiving auxiliary data, in particular, determining the angle of inclination of the user's hand and the distance to the device 1 to which commands are sent.

The microprocessor processes the information received using the radar and the sensor unit, determines which command is triggered by the recognized gesture and transmits this information to the data receiving and transmitting module via the wireless communication protocol of the auxiliary device, which transmits the command received from the user to device 1.

Data exchange between the auxiliary device and device 1 can be carried out using any type of communication/connection, such as Wi-Fi and/or Bluetooth, and/or mobile cellular GSM, 3G, 4G, LTE, and/or through data links with ultra-low power consumption, forming complex wireless networks with mesh topology, such as ZigBee, and/or LoRa, and/or LoRaWAN, and/or LPWAN, and/or 6L0WPAN, and/or Z-wave.

However, this is not the only possible gesture recognition sensor used. It should be understood that the sensor may also be configured as a potential integrated circuit sensor capable of recognizing a user's gesture using the potential integrated circuit sensor and performing a function corresponding to the gesture.

Another option for implementing a gesture interface is a gesture interface based on EPIC (Electric Potential Integrated Circuit) sensors, which are a chip with an electrical potential measurement sensor. Sensors of this type can be used to detect any disturbance in the local electric field at a distance of up to several tens of meters. Since the human body is a collection of conductive and polarizable substances, it causes a large disturbance in the electric field, which can be detected by the EPIC sensor. A slight change in the position of the hand causes a signal to appear that is easily recorded by a sensor located several meters from the person, for example, on the body of device 1, which allows the use of EPIC sensors to implement a gesture control system.

The gestural interface implementation schemes described above are given as examples and do not limit other possible options for implementing a control system using gestures.

In addition, the gesture recognition sensor can be made on the basis of a wireless signal, in which the gesture recognition sensor is coordinated with the equipment, receives information about radar measurements, and then calculates information about gestures. In this case, when used as a gesture recognition sensor, a wireless signal-based interaction management system can exploit the high-speed transmission and high-bandwidth characteristics of 60 GHz millimeter waves, which in turn allows hand gestures to be accurately measured and integrated. Therefore, the user can input and operate the equipment without using, for example, a touch screen, and the camera does not need to be additionally placed in the equipment, which improves the control efficiency of the smart device and improves its usability.

The control smart device 1 can be synchronized with other applications installed on the portable device 11 with a wireless remote control application. Such applications, in particular, may include a calendar. In this case, the device can, upon user request, reproduce information, for example, about scheduled business meetings, events and holidays. The device can also be synced with other applications selected by the user, such as notes, messages, email, etc.

As follows from the description of possible implementations of the claimed invention, the proposed method of controlling smart household appliances ensures the achievement of the stated technical result, namely simplification of remote control of a centralized system of household devices while simultaneously expanding the list of available functions.

Taking into account the novelty of the set of essential features, the technical solution of the problem, the inventive step and the materiality of all general and particular features of the invention, proven in the section “State of the art” and “Disclosure of the invention”, the technical feasibility and industrial applicability proven in the section “Implementation and industrial implementation of the invention” invention, the solution of the set inventive problems and the confident achievement of the required technical result in the implementation and use of the invention, in our opinion, the claimed invention satisfies all the requirements for patentability for inventions.

The analysis also shows that all the general and specific features of the invention are essential, since each of them is necessary, and together they are not only sufficient to achieve the purpose of the invention, but also allow the invention to be implemented industrially.

Claims (15)

1. A method for controlling smart household appliances, which consists in the fact that the control and monitoring of the state of household devices is carried out by sending corresponding voice commands to a device equipped with speech recognition, gestures, receiving calls to a virtual telephone number and a data receiving and transmitting unit via a wireless network , gestures, by wireless transmission from an application installed on a portable device, or by making calls to a virtual phone number; and this device receives, processes and performs the required actions or transmits specified commands over a wireless network to one or more other household devices, and also receives and processes data transmitted by these devices and transmits them to the user, while at the request of the user, the device receiving the commands , temporarily stops the operation of all active network devices, generates and provides the user with a complete report on the state of the system. 2. The method according to claim 1, characterized in that the device receiving the commands is equipped with means for generating voice responses. 3. The method according to claim 2, characterized in that the device receiving the commands is synchronized with the “Calendar” application installed on the portable device and, upon the user’s request, reproduces the information recorded in this application. 4. The method according to claim 2, characterized in that the device receiving the commands reproduces the received request and starts or cancels the execution of the corresponding procedure after receiving confirmation or cancellation from the user. 5. The method according to claim 2, characterized in that, at the user’s request, the device receiving commands reproduces information about the current statuses of all smart household devices transmitted by these devices via a wireless network. 6. The method according to claim 2, characterized in that, at the user’s request, the device receiving commands temporarily stops the operation of all active devices on the network, generates and provides the user with a complete report on the state of the system, including at least information about the current status of each device network, as well as information about all errors in the system that occurred during a given time interval. 7. The method according to claim 2, characterized in that the device receiving the commands, at the user’s request, reproduces the five most popular names assigned to similar devices by other users and stored in the application. 8. The method according to claim 1, characterized in that the gesture recognition device is a gesture recognition sensor that determines the position of the user’s hands by reading an image obtained, for example, using a camera. 9. The method according to claim 8, characterized in that the gesture recognition sensor recognizes nine types of hand movements: up, down, left, right, forward, backward, circular movement clockwise and counterclockwise, wave-like movement. 10. The method according to claim 1, characterized in that the gesture recognition circuit is implemented on the basis of a microcircuit with an electric potential sensor that registers disturbances in the electric field caused by the movement of the user’s hands. 11. The method according to claim 1, characterized in that an auxiliary device placed on the user’s body, such as a ring or bracelet, is used to control the system of devices using gestures. 12. The method according to claim 11, characterized in that the auxiliary device contains a source of high-frequency radiation with a frequency of 60 GHz, operating in both pulsed and continuous modes, a receiver of radiation reflected from the user’s hand, a sensor that determines the angle of inclination of the user’s hand, a microprocessor and a unit for receiving and transmitting a signal via a wireless communication protocol. 13. The method according to claim 12, characterized in that pulsed radiation is used to determine the position of the hand, and continuous radiation is used to determine the speed of its movement; the microprocessor receives and processes data coming from the reflected radiation receiver and sensor, determines which command is triggered by the recognized gesture, and transmits the corresponding data to the information receiving and transmitting unit via a wireless communication protocol, which, in turn, transmits it to the device, receiving commands. 14. Method according to paragraphs. 1-13, characterized in that the reception and transmission of data between a portable device with an installed application and a device receiving commands, as well as between smart household devices, is carried out using any type of communication/connection, such as Wi-Fi and/or Bluetooth, and/or GSM, 3G, 4G, LTE mobile cellular communications, and/or via ultra-low power data links forming complex wireless mesh networks, such as ZigBee, and/or LoRa, and/or LoRaWAN, and/or LPWAN, and/or 6LoWPAN, and/or Z-wave. 15. Method according to paragraphs. 1-14, characterized in that calls to a virtual number can be made using mobile cellular communications GSM, 3G, 4G, LTE.

Images