WO2015199571A1

WO2015199571A1 - System and method for the interaction of a human with at least one device to be controlled

Info

Publication number: WO2015199571A1
Application number: PCT/RU2014/000450
Authority: WO
Inventors: Artem Vladimirovich OZHIGIN; Sergey Pavlovitch Sobolev
Original assignee: Siemens AG; Siemens Corp
Current assignee: Siemens AG; Siemens Corp
Priority date: 2014-06-24
Filing date: 2014-06-24
Publication date: 2015-12-30
Anticipated expiration: 2016-12-24

Abstract

The present invention relates to a system (1) and method for the interaction of a human person with at least one device to be controlled (2), comprising at least one augmented reality device (3) and at least one central information storage device (4) to store and/or exchange information with the at least one augmented reality device (3) and the at least one device to be controlled (2). The system (1) is formed to identify a person comparing stored data from the at least one central information storage device (4) and determined data about the person measured by the at least one augmented reality device (3), and to identify the at least one device (2) to be controlled comparing stored data from the at least one central information storage device (4) and determined data about the at least one device to be controlled (2) measured by the at least one augmented reality device (3), providing access for the identified person to the at least one device to be controlled (2) and/or to pre-defined components of the device (2), depending on stored access data for the identified person.

Description

SYSTEM AND METHOD FOR THE INTERACTION OF A HUMAN

WITH AT LEAST ONE DEVICE TO BE CONTROLLED

DESCRIPTION

The present invention relates to a system and method for the interaction of a human person with at least one device to be controlled, comprising at least one augmented reality device and at least one central information storage device to store and/or exchange information with the at least one augmented reality device and the at least one device to be controlled, and the method comprising the exchange of information between the at least one central information storage device and the at least one augmented reality device as well as the at least one device to be controlled, and/or storing information from these devices.

Systems and methods for the interaction of a human person with a device, so called human-machine interfaces (HMI) are widely used. HMI can be the source of mistakes and produce certain risks for example in the field of safety issues. An update of HMI is often not possible without expensive system upgrades. A flexible approach with up-to-date HMI at all time and detailed instructions can reduce the risk of misuse and failures. On the other hand it is important to limit access to HMI to avoid unauthorized operations. This requires an HMI which provides information and controls strictly following access rules.

In the state of the art, see for example US7668605B2, HMI is located in a special housing and password protected, with controls being divided into classes. Users with super user privileges require for example special keys. The HMI is part of hardware and/or detached from hardware. HMI detached from hardware is easier updated and can include additional information, for example up-to-date instructions which are easier to use than in paper form.

HMI with password protection includes the risk of misuse by unauthorized persons and needs effort from the authorized person to memorize, protect and type in the password. Keys in hardware form or as letters can be lost or misused. Instructions and information about the use of devises are often not up to date, are difficult to use especially with no hands free by the user and it is difficult to provide specific information for a specific persons and parts of the devices to be controlled.

The object of the present invention is to present a system and a method for the interaction of a human person with at least one device to be controlled, solving the above described problems. Particularly the system and method presented prevent misuse and failures, improve the usability and allow to provide additional, specific, up- to-date information about usage.

The above objects are achieved by a system for the interaction of a human person with at least one device to be controlled according to claim 1 and a method for the interaction of the human person with the at least one device to be controlled, particularly comprising a system as described before, according to claim 9.

Advantageous embodiments of the present invention are given in dependent claims. Features of the main claims can be combined with each other and with features of dependent claims, and features of dependent claims can be combined together.

The system for the interaction of a human person with at least one device to be controlled comprises at least one augmented reality device and at least one central information storage device to store and/or exchange information with the at least one augmented reality device and the at least one device to be controlled. The system is formed to identify a person comparing stored data from the at least one central information storage device and determined data about the person measured by the at least one augmented reality device, and to identify the at least one device to be controlled comparing stored data from the at least one central information storage device and determined data about the at least one device to be controlled measured by the at least one augmented reality device, providing access for the identified person to the at least one device to be controlled and/or to pre-defined components of the device, depending on stored access data for the identified person in the at least one central information storage device.

The identification of the user, i.e. human person and the device to be controlled combined with predefined access rules for the person, reduce the misuse and failures, improve the usability and allow to provide additional, specific, up-to-date information about correct usage. The system can provide specific information for the identified person, which help the person to control the device to be controlled the best way, save and effective. An up-to-date user manual and/or up-to-date specific information about the device to be controlled can be provided.

The at least one central information storage device can be a cloud computer, particularly within a proteaed cloud of a company network. Data in a cloud can be kept up-to-date in an easy way and can be exchanged between different users and devices. A high amount of data can be stored for many devices and users in a secure way.

The at least one device to be controlled can be a device in industrial application, particularly a machine. Also parts / components of a machine can be identified and specific controlled by the user. The machine can for example comprise a robot or vehicle and its movements can be controlled. All other kinds of devices can be used as machine, especially devices in an industrial production process.

The at least one augmented reality device can comprise a special glass or lens, particularly with a camera and/or a display, particularly for transferring information to the person and/or picturing a person and/or device. For example information to the user can be presented in form of a hologram in 3D. Glasses or lenses are easy to wear, with all hands free for further actions and handling of additional devices during control. Cameras, for example CCD-cameras are easy to use, cost effective and compact to fix for example to glasses.

The at least one augmented reality device and the at least one central information storage device and/or the at least one central information storage device and the at least one device to be controlled can be directly or indirect connected to exchange information by a wireless connection, particularly by WLAN. The usage of WLAN increases the freedom to move during control and makes a data transfer easy, even in an complex and harsh environment, which can change fast for example if moving parts from a robot or vehicle are involved and/or the person controlling the device moves around.

An identifier can be comprised by the at least one device to be controlled, particularly at least one RFID tack, at least one bar code and/or at least one QR-code. These identifiers are easy optically to identify and can be fixed to the device to be controlled. Identifier can be comprised by and/or fixed to components of the at least one device to be controlled, particularly by all visible components at least one identifier can be comprised. This makes it easy to identify the component and control it specifically, even with a high number of components.

A component can be comprised for initiating a pairing mechanism between the at least one augmented reality device and the at least one central information storage device and/or between the at least one central information storage device and the at least one device to be controlled and/or between the at least one augmented reality device and the at least one device to be controlled, particularly a component comprising a button to be pressed on the device to be controlled to initiate the pairing mechanism. Only after pressing the button and/or entering an access code pairing occurs and can for example allow the control of the device by the user identified. This provides for example additional security against misuse.

A method for the interaction of a human person with at least one device to be controlled, particularly comprising the before described system, comprises the exchange of information between at least one central information storage device and at least one augmented reality device and at least one device to be controlled and/or storing information from these devices. It further comprises the identification of a person by comparing stored data from the at least one central information storage device and determined data about the person measured by the at least one augmented reality device, and identifying the at least one device to be controlled comparing stored data from the at least one central information storage device and determined data about the at least one device to be controlled measured by the at least one augmented reality device, providing access for the identified person to the at least one device to be controlled and/or to pre-defined components of the device, depending on stored access data for the identified person in the at least one central information storage device.

The access can be allowed according to predefined access rules, particularly stored in the at least one central information storage device.

Optical data measured by the at least one augmented reality device from the person and/or the at least one device to be controlled can be used to identify the person and the device and to provide or deny access for the control of the device particularly depending on stored access rules, particularly after a pairing mechanism is started particularly by pressing a button and/or after entering a identification code.

Predefined settings for the control interface depending on the identified person and/or devices can be used.

The exchange of information and/or data can comprise transmission of data via

WLAN, particularly between the at least one augmented reality device and/or the at least one device to be controlled and the at least one central information storage device, particularly with the at least one central information storage device being comprised by a cloud, particularly a secured company cloud.

Optical identifiers can be used to identify the at least one device to be controlled and/or components of the at least one device to be controlled, particularly QR-codes, bar codes and/or RFID tags.

A retrieved human machine interface can be synthesized in front of the human person in control with augmented reality technology particularly depending on the identified person and the access level of the person, particularly with visualization and/or control parameters stored for the identified person in the at least one central information storage device.

The advantages in connection with the described method for the interaction of a human person with at least one device to be controlled according to the present invention are similar to the previously, in connection with the system for the interaction of a human person with at least one device to be controlled described advantages and vice versa.

The present invention is further described hereinafter with reference to the illustrated embodiment shown in the accompanying drawing, in which: '

FIG 1 is a schematic view of a system 1 for the interaction of a human person with a device 2 to be controlled according to the present invention.

In FIG 1 a system 1 according to the present invention for the interaction of a human person with a device 2 to be controlled using an augmented reality device 3 is shown in schematic view. The system 1 further comprises a central information storage device 4 to exchange with and store information from the augmented reality device 3 and the device to be controlled 2. The exchange of data can be done via WLAN or wire communication. The central information storage device 4 can be comprised by a cloud, particularly a secured company cloud. The device 4 is part of a computer network, particularly a secured company computer network with storage devices and computing devices.

The system 1 identifies a person, comparing stored data from the central information storage device 4 and determined data of the person measured by the augmented reality device 3. A camera can be comprised by the augmented reality device 3 to capture pictures of the person, particularly the iris and process it as iris scan to identify the person. The system 1 identifies a device 2 to be controlled, comparing stored data from the central information storage device 4 and determined data of the device 2 measured by the augmented reality device 3. A second camera or the same camera can be comprised by the augmented reality device 3 to capture pictures of the device 2 to be controlled, particularly the shape of the device 2 and/or identifiers 6 like for example bar codes or QR-codes. The device 2 can also be identified by identifiers 6 like RFID tags.

Depending on access data stored by the central information storage device 4 for the identified person and device 2 the person receives access to control the device 2, particularly according to a predefined access level. The access can also be allowed for selected components only of the device 2 instead of access for the whole device 2.

The described system 2 and method provides HMI detached from the hardware, by storing identification and access data in the central information storage device 4 and projecting information to the person. The detached HMI forms a virtual HMI 5. Special information for the control of the device 2 can be provided to the person by the augmented reality device 3, by projecting it to the eye of the person for example as plane picture or hologram. Information like manuals, names of parts, information about parts like material, age, time limit to exchange the part and so on can be displayed to the person. The display language, control style and other features can be used according to stored predefined settings for the identified person. The access level of the identified person can be stored in central information storage device 4.

The augmented reality device 3 can comprise or be in form of special glasses or lenses, projecting information direct to the eyes of the person or in-between the person and the device 2, for example in form of a 3D hologram. To allow access to control the device 2 a pairing mechanism can be used. The augmented reality device 3 is paired with the device 2 and/or central information storage device 4 to exchange information. The pairing can be automatic after identifying the person and/or the pairing can be started after the pairing mechanism is activated for example by pressing a button and/or entering an access code.

With stored data in the information storage device 4, for example data about the person to control the device 2, the access level, information about the device and its usage, these data can be kept up-to-date particularly with data stored in a cloud. Entries to the company network and/or a connection to the internet can be used to update data.

With every part/component of the device 2 marked with an identifier 6, a specific control of every single component is possible. The person to control the device 2, wearing the augmented reality device 3 can approach the device 2 and is recognized by the system 2 to control the components.

The system 1 can be used in industry, particularly in very harsh environments, with high level of convenience and freedom, since the person can move free in space and has its hands free to act. The access only after identification and the use for example of a secured company network and/or cloud gives a high level of security. Misuse is minimized, up-to-date data are available and the access level can be adjusted by changing stored data in the central information storage device 4. With identification of components, special information about the component and its usage can be displayed to the person.

The above described features of the embodiment according to the present invention can be combined and can be combined with embodiments known from the state of the art. For example more than one person and augmented reality device 3 can be used to control a device 2 at the same time. Instead of a cloud and/or company network a central fixed single computer with hard drive can be used as central information storage device 4. Instead of identifiers 6 like codes, colors or numbers can be used tacked to components of the device 2.

Advantages of the system 1 according to the present invention are as described before and are for example a high level of safety and security. Only authorized persons get access to HMI on software and hardware level. Risks arising from stolen, lost or misused mechanical keys can be avoided and a remote maintenance is possible. It is easier to implement new features or fix bugs in the HMI, giving a higher flexibility to product developers. Changes are easier to implement, since a main part is on software side and hardware can be adjusted independently, part by part. All information can be kept up-to-date easily. A smaller number of mechanical parts increases hardware reliability and decreases costs. New features as operator feedback and accumulation of feedback on a server as well as the integration of social networks are possible. Expert knowledge can be stored, accumulated and provided to the person in control, for example in form of downloads. A virtual HMI allows space reduction for device equipment, less mechanical components, a higher ability to move around for the person in control, with providing the HMI where it is needed. An increase of safety can be reached by keeping a necessary security distance between the person and the device to be controlled, since no direct manual interaction to the device to be controlled is necessary. For easy configuration of neighboring devices or components identifiers can be attached next to each other. The energy consumption can be reduced, since HMI can be materialized depending on demand.

Claims

1. System (1) for the interaction of a human person with at least one device to be controlled (2), comprising at least one augmented reality device (3) and at least one central information storage device (4) to store and/or exchange information with the at least one augmented reality device (3) and the at least one device to be controlled (2), characterized in that the system (1) is formed to identify a person comparing stored data from the at least one central information storage device (4) and determined data about the person measured by the at least one augmented reality device (3), and to identify the at least one device (2) to be controlled comparing stored data from the at least one central information storage device (4) and determined data about the at least one device to be controlled (2) measured by the at least one augmented reality device (3), providing access for the identified person to the at least one device to be controlled (2) and/or to pre-defined components of the device, depending on stored access data for the identified person in the at least one central information storage device (4).

2. System (1) according to claim 1, characterized in that the at least one central information storage device (4) is a cloud computer, particularly within a protected cloud of a company network.

3. System (1) according to any one of claims 1 or 2, characterized in that the at least one device to be controlled (2) is a device in industrial application, particularly a machine.

4. System (1) according to any one of claims 1 to 3, characterized in that the at least one augmented reality device (3) comprises a special glass or lens, particularly with a camera and/or a display, particularly for transferring information to the person and/or picturing a person and/or device.

5. System (1) according to any one of claims 1 to 4, characterized in that the at least one augmented reality device (3) and the at least one central information storage device (4) and/or the at least one central information storage device (4) and the at least one device to be controlled (2) are directly or indirect connected to exchange information by a wireless connection, particularly by WLAN.

6. System (1) according to any one of claims 1 to 5, characterized in that identifier (6) are comprised by the at least one device to be controlled (2), particularly at least one RFID tack, at least one bar code and/or at least one QR-code.

7. System (1) according to claim 6, characterized in that identifier (6) are comprised by components of the at least one device to be controlled (2), particularly at all visible components at least one identifier (6) is comprised.

8. System (1) according to any one of claims 1 to 7, characterized in that a component is comprised for initiating a pairing mechanism between the at least one augmented reality device (3) and the at least one central information storage device (4) and/or between the at least one central information storage device (4) and the at least one device to be controlled (2) and/or between the at least one augmented reality device (3) and the at least one device to be controlled (2), particularly a component comprising a button to be pressed on the device to be controlled to initiate the pairing mechanism.

9. Method for the interaction of a human person with at least one device to be controlled (2), particularly comprising a system (1) according to any one of claims 1 to 8, with

exchanging information between at least one central information storage device (4) and at least one augmented reality device (3) and at least one device to be controlled (2) and/or storing information from these devices (2, 3), characterized in that identifying a person by comparing stored data from the at least one central information storage device (4) and determined data about the person measured by the at least one augmented reality device (3), and identifying the at least one device (2) to be controlled comparing stored data from the at least one central information storage device (4) and determined data about the at least one device to be controlled (2) measured by the at least one augmented reality device (3), providing access for the identified person to the at least one device to be controlled (2) and/or to pre-defined components of the device, depending on stored access data for the identified person in the at least one central information storage device (4).

10. Method according to claim 9, characterized in that access is allowed according to predefined access rules, particularly stored in the at least one central information storage device (4).

11. Method according to any one of claims 9 or 10, characterized in that using optical data measured by the at least one augmented reality device (3) from the person and/or the at least one device (2) to be controlled to identify the person and the device (2) and to provide or deny access for the control of the device (2) particularly depending on stored access rules, particularly after a pairing mechanism is started particularly by pressing a button and/or after entering a identification code.

12. Method according to any one of claims 9 to 11, characterized in that using predefined settings for the control interface depending on the identified person and/or devices.

13. Method according to any one of claims 9 or 12, characterized in that the exchange of information and/or data comprises transmission of data via WLAN, particularly between the at least one augmented reality device (3) and/or the at least one device (2) to be controlled and the at least one central information storage device (4), particularly with the at least one central information storage device (4) being comprised by a cloud, particularly a secured company cloud.

14. Method according to any one of claims 9 or 13, characterized in that optical identifiers (6) are used to identify the at least one device (2) to be controlled and/or components of the at least one device (2) to be controlled, particularly QR-codes, bar codes and/or RFID tags.

15. Method according to any one of claims 9 or 14, characterized in that a retrieved human machine interface is synthesized in front of the human person in control with augmented reality technology particularly depending on the identified person and the access level of the person, particularly with visualization and/or control parameters stored for the identified person in the at least one central information storage device (4).