WO2021073733A1

WO2021073733A1 - Method for controlling a device by a human

Info

Publication number: WO2021073733A1
Application number: PCT/EP2019/078099
Authority: WO
Inventors: Boris GROMOV; Jérôme GUZZI; Alessandro Giusti; Luca Maria Gambardella
Original assignee: Supsi
Current assignee: Supsi
Priority date: 2019-10-16
Filing date: 2019-10-16
Publication date: 2021-04-22
Anticipated expiration: 2022-04-16

Abstract

It is disclosed a method for controlling a device (10) by a human (1), wherein a position (12) of the device (10) and a position (2) of the human (1) is known in a system (100), and wherein the system (100) calculates a pointing ray (101) based on a posture of the human (1). The system (100) defines at least two virtual surfaces (120, 130) and supports selection of one virtual surface (120) of said at least two virtual surfaces (120, 130) for movement of the device (10) on said one virtual surface (120) or selection of another virtual surface (130) of said at least two virtual surfaces (120, 130) for movement of the device (10) on said another virtual surface (130); a configuration (120P) in a 3D space of the selected surface among said at least two virtual surfaces (120, 130) is calculated by the system (100) based on at least said position (12) of the device (10) in the 3D space when said surface selection is made; the system (100) further calculates an intersection point (102) of the pointing ray (101) with the selected surface among said at least two virtual surfaces (120, 130) and set the intersection point (102) as a target destination (13) of the device (10); the system supports switching selection between said one virtual surface (120) and said another virtual surface (120).

Description

Title: Method for controlling a device by a human Field of application

The present invention relates to a method for controlling a device by a human, wherein a position of the device and a position of the human is known in a system, in particular a computer system, and wherein the system calculates a pointing ray based on a posture of the human to determine a target destination of the device.

For instance, the device is a drone and the target destination of the drone is a point in the 3D space.

The present invention also relates to a system implementing the above cited method, and in particular a system for controlling a device by a human, the system including electronic means to determine a position of the device and a position of the human, and to calculate a pointing ray based on a posture of the human.

Prior art

It is known that human users increasingly need simplified method to control devices remotely. For instance, it is the case of a drone to be controlled in close proximity of the human user, within visual contact, and along complex 3D trajectories.

It is desirable for the human user to control the device by pointing at a desired target position p_t in 3D space for the device to reach and to see the device moving at that position. This allows the human user to finely control the device in a continuous manner, and drive the device along complex trajectories. This method to control would be very attractive because it avoids any intermediate means, at least to the perception of the user. Once the human user has visually identified a target position for the device, the cognitive effort required to point at it is minimal.

However, regardless of the way it is sensed, the act of pointing by itself does not univocally identify the desired target position p_t of the device; in fact, a given pointing stance identifies a pointing ray r in 3D space, originating at the user’s position and extending to infinity along a given 3D direction.

In other words, p_t might lie anywhere on r.

In some scenarios, certain assumptions allow to identify target point pt, for instance when: a) the movement of the device is constrained to a plane (e.g. a ground robot on flat ground, or a flying robot flying at a fixed height)

In such a case, p_t may be identified as the intersection between r and the plane. b) p_t lies very close to a solid surface (e.g. a ground robot on generic terrain, including legged robots in indoor environments with stairs or a flying inspection robot directed to closely observe details of a complex 3D structure).

In such a case, p_t may be identified by intersecting r with a model (assumed known) of the world’s surfaces; and if r intersect such model in multiple points, the one closest to the human used is selected, as it is the one that the human user directly sees and most likely intends.

However, in some cases, it is more difficult to identify the target position.

For instance, it is the case of a drone that freely moves in 3D space, since the above mentioned constraints a) and b) do not apply. It is therefore not known to unambiguously determine p_t from r. Even in the assumption that the human user may controls the drone in a continuous manner accounting for his perception of the drone’s position, it is not presently known how to actually move the device. For instance, if the human user is pointing at the drone while it flies a few centimetres from the ground and one meter in front of him, it is not possible to adjust the pointing stance by slightly increasing the elevation of human user’s arm. Indeed, a computer system, equipped with the presently known method, is not enabled to determine whether such movement of the arm corresponds to a request of the human user to move the drone farther from the user while staying at the same height, or that of moving the drone only in the up direction.

More particularly, the problem at the base of the invention is to provide a method to improve the use of pointing gestures to control remote devices also when the above mentioned constraints are not valid.

The reason to use pointing gestures is that they are used in everyday lives to efficiently communicate locations and directions in the surrounding 3D space to other people, and so they are also a popular choice in robotics, where they are may be efficiently used for such tasks such as pick-and-place object and area labelling, teaching by demonstration, point-to-goal, selection of a robot within a group, and assessment of the joint attention. However, acquiring pointing gestures requires two macro-processes. The first macro process is perception of the gesture itself and the second one is the estimation of the pointed location.

The perception of gestures in the first macro-process can be performed based on different techniques, meaning by a robot or by a group of robots, by instrumented environment or by a device worn by the user. The first technique (perception by robot) is the most popular in human-robot interaction (HRI) research, however it requires solving a challenging perception problem because the robot has to continuously sense the user. On the other hand, relying on sensors placed in the environment according to the second known technique relaxes the requirements on the robots but limits the applicability of the first macro-process to properly instrumented areas. In both the first and second techniques, the positions being pointed at need to be inferred by external observation, which is typically performed with cameras or RGB-D sensors.

Moreover, regardless of the specific technique adopted for sensing and assuming a perfect knowledge of the user’s posture, it has to be solved the problem of interpreting such a posture and mapping it to the point in the environment that the user wants to indicate, in the second macro-process.

This is typically solved in two steps: the first step is to identify a direction (i.e. a ray in 3D space) and put such ray in relation with the environment to get a point or object. The first step may be solved by defining a ray that originates at a head or at an arm and passes through another point located on the arm, for example, a tip of the index finger. Different configurations of these two points define a number of pointing models, in particular in robotics: head- finger, upper arm, and forearm models. Once the pointing direction is found, the pointed location or object has to be identified as a second step. This may be done by intersection of the pointing ray with the environment. For example, with the large public displays or with the model of the environment acquired with the depth sensors. However, in the absence of the above mentioned constraints a) and b), the second macro process cannot be carried out appropriately.

The problem at the base of the present invention is that of providing a method to control movement of a device using pointing rays, also when it is not possible to identify a target position for the device based on the presence of an obj ect for intersection with the pointing rays, and more particularly providing a method adapted to enable the human user to drive, by means of his free gesture, the path where a drone has to move, both in case where long distance between the device and the human user are involved and where fine adjustment of the position of the device has to be applied in closed spaces, therefore overcoming all the limitations that currently affect the known methods.

Summary of invention

The solution idea at the base of the present invention is to define a target 3D position in the free space by introducing virtual workspace surfaces of different shapes, in the following referred as virtual surfaces, adapted to constrain the movement of a device (a robot, a drone), e.g. by plane, cylinder, or sphere, just to cite some examples.

According to the solution idea given above, the technical problem is solved by a method for controlling a device by a human, wherein a position of the device and a position of the human is known in a system, for instance a computer system, and wherein the system calculates a pointing ray based on a posture of the human, characterized by the fact that

-the system defines at least two virtual surfaces,

-the system supports selection of one virtual surface of said at least two virtual surfaces for movement of the device on said one virtual surface or selection of another virtual surface of said at least two virtual surfaces for movement of the device on said another virtual surface wherein

-a configuration in a 3D space of the selected surface among said at least two virtual surfaces is calculated by the system based on at least said position of the device in the 3D space when said surface selection is made, wherein

-the system further calculates an intersection point of the pointing ray with the selected surface among said at least two virtual surfaces and set the intersection point as a target destination of the device, and wherein

-the system provides switching selection between said one virtual surface and said another virtual surface of said at least two virtual surfaces.

In one embodiment, when selection is switched between said one virtual surface to said another virtual surface of said at least two virtual surfaces to select said another virtual surface, the configuration in the 3D space of said another virtual surface is calculated by the system based on at least the position reached by the device in the 3D space on said one virtual surface.

In an aspect of the invention, a virtual surface of the at least two virtual surfaces is a plane.

In another aspect of the invention, the position of the device is at a predetermined height from the ground.

In a further aspect of the invention, the one virtual surface of the at least two virtual surfaces may have a predetermined orientation with respect to the ground, preferably parallel to the ground.

In an aspect of the invention, said another virtual surface of said at least two virtual surfaces is a plane parallel to said one virtual surface and a position of said another virtual surface is at a predetermined distance with respect to said one virtual surface, wherein the device is moved from said one virtual surface to said another virtual surface when said selection is switched.

In an embodiment, said another virtual surface of said at least two virtual surfaces is calculated based on a surface of an object, for instance a wall or a ground shape, and said another virtual surface is a surface parallel to the surface of the object, at a predetermined distance from the surface of the object.

In still another aspect of the invention, said another virtual surface of the at least two virtual surfaces is a plane perpendicular to the ground.

In a further aspect of the invention, the another virtual surface of the at least two virtual surfaces is calculated based on the position of the human.

Further aspects of the invention are summarized below and described in more detail according to possible embodiments of the invention.

The another virtual surface may be at a predetermined distance from the human.

The another virtual surface is a cylindrical surface with an axis perpendicular to the ground and passing through the human or a spherical surface with an origin on or in the human body, and the radius of the cylinder being calculated based on the position of the device with respect to the human or based on the position of the device with respect to the human when said selection is switched from said one virtual surface to said another virtual surface of said at least two virtual surfaces to activate said another virtual surface.

The device may be a drone.

The system may be a computer implemented system.

The human may be provided with an interface for switching selection between said one virtual surface and said another virtual surface, for instance a command device including a push button which is kept pressed for selecting said one virtual surface and released for selecting said another virtual surface, a voice command, or a gesture.

The technical problem above identified is also solved by a system according to the present invention and in particular by a system for controlling a device by a human, the system including electronic means to:

-determine a position of the device and a position of the human, and -calculate a pointing ray based on a posture of the human, characterized by the fact that

-the system further includes processing means to define at least two virtual surfaces and to select one virtual surface of said at least two virtual surfaces for movement of the device on said one virtual surface or another virtual surface of said at least two virtual surfaces for movement of the device on said another virtual surface wherein

-the processing means are programmed to calculate configuration in a 3D space of the selected surface among said at least two virtual surfaces based on at least said position of the device in the 3D space when said surface selection is made, wherein

-said processing means further calculates an intersection point of the pointing ray with the selected surface among said at least two virtual surfaces and set the intersection point as a target destination of the device, and wherein

-the system further includes a switch for switching selection between said one virtual surface and said another virtual surface of said at least two virtual surfaces.

Brief description of Drawings

Figure 1 schematically represent how to control a device by a human, in a phase of the method of the present invention. Figure 2 schematically represent how to control a device by a human, in another phase of the method of the present invention.

Figure 3 schematically represent how to control a device by a human, in further phase of the method of the present invention.

Figure 4 schematically represent how to control a device by a human, in still another phase of the method of the present invention.

Detailed description

With reference to figures 1-4, it is here below described an embodiment of the method for controlling a device 10 by a human 1 according to the present invention.

Without limiting the scope of protection of the invention, the following description is given taking in consideration, for the device 10, a drone 10. However, different devices remotely controllable from the user 1 may be driven with the method of the invention, in the same way as described below.

The method is executed by a system 100, for instance a computer system 100 or another electronic system 100, taking in consideration a position of the device 10 and a position 2 of the human 1, which are therefore information known in the system 100.

The system 100 calculates a pointing ray 101 based on a posture of the human 1.

For instance, and still without limiting the scope of protection of the invention, the pointing ray 101 is a virtual ray calculated by the computer system 100 based on a position of an arm (or a hand or a finger thereof) of the user 1 and the head of the user 1, meaning that the pointing ray is calculated by the system 100 as the virtual straight line passing through a point on the user’s head and a point on the user’s arm (preferably a finger extended from the hand). Other way of processing the pointing ray may be adopted, for instance calculating the point ray based on different features or postures of the user’s body.

According to the method of the present invention, the system 100 defines at least two virtual surfaces 120, 130.

The shape of the virtual surfaces 120, 130 may be stored in a memory of the system 100. For example, one virtual surface 120 may be a cylindrical surface and another virtual surface may be a planar surface 130. A location of the at least two virtual surfaces and their geometric features, for instance the radius of the cylindrical surface 120 or the height of the planar surface 130 in the 3D space are not necessarily stored in the memory of the system 100 from the beginning, for instance not before executing other steps of the method which are described below. In other words, in a preferred embodiment of the invention, only the shape (cylindrical, planar) of the at least two virtual surfaces is defined in the system 100 at the beginning and the configuration of the shape, such as the location and other geometric features thereof, is determined subsequently at run time, when the virtual surface is selected, in particular based on a position of the device 10.

The system 100 may include a program enabling the user 1 to choose which virtual surfaces 120, 130, ... to use for controlling movement of the device 10, meaning how many virtual surfaces to use (at least two virtual surfaces) and the shapes of each of the virtual surfaces. For convenience only, the following description is made on the assumption that two virtual surfaces 120, 130 are defined and therefore that the subsequent steps of the method are executed moving the device on two virtual surfaces.

In particular, in the example given with reference to figures 1-3, the system 100 supports selection of one virtual surface among the cylindrical surface 120 and the planar surface 130.

In figure 1, the selected surface 120 is the cylindrical one. The cylindrical surface of figure 1, in particular has specific location and geometric features, such as the radius, which are determined at selection time, based on the location of the device 10 at selection time, as described later on. After the cylindrical surface 120 is selected (fig. 1) and configured, movement of the device 10 may be controlled along the cylindrical surface (i.e. appropriately configured in the 3D space), for instance from a starting position 11, corresponding to a position where the device 10 is located at the time of selection, and a target position 13, depending on the user command, both position 11, 13 laying on the cylindrical surface 120 as configured.

Instead, when the planar surface 130 is selected (fig. 3) and configured, movement of the device 10 is controlled along the planar surface 130, still from a start position 13, corresponding to a position where the device 10 is located at the time of selection of the planar surface, to an end position 14, both on the planar surface 130. In the example above, the start position 13 on the planar surface corresponds to the end position 13 on the cylindrical surface whereto the device was moved along the previously selected surface (the cylindrical surface 120).

In order to determine how to move the device 10 on the selected surface, the system 100 calculates an intersection point 102 of the pointing ray 101 with the selected surface.

For instance, with reference to fig. 1, where the cylindrical surface 120 is the selected surface, the system 100 detect the posture of the user and calculates the pointing ray 101 based on the posture and the intersection point 102. The intersection point 102 is set by the system as a target destination 13 of the device 10. Therefore, the device 10 moves from the position 11 to the target destination 13 indicated by the user.

Advantageously, the user knows that the selected surface has a predetermined shape and configuration (in this example the cylindrical surface with a radius corresponding to the distance between the user 1 and the device 10, as described in more detail later on), and therefore it is natural and intuitive for him/her to point the target destination 13 by means of a posture and then expecting the device 10 reach this target destination 13 as a results of calculations and commands set by the system 100.

Once the device 10 reaches the target destination 13, another movement of the device 10 may be controlled on the virtual surface 120, if the cylindrical surface is kept selected. In this case, the location and geometric features of the virtual surface (cylindrical surface) remain, i.e. the cylindrical surface 120 remains configured as at the time of its selection.

According to the present invention, the system 100 provides switching selection between the one virtual surface 120 and another virtual surface 130, in this example between cylindrical surface 120 and the planar surface 130, as schematically represented in figure 2.

After switching (fig. 3), in order to determine how to move the device 10 on the selected surface (now the planar surface 130), the system 100 still calculates an intersection point 102 of the pointing ray 101 with the selected surface and set the intersection point 102 as a target destination 14 of the device 10.

Also in this case, the user knows that the selected surface has a predetermined shape and configuration (now a planar surface with a predetermined height from ground, as described in more detail later on), and therefore it is natural and intuitive for him/her to point the target destination 14 with the pointing ray 101 and then expecting the device 10 to reach this target destination 14 moving at said height.

Here below, and still with reference to fig. 1 to 3, it is described how the selected surface is configured, i.e. how its location and geometric features are set by the system 100, for convenience only still relating to the cylindrical surface 120 and planar surfaces. In particular, in figure 1, the device 10 is at a position 11 when the cylindrical surface 120 is selected, for instance after a movement on the planar surface 130, previously selected. The system 100 configures the cylindrical surface 120 in the 3D space based on the position 11 of the device 10. In other words, configuration is performed at run time.

Accordingly, a virtual surface 120, for instance the cylindrical surface 120, may be configured differently, for instance with a different radius, in occasion of difference selections thereof. In the example of figure 1, the radius is the distance between the position 2 of user 1 and the position of device 10, preferably the horizontal distance 1 In with respect to a surface (floor or ground) on which the user 1 is positioned. Still in the example of figure 1, a centre of a base B of the cylindrical surface 120 is the position 2 of the user 1. The cylindrical surface may be constrained along its axis, for instance when movement has to be controlled indoor, or no set, meaning that movement is controlled upwards with no predetermined limits.

The configuration of the virtual surface given above is provided, however, only for exemplificative purpose, meaning that the system 100 may process different calculation to determine the location and geometric features of the virtual surface. More in particular, depending on a program stored in the system 100, different calculation may be provided to determine a virtual surface. These calculations are programmed taking into consideration the specific features of the virtual surface, for instance considering that in order to calculate a virtual sphere it is necessary to define the centre of the sphere and the radius thereof.

With reference to figure 2, the device 10 is at a position 13 when the planar surface 120 is selected, after movement on the cylindrical surface 120 from position 11 to position 12. The planar surface is selected when the device 10 is at position 12. Here, the system 100 configures the planar surface 130 in the 3D space based on the position 12 of the device 10. In particular, the planar surface is configured at run time, taking into consideration the height of the device 10 at selection time, with respect to the ground.

In this example, the planar surface is parallel to the ground. However, according to different embodiments of the invention, the planar surface 130 may be at a predetermined inclination with respect to the ground. The inclination may be set in the system 100 and determined differently, according to the use.

Moreover, the planar surface is schematically delimited by continues lines in figure 1-4 for convenience only. Indeed, according to different embodiments of the invention, the planar surface may be constrained, i.e. limiting movement of the device within a predetermined range, or infinite.

With reference to figure 3, it is schematically represented movement of the device 10 on the planar surface 130, from the starting point 13 to the end point 14.

The example given above are not limiting the scope of the invention and various modification may be considered and covered by the claims.

For instance, the system 100 may configure two virtual surfaces to move the device 10, the virtual surfaces being a first planar surface and a second planar surface, parallel one to the other, placed at a predetermined distance. The device 10 is moved from the first planar surface to the second planar surface when selection is switched. Also according to this embodiment of the invention, the position of the device 10 may be used to determine the configuration of the first planar surface; for instance, in case the device 10 is a drone 10, it may be set at a predetermined height from ground, i.e. at a starting flying position, when turned on. The starting flight position is used to configure the first planar surface, for instance as a planar surface parallel to the ground and passing through the starting flight position. The second planar surface is set at a predetermined distance from the planar surface by the system 100, and the drone 10 moves, for instance perpendicularly between the first and second planar surfaces. In other words, according to this embodiment, the configuration of the second planar surface is not determined based on the position of the device 10 at selection time but it is set with respect to the configuration of the first planar virtual surface.

Further embodiments are covered by the method of the present invention. For instance, the another virtual surface 130 may be calculated based on a surface of an object, for instance a wall or a ground shape, and is parallel to the surface of the object, at a predetermined distance therefrom. For instance, the object may be a hill having an irregular profile comprising reliefs and valleys and the another virtual surfaces allows a drone 10 to be controlled flying at a predetermined distance from such reliefs and valleys.

With reference to figure 4, the another virtual surface is a plane perpendicular to the ground. In such an example, the configuration of the perpendicular plane is determined based on the position 14 of device 10 reached along the virtual surface of fig. 3 previously selected, according to the controls already explained above.

According to an aspect of the present invention, the another virtual surface 130 may also be calculated based on the position 2 of the human 1 (user) for instance at a predetermined distance from the user.

The system 100 is provided with an interface for switching selection between the one virtual surface 120 and the another virtual surface 130, for instance a command device including a push button which is kept pressed for selecting the one virtual surface and released for selecting the another virtual surface, a voice command, or a gesture. In this respect, it is more natural to keep pressed a push button for moving the device on a virtual surface and releasing it for switching to another virtual surface instead of pushing the button each time the switch is desired.

However, different hardware may be used to implement the method of the invention.

In the description below, a formal model and a corresponding implementation thereof is described according to the method of the present invention.

In the model, it is assumed that the device and the user are localized in a common reference frame. In practice, the transformation between their relative frames may be estimated with collaborative visual-inertial SLAM methods (such as the ones disclosed in M. Karrer, P. Schmuck, and M. Chli, “CVI-SLAM — collaborative visual-inertial SLAM,” IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 2762-2769, 2018.), which however requires instrumenting the operator) or an ad-hoc technique (such as the one disclosed in B. Gromov, L. Gambardella, and A. Giusti, “Robot identification and localization with pointing gestures,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Oct. 2018). The user indicates a target point pt by pointing. The pt lies on a pointing ray r. To identify r, a simplified version of the head-finger model is assumed (in particular the one referred by eye-finger ray cast method by Mayer et al, i.e. S. Mayer, V. Schwind, R. Schweigert, and N. Henze, “The Effect of Offset Correction and Cursor on Mid-Air Pointing in Real and Virtual Environments,” Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018.). The head- finger model defines r as the half- line originating at the operator’s dominant eye at point p₀ and passing through the tip of the pointing finger, located at the end of a straight-arm with orientation co₀ , which is measured by a wrist-mounted IMU.

A further assumption may be made in that eye and shoulder are vertically aligned (and that shoulder height, shoulder- finger length, shoulder-eye distance are known); accordingly, r may be reconstructed as r = r(co₀ ; p₀). When the origin p₀ of the ray is fixed, the user has two degrees of freedom (the arm orientation) to move r and point to a different target. pt is defined at an intersection of the pointing ray r with a virtual surface S pt = r P S, i.e., the operator, by moving its arm, moves a univocal, well-defined 3D target point on a two dimensional surface.

Intersecting the pointing ray with a horizontal plane is effective to define target positions for a ground robot or a quadrotor constrained to fly at a fixed height, for instance about 30 cm. (effectiveness of such a definition of target positions is given in B. Gromov, G. Abbate, L. Gambardella, and A. Giusti, “Proximity human-robot interaction using pointing gestures and a wrist-mounted IMU,” in 2019 IEEE International Conference on Robotics and Automation (ICRA), May 2019).

According to the method of the present invention, to control the drone or quadrotor in 3D space, the user may switch between different virtual surfaces or workspace surfaces, for instance the ones of figures 1 to 4.

In the model, the virtual surfaces may be defined as follows.

For exemplificative purpose only, three possible shapes for virtual surfaces are given below, i.e. Horizontal plane, Cylinder and Sphere. Each shape is defined as a one parameter family of surfaces: when the user switches to the shape, the free parameter is set in such a way that the virtual surface passes through the current position probot of the device or robot. a) Horizontal plane Sh-piane : with a distance from the ground plane as the free parameter. This virtual surface serves a similar role to the ground plane that humans have a life-long experience with: it is a natural choice for indicating locations and an intuitive tool to control robot’s position along the pointed direction on that plane. b) Cylinder S_cyiinder: with a vertical axis passing through the user’s head (p₀); the cylinder radius is the free parameter. This option allows the operator to control the robot’s vertical position without limitations, never affecting the horizontal distance to the operator. c) Sphere S_sphere : centered at the user’s head (p₀); the sphere radius is the free parameter. Operating in this virtual surface roughly corresponds to the user holding a rod in their hand, with the robot affixed at its extremity. To achieve intuitive interaction, an operator should always have a clear idea of the virtual surface the robot is operating in; if the virtual surface shape is known, the robot position itself uniquely defines the virtual surface S. E.g., if the virtual surface is Sh-piane, the user can expect that the robot will keep its current vertical position; if the the virtual surface is S_cyiinder, one can easily visualize the user-centered cylinder passing through the robot.

Still within the example given above, the switch among virtual surface is implemented as follows.

To minimize the complexity of the method, it is here below assumed that the user is limited to toggle between two possible virtual surfaces; properly choosing these virtual surfaces allows one to reach any 3D position. Among the possible choices, <S_cyiinder , S_Sphere> pair is preferably discarded as it prevents control of the robot horizontal position independently from height. For example, if the initial position is close to the user’s feet, reaching a point a few meters away can only be executed through a convoluted operation: choosing the cylindrical virtual surface, raising the robot to the corresponding height, then switching to a spherical virtual surface and lowering the robot again while it gains distance.

Similarly, <Sh-_Piane , S_sphere> does not allow independent control of the robot’s vertical position, which is something one would expect. Reaching a point at a considerable height in this model requires one to fly the robot farther than necessary using the plane virtual surface, then switch to sphere virtual surface to gain height while reducing distance.

For the reasons above, in the example of implementation given below, the user is allowed to toggle only between Sh-_Piane and S_cyiinder virtual surfaces.

• Gesture sensing

In the example, an inexpensive wearable IMU (such as Mbientlab MetaWearR, Wearable technology for healthcare. Mbientlab official web-page https://mbientlab.com/. [Online; accessed: 2018-09-13]) that has a form-factor of a wrist smartwatch. The wearable device is equipped with a 3 -DoF accelerometer, 3 -DoF gyroscope, and 3 -DoF magnetometer.

The firmware onboard the wearable device runs the necessary sensor fusion algorithms in real time and outputs an accurate estimation of the device’s absolute 3D-orientation in an arbitrary fixed reference frame whose z-axis points up. The wearable device also features a micro switch button, an LED light, and a vibration motor. The data is streamed to the host PC with approximately 50 Hz rate via Bluetooth Low-Energy (Bluetooth 4.0) link.

The acquired orientation is used within the head-finger pointing model (described above) to recover r, which is then intersected with the active virtual surface s_a to define the pointed-to location.

• Switching virtual surfaces

S_c is defined as the primary virtual surface, i.e. the one that is active by default; S_h is considered a secondary shape, i.e. one that user can switch to upon request.

The operator switches between the two shapes using a single spring-loaded trigger button on a joystick (for instance Logitech F710); the other joystick buttons and controls are otherwise ignored, for the purpose of the experiment. When the trigger is in its default position (not pressed), the primary virtual surface is selected; keeping the button pressed the secondary virtual surfaces is selected.

This specific choice is very useful for usability, for two reasons. First, mapping the explicit state of the button (the user must consciously keep it pressed) to the chosen virtual surface — an important state of the system — ensures that the user is kept aware of the system state. Contrary, using a discrete button with an implicit state (the one the user cannot directly observe) to toggle between the shapes would lack this property.

Second, the primary virtual surface is considered to be the safest (the user is unable to change the distance to the robot) and can always be used. Instead, the system may refuse to switch to the secondary virtual surface if such a switch is considered unsafe; this occurs when the pointing ray is almost parallel to the horizontal plane (e.g. the drone is flying at the height of user’s eyes); then, small changes in arm elevation would result in very large displacements of the pointed location. Therefore, we prevent switching to the secondary virtual surfaces if the elevation angle of the drone with respect to the user’s head is within ±5° from the horizontal direction. Whenever a requested switch to the secondary virtual surface is refused, the joystick vibrates to make sure the user is notified.

• Experiments with drone control

Experiments with a miniature quadrotor Bitcraze Crazyflie 2.0 (Bitcraze, “The crazyflie nano quadcopter,” https://www.bitcraze.io.) have been used to validate presented invention. Superior results, as compared to conventional control interfaces, have been achieved in precision of control, ease of learning by the user and ease in controlling once learned, in particular for 3D control of a drone by switching between different virtual surfaces to intersect the pointing ray. The system is particularly intuitive, equipped with a user friendly interface to control a drone in 3D space using pointing, a control mode which has only two degrees of freedom. In particular, the combination of a cylindrical and a planer shape, allow to precisely control a quadrotor that should fly across targets located at different altitudes.

Claims

1. Method for controlling a device (10) by a human (1), wherein a position (12) of the device (10) and a position (2) of the human (1) is known in a system (100), and wherein the system (100) calculates a pointing ray (101) based on a posture of the human (1), characterized by the fact that

-the system (100) defines at least two virtual surfaces (120, 130)

-the system supports selection of one virtual surface (120) of said at least two virtual surfaces (120, 130) for movement of the device (10) on said one virtual surface (120) or selection of another virtual surface (130) of said at least two virtual surfaces (120, 130) for movement of the device (10) on said another virtual surface (130) wherein

-a configuration (120P) in a 3D space of the selected surface among said at least two virtual surfaces (120, 130) is calculated by the system (100) based on at least said position (12) of the device (10) in the 3D space when said surface selection is made, wherein

-the system (100) further calculates an intersection point (102) of the pointing ray (101) with the selected surface among said at least two virtual surfaces (120, 130) and set the intersection point (102) as a target destination (13) of the device (10), and wherein

-the system (100) provides switching selection between said one virtual surface (120) and said another virtual surface (130) of said at least two virtual surfaces (120, 130).

2. Method according to claim 1 wherein, when selection is switched between said one virtual surface (120) to said another virtual surface (130) of said at least two virtual surfaces (120, 130) to select said another virtual surface (130), the configuration (130P) in the 3D space of said another virtual surface (130) is calculated by the system (100) based on at least the position reached by the device (10) in the 3D space on said one virtual surface (120).

3. Method according to claim 1 wherein said one virtual surface (120) of the at least two virtual surfaces (120, 130) is a plane.

4. Method according to claim 1 wherein the position (12) of the device (10) is at a predetermined height from the ground.

5. Method according to claims 1 to 3 wherein said one virtual surface (120) of the at least two virtual surfaces (120, 130) has predetermined orientation with respect to the ground, preferably parallel to the ground.

6. Method according to claim 3 wherein said another virtual surface (130) of said at least two virtual surfaces (120, 130) is a plane parallel to said one virtual surface (120) and a position of said another virtual surface (130) is at a predetermined distance with respect to said one virtual surface (120) , wherein the device (10) is moved from said one virtual surface to said another virtual surface when said selection is switched.

7. Method according to claim 1 wherein said another virtual surface (130) of said at least two virtual surfaces (120, 130) is calculated based on a surface of an object, for instance a wall or a ground shape, and said another virtual surface is a surface parallel to the surface of the object, at a predetermined distance from the surface of the object.

8. Method according to claim 1 wherein said another virtual surface (130) of the at least two virtual surfaces (120, 130) is a plane perpendicular to the ground.

9. Method according to claim 1 wherein said another virtual surface (130) of the at least two virtual surfaces (120, 130) is calculated based on the position (2) of the human

(I)·

10. Method according to claim 9 wherein said another virtual surface (130) is at a predetermined distance from the human (1).

11. Method according to claim 10 wherein said one virtual surface (120) is a cylindrical surface with an axis perpendicular to the ground and passing through the human (1) or a spherical surface with an origin on or in the human body, and the radius of the cylinder being calculated based on the position (12) of the device (10) with respect to the human or based on the position of the device (10) with respect to the human when said selection is switched from said one another virtual surface (130) to said one virtual surface (120) of said at least two virtual surfaces (120, 130) to activate said one virtual surface (120)

12. Method according to claim 1 wherein the device is a drone.

13. Method according to claim 1 wherein the system is computer implemented system.

14. Method according to claim 1 wherein the human is provided with an interface for switching selection between said one virtual surface (120) and said another virtual surface (130), for instance a command device including a push button which is kept pressed for selecting said one virtual surface and released for selecting said another virtual surface, a voice command, or a gesture.

15. System for controlling a device (10) by a human (1), the system including electronic means to:

-determine a position (12) of the device (10) and a position (2) of the human (1), and -calculate a pointing ray (101) based on a posture of the human (1), characterized by the fact that

-the system (100) further includes processing means to define at least two virtual surfaces (120, 130) and to select one virtual surface (120) of said at least two virtual surfaces (120, 130) for movement of the device (10) on said one virtual surface (120) or another virtual surface (130) of said at least two virtual surfaces (120, 130) for movement of the device (10) on said another virtual surface (130) wherein

-the processing means are programmed to calculate configuration (120P) in a 3D space of the selected surface among said at least two virtual surfaces (120, 130) based on at least said position (12) of the device (10) in the 3D space when said surface selection is made, wherein

-said processing means further calculates an intersection point (102) of the pointing ray (101) with the selected surface among said at least two virtual surfaces (120, 130) and set the intersection point (102) as a target destination (13) of the device (10), and wherein

-the system (100) further includes a switch for switching selection between said one virtual surface (120) and said another virtual surface (120) of said at least two virtual surfaces (120, 130).