FR3074325A1 - HANDLING VIRTUAL OBJECTS IN AN IMMERSIVE ENVIRONMENT - Google Patents

Abstract

A method of manipulating a virtual object (18) in an immersive environment (100) comprising: an immersive room (11) equipped with a detection system (12s), a projection system (14s) of images on the walls (13) of the room, by a user (10) carrying: - a user's vision tracking system (16) - a user's gesture tracking system (19), characterized in that that the method comprises a gripping phase defining the gripping of the virtual object (18), which comprises a gestuality with both hands of the user.

Description

Technical field of the invention

The present invention relates to an immersive environment for study elements in virtual reality.

The present invention relates more particularly to a method for manipulating an object in an immersive virtual environment.

State of the art

The current technological advancement in virtual reality makes it possible to project an operator within a virtual universe in which various contents are displayed, for example a 3D or three-dimensional object to be studied. It is not only possible to view different 3D content but also to interact with it in real time, which opens the way to many applications. The immersive digital project review, for example, materializes through the use of virtual reality as a control tool and a discussion medium between the different interlocutors of a project.

In the automotive field, digital models of vehicles can then be used by different interlocutors including designers, engineers or ergonomists who can interact during immersive meetings centered on the digital model, said model serving as a communication and exchange medium of points of view between the different trades. Said digital model also makes it possible to verify that the vehicle model does indeed meet criteria of architecture, ergonomics, perceived quality or even livability among others.

While physical models have been used extensively so far, the current objective is to drastically reduce their use because they are expensive and not very ecological. Conventionally, we order a physical model that corresponds to the state of the prototype at a given time. After significant delays, the model is delivered but it may already be out of date. On the other hand, a digital model is available more quickly, in a more up-to-date version that can be modified in real time, which allows greater reactivity by all the players and demonstrates the interest of the digital model.

Another advantage is that unlike the usual physical models of vehicles which are often on a quarter scale, virtual models appear on scale 1, which means that the operator can put himself in the shoes of an end user. and enter the cabin.

The visualization of motor vehicle architecture in a virtual environment is therefore a major issue for car manufacturers because it allows to accelerate the design process. We generally use an immersive room that provides excellent visual rendering and in which we deploy two technologies for depth perception: stereovision which exploits the effect of binocular disparity and the capture and tracking of movements or " tracking ”in English which allows the parallax of the movement.

In general, the objects in an immersive room are placed or circulate substantially on a plane and it is not possible to fully understand all the shape characteristics of the virtual object that we are studying. For example, it is impossible to manipulate these objects in order to see the underside of said objects. In the field of motor vehicles, the users who may be designators, designers or engineers can see and appreciate the top of the vehicle at least all that is up to their vision with naturally all the elements of the side walls of said vehicle. However, these users cannot comfortably see decorative elements arranged on the roof of the vehicle or the elements below the vehicle. They cannot have a complete vision of the vehicle when they have maps and digital data.

There is therefore a need to manipulate objects in an immersive virtual environment, in particular for bulky objects such as a vehicle body or a vehicle itself.

Publication US 2009/0177452-A1 provides a system for manipulating an object in a virtual environment, which object is tracked by detectors. Said object is a tool for an operation for example.

One problem is tracking by detecting the tool and not the hand gestures, which significantly reduces the range of experience offered by this publication. In addition, the installation of the tool is not taken into account then it is the tool which is followed and not the hand itself.

The object of the invention is to remedy these problems and one of the objects of the invention is a method of manipulating an object in an immersive environment to authorize the grasping, handling and placing of an object.

Presentation of the invention

The present invention relates more particularly to a method of manipulating a virtual object in an immersive environment comprising:

-an immersive room equipped with

-a detection system,

-an image projection system on the walls of the room, by a user carrying:

-a user vision tracking system

a system for monitoring the gestures of the user, characterized in that the method comprises a gripping phase defining the gripping of the virtual object, which includes a predetermined gesture with the two hands of the user.

Advantageously, the method of manipulating a virtual object makes it possible to take and move a virtual object in an immersive virtual environment such as the CAVE comprising an immersive room with its device for projecting images onto the walls of said room comprising projectors, its tracking device comprising detection cameras, into which the user has entered while wearing a vision tracking system, for example a pair of glasses equipped with a tracking element which may be a sphere arranged in the middle a pair of glasses, and a gesture tracking system, including fingers to each of the user's two hands. According to the invention, the recognition of the grip of the virtual object is consecutive to the detection of a predetermined gesture with both hands of the user to avoid malfunctions following errors in gestures.

Advantageously, the two-handed grip of the virtual object also makes it possible without difficulty to print translational or rotational movements to said object along different axes. Indeed, it is then easier to detect the relative positions between the two hands and then to calculate the movements of the virtual object.

According to other features of the invention,

- the user's gesture tracking system includes tracking elements carried by the user.

Advantageously, the user’s gesture tracking system includes tracking elements carried by the user. Therefore, the spatial positions of the fingers of each of the hands are associated with the spatial positions of the tracking elements carried. A calibration phase can take place to specify the pinching movements of the fingers.

- the tracking elements worn are in a list comprising tracking gloves comprising at least two fingers capable of being tracked, tracking rings adapted to surround the user's finger.

Advantageously, the tracking elements worn in each hand by the user can a pair of tracking gloves formed by at least two tracking tips detected by the tracking system of the immersive room.

Said elements can also be a ring with radial arms holding a ball identified and followed by the tracking system of the immersive room.

There are many elements of finger tracking and technology may offer other less bulky products.

-the gripping phase includes the detection of pinching of two fingers of one hand.

Advantageously, the gripping phase comprises the detection by the tracking system of the immersive room of the pinching of two fingers of the same hand. Said hand can adjoin the virtual object, that is to say that the distance between said hand and the virtual object is less than a distance threshold. For example, said hand may be in contact with the virtual object to reduce detection errors. Indeed, the spatial coordinates of the hand can be deduced from the spatial coordinates of the fingers and then compared with the spatial coordinates of the virtual object. The hand must therefore be close to the virtual object or in contact with it to specify the will to grasp the virtual object, then the fingers must come to the pinch in contact with each other.

-the gripping phase includes a validation step comprising the detection of the pinching of fingers of each of the two hands which adjoin the virtual object.

Advantageously, the gripping step comprises a step of validating the gripping with the detection of the pinching of the two fingers of each of the two hands. Each of the hands adjoins or is close to the virtual object. In this way, the difference between the two hands is calculated and taken as a reference value for the movements to come.

-the gripping phase of the object includes a set of symbolic gestures, each of the symbolic gestures is associated with a particular movement of the virtual object.

Advantageously, the gripping phase comprises a set of symbolic gestures, each of these symbolic gestures is associated with a particular movement of the virtual object. Symbolic gestures are advantageously intuitive

- the gripping phase allows a vision of enlargement of the virtual object by a symbolic gesture of magnification.

The gripping phase also advantageously allows an enlargement of the virtual object by a particular gesture, simple and intuitive with the two hands, each hand having the two fingers pinched.

-the symbolic gesture of magnification is in a list including a separation between the two hands of the user.

Advantageously, the gesture consisting in spreading the hands and arms is intuitive to enlarge the vision of the virtual object, each of the hands having the two fingers in pinch. This gesture is analogous to that symbolic on current laptops.

-the gripping phase is stopped upon detection of the loosening of two fingers of one hand.

Advantageously, the loosening of the two fingers of one hand is detected by the monitoring system of the immersive room makes it possible to stop the gripping phase. The gestures associated with stopping the grip are therefore very simple.

- the manipulation stop phase leaves the virtual object immobile in its last spatial position preceding said stop phase.

Advantageously, the manipulation stop phase leaves the virtual object immobile in its last position preceding said stop phase. Thus, it is easy to move the virtual object in the desired directions, then to immobilize it in order to visually perceive all the aspects of said virtual object.

Brief description of the figures

Other characteristics and advantages of the invention will appear on reading the following description of particular embodiments of the invention given by way of nonlimiting examples and represented in the appended drawings, in which:

FIG. 1 is a schematic view of the user in an immersive virtual environment with a virtual object.

FIG. 2 is a schematic view of the hand with the user's gesture tracking system.

FIG. 3 is a schematic view of the manipulation of the virtual object.

FIG. 4 is a diagram of the grip of the virtual object.

FIG. 5 is a view of a flow diagram representing the manipulation process.

Detailed description of the figures

In the following description, identical reference numerals designate identical parts or having similar functions.

The term “3D” is understood to mean the three-dimensional spatial character.

FIG. 1 represents an immersive virtual environment 100 in which a user is immersed. Said immersive virtual environment a "CAVE ©" for "Cave Automatic Virtual Environment" in English. This immersive environment is formed by an immersive room 11 comprising three vertical walls 13 joined arranged between a ceiling and a floor and equipped with cameras and position sensors 12 of a user here the driver at the corners of the ceiling and / or floor said room and projectors 14 for diffusing on the various walls 13 of said room 11 images of the environment recorded in a memory of a control unit 15 in order to realistically recreate the environment of the user. Said sensors 12 form a detection system 12s and the projectors 14 form a projection system 14s of the room, said systems are connected to said control unit 15. The user 10 is introduced into the immersive room.

The control unit 15 controls the continuous projection of environment images to form the virtual environment of the user 10. The virtual environment of the user is capable of being modified according to the navigation of said user as well as by displacements from the point of view of the user. To be able to define said point of view, the user can be provided with a vision tracking system 16 which is here a pair of glasses comprising a tracking ball 17 covered for example with silver foil, said tracking ball being capable to be spotted by the sensors 16 of the immersive room. The spatial position of the tracking ball 17 can then be easily detected as well as the direction of the user's vision.

Movements in the head space are thus also detected by the sensors 12 of the immersive room 11 in order to be able to define the point of view as well as the field of vision of the user 10. The detection information by said sensors 12 are then preferably transmitted in real time by said sensors to the control unit 15 which is capable of performing calculations to define the virtual environment 35 of the user and of controlling the projections of the images by the projectors 14. On can see the virtual environment 35 for example in FIGS. 3 and 4.

In the immersive room 11 is placed a virtual object 18 which is here a virtual vehicle 18v which can be a 3D image of a vehicle or of a study model as in our example. This virtual object 18 "3D" is obtained in the immersive room 11 by the projection of video streams on the different walls 13 of the room.

A session for examining the virtual object 18 can then be started simply.

According to the invention, the user has a method 200 for manipulating the virtual object 18 which makes it possible to validate the gripping of the virtual object 18 and then the manipulation of said object. According to FIG. 5, said method 200 comprises:

-a gripping phase 210 of the virtual object which comprises .a validation step 215 of the gripping,

-a manipulation phase 220,

-a phase of stopping the manipulation 230.

As can be seen in FIGS. 1 and 2, the user also carries with his two hands a gesture tracking system 19. This gesture tracking system 19 preferably comprises a pair of tracking gloves 20. The tracking glove comprises end caps 21 connected to a unit 22 fixed on the back of the hand. The end pieces 21 preferably are at least two in number and they are adapted to accommodate the ends of the fingers of the user's hand; according to the embodiment presented, the fingers include at least the thumb. Thus the user 10 can sketch a pinching movement 23 of the fingers which can be detected by the detection system 12s of the immersive room 11; pinching fingers 23 consists of bringing the fingers together until they contact the free ends of said fingers, here the thumb and another finger each carrying a follow-up end piece 21.

As illustrated in FIGS. 3 and 4, the method for manipulating a virtual object according to the invention comprises a gripping phase 210 of the virtual object. Said gripping phase involves the detection of several different gestures on the part of the user.

Said gripping phase comprises in particular a step 215 of validation of the gripping of the virtual object 18.

According to one embodiment, this validation step 215 comprises a criterion which is the bringing together of the hands 30 of the user 10 towards a contact with the virtual object 18, said hands 30 then adjoining the virtual object 18. By adjoin the fact that the hands come close to the virtual object until a possible contact. The spatial positions obtained by hand detection are compared with the known spatial positions of the virtual object 18, so contact distances between the hands and the virtual object can be easily calculated. Said distances must be less than a distance threshold to confirm the user's desire for the virtual object 18 to be grasped 10. The distance threshold can be of the order of 1 to 3 m.

The validation step 215 of the gripping includes another criterion which is the detection of the pinching of the fingers of the hand and more precisely of the two hands. Then, the user 10 can manipulate the virtual object 18. In fact, the manipulation of the virtual object 18 requires at least two gripping points to then allow different possible movements of the object. According to FIGS. 3 and 4, the user has thus extended his arms until the hands come into contact with the virtual object and pinched the fingers of his two hands. The validation of the grip of the virtual object is effective. The user can then manipulate said virtual object 18. According to FIG. 4, the user 10 raises his hands while keeping his fingers 30 pinched. The virtual object 18 is then moved in a movement from the floor to the ceiling of the immersive room. The user can manipulate the virtual object with the movements of his hands while keeping his fingers 30 in the pinch position 23.

Indeed, the manipulation phase 220 is stopped on detection of the defect of a criterion of the validation step 215 of the gripping phase 210. In the embodiment presented, the manipulation is stopped on detection of non-pinching or loosening the fingers of at least one hand. In the manipulation stop phase 230, the virtual object remains stationary in the last spatial position preceding the stop. The user can then check or validate by viewing certain aspects of the virtual object 18 out of reach when said object is in its normal or initial position on the floor of the immersive room. For example, he can visually examine the underside of the vehicle.

During the gripping phase, the user can perform various gestures which are recognized and associated with modifications to the virtual object.

Among the recognized gestures, the enlargement of the virtual object or its shrinking can be obtained respectively by spreading or bringing the hands 30 together from the initial separation of the hands. The term “initial hand gap” is understood to mean the difference calculated between the hands following the detection of the criteria of the validation step 215 of the gripping phase 210, that is to say the hands 30 of the user adjoining the virtual object 18 and a pinching of the fingers of each of the hands. In this simple way, the user can perceive all the desired details of the different aspects of the virtual object.

The user can move the virtual object in different ways. For example, he can tilt the virtual object by tilting the axis connecting two hands. In another example, it can rotate the virtual object around an axis by moving one hand 30 around the other. In general, the movements of the manipulated virtual object follow the relative movements of the two hands of the user, it being understood that this user does not release the pinching of the fingers in each hand in the sequence of movements. Said relative movements as described above are in a set of symbolic gestures recognized by the manipulation process, each symbolic gesture is then associated with a particular movement of the virtual object. It should be noted that all symbolic gestures are intuitive and do not require any learning effort.

According to the invention, the manipulation method also authorizes the recording of notes on the virtual object 18 manipulated. The movement of the fingers inserted into the tips of the gesture tracking system is monitored by the 12s detection system in the immersive room, leaving the user the possibility of writing annotations or markings of areas of the virtual object. These annotations or markings are kept and associated with the examination session of the virtual object.

According to the invention, the manipulation method also authorizes, during the examination session, measurements of particular dimensions of the virtual object. Indeed, the method includes distance measurement tools. The user can thus point the two ends of a segment to obtain the length of said segment. The measured value can then be displayed as a 3D annotation with the virtual object.

The objective has been achieved: the manipulation process makes it easy to manipulate a virtual object with very intuitive gestures. The manipulation of said virtual object allows a detailed perception of the virtual object and of the interesting and necessary functionalities during a session of examination of a virtual object.

As it goes without saying, the invention is not limited to the sole embodiments of this socket, described above by way of examples, on the contrary it embraces all variants.

For example, the validation step 215 can take into account the relative positions of the user's vision with both hands to confirm the user's desire for the virtual object to be grasped.

For example, too, the position of the hands in relation to the virtual object may be irrelevant to allow other movements of the virtual object.

Claims (12)

1. Method for manipulating (200) a virtual object (18) in an immersive environment (100) comprising: -an immersive room (11) equipped with -a detection system (12s), -a projection system (14s) of images on the walls (13) of the room, by a user (10) carrying: -a vision tracking system (16) for the user a system for monitoring the gestures (19) of the user, characterized in that the method comprises a gripping phase defining the gripping of the virtual object (18), which comprises the detection of the relative positions between the two hands of the user for manipulating the virtual object with both hands (30) of the user. 2. Method (200) according to claim 1, characterized in that the gesture tracking system (19) of the user comprises tracking elements (20) carried in the hands (30) by the user. 3. Method (200) according to claim 2, characterized in that the tracking elements worn are in a list comprising gloves (20) for tracking comprising at least two fingers capable of being tracked, tracking rings adapted to surround the user's fingers. 4. Method (200) according to any one of claims 1 to 3, characterized in that the gripping phase comprises the detection of pinching (23) of two fingers of one hand (30). 5. Method (200) according to claim 4, characterized in that the gripping phase (210) comprises a validation step (215) comprising the detection of the pinching (23) of fingers of each of the two hands (30) adjoining the virtual object. 6. Method (200) according to any one of claims 1 to 5, characterized in that the gripping phase (210) of the virtual object (18) comprises a set of symbolic gestures, each of the symbolic gestures is associated with a particular movement of the virtual object (18). 7. Method (200) according to any one of claims 1 to 6, characterized in that the gripping phase (210) allows a vision of enlargement of the virtual object by a symbolic gesture of magnification. 8. Method (200) according to claim 7, characterized in that the symbolic gesture of magnification is in a list comprising spacing of the two hands (30) of the user, each of the hands keeping the fingers pinched. 9. Method (200) according to any one of claims 4 to 8, characterized in that the gripping phase is stopped upon detection of the loosening of two fingers of one hand. 10. Method (200) according to claim 9, characterized in that the stop phase (230) of manipulation leaves the virtual object (18) immobile in its last spatial position preceding said stop phase. 11. Method (200) according to any one of claims 1 to 10, characterized in that the method allows the writing of annotations on the virtual object (18). 12. Method (200) according to any one of claims 1 to 10, characterized in that the method allows the measurement of length of segments of the virtual object (18).