WO2019149625A1 - Method for superimposing a virtual graphical object on a real environment by means of an ar display device for augmented reality, and motor vehicle - Google Patents

Abstract

The invention relates to a method for superimposing a virtual graphical object (14) on a real environment (16) in an AR display device (12) for AR, augment reality, wherein: - a display position (15) of the graphical object (14) on the AR display device (12) is updated by a control device (13) at predefined output times following one another; - the display position (15) is set on the basis of observation values (22), which each describe the real environment (16) and/or a position of the AR display device (12) relative to the environment (16). According to the invention, the observation values (22) are provided with respective time stamps (23), each time stamp (23) indicating an observation time at which the associated observation value (22) was captured, and, on the basis of the time stamp (23) of at least one of the observation values (22), at least one future observation value (22') and/or the next, future display position (15') being extrapolated for the next, future output time (28).

Description

 A method for superimposing a virtual graphic object on a real environment by means of an augmented reality AR display device and motor vehicle

DESCRIPTION:

The invention relates to a method for superposing a virtual graphic object over an indication of a real environment in an AR display device for augmented reality. The invention also includes a motor vehicle with an AR display device, wherein the motor vehicle the method according to the invention can perform.

Currently available augmented reality systems are each based on a closed, highly-integrated computer system in which the detection sensor system, the control device with the processing processor and the actual AR display device (display) are directly connected to one another, that is to say a single, closed device , The transmission paths and therefore the transmission times from the detection sensor system to the AR display device are correspondingly short. The development target for an AR display device, for example an AR head-up display, is the time-accurate or precise overlay of the displayed augmenting graphical objects over the real environment to be augmented with the AR display. For a temporal and / or spatial deviation of the display position of a graphical object with respect to an associated environment object in the real environment can lead to a user being irritated by the use of the AR display device. Ensuring the exact superposition of a graphical object over a real environment object detected in the environment by means of the AR display device requires that the acquired and / or fused sensor values be ready in time in the control device for the display position calculated for a graphical object to be up-to-date means the current position of the environment object visible in the real environment.

If one now wants to provide an AR display device for AR in a motor vehicle, then there is the problem that therein the Beobachtsein direction, which is used for generating observation values, such as sensor values, not usually close to the Steuerein direction for the AR display device can be arranged, but observation values of an observation device must be transmitted over a data network. If fused sensor data are generated in addition, this usually also takes place in a separate control unit. Thus, in the implementation of an AR display in a motor vehicle, the need may arise for a distributed system comprising at least one observation device and / or one Control unit for a sensor data function and a control device for the AR display device must be implemented, which requires the transmission of Be obachtungswerte over a data network and thus brings the corre sponding transmission latency From US 2015/0268473 A1 is known, an AR display device with to operate a different smartphone via a cable.

From CN 105913371 Al is known impose a common synchronous system clock for an AR representation of the participating devices th. However, this requires a complex redesign of all devices in order to realize this common system clock.

An example of an AR display device is known from US 2015062000 A1. The invention is based on the object of positioning an illustrated virtual graphic object precisely with respect to the real environment represented in the AR display device in the case of an AR display device.

The object is achieved by the subject matters of the independent patent claims. Advantageous embodiments of the invention will become apparent from the dependent claims, the following description and the figures

The invention provides a method for superimposing a virtual graphical object over a real environment by means of an AR display device for augmented reality AR. For example, the real environment may be displayed in the AR display as a camera image. The AR display device may also include a transparent pane through which the real environment is visible directly or through a mirror. The AR display device may, for example, be configured as a head-up display or AR glasses. The AR display device can also be configured, for example, as a screen. In the method, in the manner known for augmented reality, a control device updates a display position of said virtual graphic object on the AR display device at successive predetermined output times (ie, displaces the object on the AR display device), for which purpose the display position is determined or set on the basis of observation values. These observation values respectively describe the real environment, that is, for example, at least one real environment object in the environment, and / or a relative position of the AR display device relative to the environment. In other words, a display position for the virtual graphic object can be calculated based on the observation values in which the graphical object then has a predetermined relative position with respect to at least one environmental object that is visible in the real environment. For example, as a graphic object, a frame or arrow may be provided on the AR display device relative to a real environment object in a predetermined relative position (eg next to or overlying) to be positioned. Of course, this applies in particular to the perspective of the viewer of the AR display device.

Now, however, the environment object and / or the AR display device may be in motion. The calculation of a suitable display position therefore requires that the observation values are current, ie the real environment and / or the relative position of the AR display device with respect to the environment at the current time when the graphic object is displayed or displayed on the AR display device. is described. Otherwise, the graphical object is positioned based on outdated observation values at a display position, which may deviate from the desired, predetermined relative position, for example because the environment object has moved and / or the AR display device has been moved relative to the surroundings.

In order to achieve a precise positioning of the graphic object on the AR display device, it is provided according to the invention that the said observation values are provided with a respective time stamp. The respective time stamp indicates a respective observation time at which the respective associated observation value was detected. Each observation value therefore also includes a time value which indicates the observation time. The fact is now known that the respective next, future output time at which the display position of the graphic object on the AR display device has to be updated or newly displayed or set is known. So you know for which future output time you should calculate or determine the display position for the graphical object. Accordingly, at least one of the observation values of the at least one future observation value, that is to say an artificial observation value, and / or the next, future display position is half estimated or extrapolated or predicted on the basis of the respective time stamp. In other words, it is determined where at least one Environment object located in the environment and / or which Relativpositi on the AR display device with respect to the environment in the future, namely the next, future issue date will have. For this purpose, at least one future observation value is estimated. For example, based on a respective observation value for a speed and a position of an environment object at a given time (indicated by the respective timestamp) by the extrapolation or prediction, the movement of the object may be predicted and thus the position of the surrounding object to the next, future output. Time to be predicted or estimated. In addition or as an alternative to an estimated observation value, the next, future display position can also be extrapolated directly, so that a future observation value does not have to be explicitly extrapolated. The invention provides the advantage that a time offset between the respective observation time of the observation values on the one hand and the respective output time for the graphic object on the AR display device can be compensated for by the extrapolation. The invention also includes embodiments that provide additional benefits.

According to one embodiment, the respective output time (for the graphic object on the AR display device) is determined by a V-sync signal of the AR display device. Through this V-sync signal, a picture of the AR display device is cyclically started or triggered or started. The V sync signal indicates that the picture should start at the top or bottom of the screen. The V-sync signal is a cyclic or periodic signal so that the next future output time can be reliably determined or known. It is repeated at intervals T_output corresponding to the inverse of the frame rate (T_output = 1 / framerate). One embodiment provides that a respective timer unit of the at least one observation device and a timer unit of the control device of the AR display device are synchronized in time with one another by means of synchronization data. This timing may, for example, provide for setting an identical absolute or relative time in each timing unit. Synchronizing the time units results in the advantage that a time value, as may be indicated in a time stamp, has the same meaning in an observation device on the one hand and the control device on the other hand

According to one embodiment, the synchronization data used for this purpose are generated and / or transmitted by means of a Time Precision Protocol (TPP). The TPP can be implemented, for example, on the basis of the description according to IEEE 1588 and / or IEC 61588. The use of this protocol has the advantage that time values or times which differ at most by a known maximum value can be set in all said timer units. In particular, the maximum value is less than 1 / framerate. One embodiment provides that said observation values, together with their respective time stamps, are transmitted from the respective observation device via a data network and / or via a gateway device to the control device. The transmission thereby has a transmission latency. For example, a CAN bus (Controller Area Network), a Flexray bus or a Media Oriented Systems Transport (MOST) bus or an Ethernet network can be used as the data network. A gateway device can interconnect multiple network threads and transmit observation values between the network threads. This results in a processing time through the gateway device, which contributes to the transmission latency. As a result of the embodiment, there is the advantage that a spatial distance between at least one observation device on the one hand and the control device for the AR display device for AR on the other hand can be at least 50 centimeters, in particular more than 1 meter. Nevertheless, one can Display position for the graphic object on the AR display device can be determined precisely.

One embodiment provides that a position and / or a speed and / or an acceleration at least one environment object of the environment and / or the AR display device are used as the respective observation value for setting the future display position of the graphic object. On the basis of the said variables, a movement dynamics of the respective surrounding object and / or the AR display device can be extrapolated or predicted.

One embodiment provides that sensor values from at least one sensor and / or fused sensor values are used as observation values. Fused sensor values are observation values which are based on more than one sensor, that is to say on sensor values of different sensors and / or on sensor values acquired at different times. At least one sensor and / or a control unit for the sensor value fusion can therefore be used as the respective observation device. An embodiment provides that the said predication or extrapolation is performed by means of a Kalman filter. This has the advantage that in addition a measurement noise can be compensated.

The method according to the invention can be realized or provided in a device by which an AR display device for AR (AR display device) is operated.

In particular, it is provided here that the method is realized in a motor vehicle with an AR display device for AR. Accordingly, the invention also encompasses such a motor vehicle with an AR display device for AR and with a control device for the AR display device and with at least one observation device for generating observation values, each of which has a real environment of the motor vehicle and / or a relative position of the AR display device describe to the environment. An observation device can be realized, for example, by a sensor and / or a control unit for a sensor value fusion. In the motor vehicle according to the invention it is provided that this is set up to carry out an embodiment of the method according to the invention. For this purpose, the Steuerein device may have a processor device which is adapted to perform the control device of the respective steps of the method. For this purpose, the processor device can have at least one microprocessor and / or at least one microcontroller. The processor device may further comprise a program code adapted to execute, when executed by the processor device, the steps of the method concerning the control device. Each observation device can also have a processor device which is set up to carry out the steps of the method relating to the observation device, ie to supplement observation values with a time stamp. The processor device of each observation device can also provide at least one microprocessor and / or at least one microcontroller. The processor device of each observation device may have a program code that is configured to execute the steps of the method concerning the observation device when it is executed by the processor device of the observation device.

The motor vehicle according to the invention is preferably designed as a motor vehicle, in particular special as a passenger car or truck.

The invention also includes the combinations of the embodiments described Ausfüh

In the following an embodiment of the invention is described. This shows:

1 is a schematic representation of an embodiment of a Ge apparatus that an embodiment of the method according to the invention rens can perform, the device is in particular a motor vehicle; and

Figure 2 is a Flußschaudiagramm to an embodiment of the inventions to the invention process.

The exemplary embodiments explained below are preferred embodiments of the invention. In the Ausführungsbeispie len the described components of the embodiments each represent individual, independently of each other to be considered features of the invention, which each further independently develop the invention and thus individually or in any other than the combination shown to be considered part of the invention. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

FIG. 1 shows a system 10 for presenting augmented reality (extended reality). The system 10 may be installed, for example, in a motor vehicle 11. The device 10 can have an output device 12 for displaying the augmented reality, which can be, for example, a head-up display (HUD). For controlling a graphic content or output content of the output device 12, a Steuerein device 13 may be provided. By the control device 13, a virtu elles graphic object 14, which is displayed on the AR display device 12, are controlled. In particular, a display position 15 may be tuned to an environment object 17 located in an environment 16 of the system 10 such that a user of the system 10, viewing the AR display 12, the object 15 with respect to the ambient object 17 in a predetermined relative position, for example superimposed, sees. For this purpose, the system 10 determines the relative position between the system 10 and the environment object 17. For this purpose, at least one sensor 18 may be provided in the system 10. A sensor 18 may be, for example, a camera or a radar or an ultrasonic sensor or a radio receiving device for receiving position data of the Be environment object 17. A control unit 19 can receive sensor data 20 and thereby generate fused sensor data 21, that is, for example, combined or averaged sensor data 20. Control unit 19 can send sensor data 20 and / or fused sensor data 21 to respective control values 13 as respective observation values. The sensors

18 and the control unit 19 each represent an observation device B.

The observation values 22 are provided with a respective time stamp 23 of the observation time point to which the observation values 22 were generated, that is to say, for example, the sensor values 20 were measured. The transmission of the observation values 22 can take place via a data network 24. In this case, a transmission via a gate way device or a gateway 25 may be provided or necessary. Overall, the transmission over the data network 24 results in a latency or transmission duration. Thus, in the control device 13 the observation values 22 are present with a time offset between the two Observation time on the one hand and the output time of the graphi's object 14 at the respective display position 15th

Nevertheless, the system 10 is prevented from causing or causing the temporal offset to cause the graphical object 14 to be sluggish or offset from the transient obstruction 17. Based on synchronization data 28, it may be achieved in respective timing units 27 of the sensors 18 , the control unit

19 and the controller 13 have synchronous time values or times, the synchronization data 28 may be generated, for example, on the basis of the IEEE1588 / IEC61588 protocol or the TPP.

The control device 13 can then, according to a method according to FIG. 2, display the display position 15 of the graphic object in a step S10 on the AR display device 12 in successive, predetermined output modes. Update timings by setting the display position 15 based on the observation values 22, respectively. For this purpose, in a step S11, the observation values can be provided with a respective time stamp 23 (performed by the sensors 18 and / or the control unit 19). In a step S12, at least one of the observation values 22 for the time stamp can then be used by the control unit 13 the next, future output time at least one future observation value 22 'and / or the next future display position 15' will be extrapolated. Such an output time may be, for example, the V-sync time 28 of the output device 12.

The use of this method for timely precise synchronization of the output at the output device 12 by means of the V-sync timing 28 based on sensor data fusion, taking into account the time stamps 23 of the sensor data 20, 21 based on the PTP, enables a future position of the surrounding object 17 is anticipated, and thus the graphic object 14 is displayed at a display position 15 that matches the relative position of the surrounding object 17 at this output time, that is, the target relative position between object 14 and surrounding object 17 in FIG the output device 12 results. At the same time, the method makes it possible to carry out the image calculation on the basis of the transmission latency of obsolete observation values 22, without the temporal synchronization deteriorating. Thus, the desired superimposition of the graphic object 14 on the AR display device 12 with real environment objects 17 of course results in multiple graphic objects 14 for several surrounding objects 17 may be provided. In addition, it is possible to adjust and correct the deviation between reality and augmented display content. For the use of the method, the following conditions can be created. The overall system 10 receives a global, synchronized (relative) time, this can be realized for example by IEEE1588 / IEC61588 protocols. The raw sensor data 20 or the fused sensor data 21 can be timestamped 23 of the global time then be provided. After starting or booting up system 10, the synchronized time is available.

After the initialization of the graphics program module in the control device 13, for example an infotainment system (information

Entertainment system) of the motor vehicle 11, the global time is read out and buffered with each V sync pulse at the V sync times 28. Since the image output in the AR display device 12 always takes place at a fixed update rate, the time point of the output of the next, future image known. This is always done at 1 / refresh rate or 1 / frame rate off the last V sync pulse. Accordingly, the output timing for the output of the next image can be calculated and buffered. Accordingly, for the next output time, T_output = T_vsync + 1 / framerate, where T_vsync is the output time of the V-sync pulse and the refresh rate is framereite.

Now the system has 1 / framerate per second available from T_vsync to calculate the image, which should be output at the time T_ausgabe. It does not matter when exactly the calculation and generation of the image content of the graphical object takes place, since the image output always takes place only at the fixed, next output time. This offers the advantage that the resources for the calculation can be flexibly assigned to the system 10.

The control device 13 can now calculate the graphics data for the future output time T_output. In other words, the graphic object and its display position can be calculated. With the aid of the time stamp 23 of the sensor data 20 and / or the sensor fusion data 21 as well as with the aid of a motion prediction, the coordinates of the object 14 to be displayed or of the objects to be displayed, that is to say the respective display position or display position, are extrapolated such that they become the output time T_ausgabe with a real environment object 17 cover over. At the time of dispensing (that is, at time V-sync), the calculated graphic data of the at least one graphic object 14 is displayed by means of the AR display 12, and the process repeats cyclically.

If no graphics data has been calculated at the time of output (V sync), the system 10 is considered overloaded. In this case, an error message can be generated accordingly to initiate appropriate optimization measures. Such an optimization measure may, for example, provide for stopping or switching off another program module.

Overall, the examples show how the invention can provide a method for synchronizing an AR display content with sensor data in a distributed system.

Claims

A method for superimposing a virtual graphic object (14) over a real environment (16) by means of an AR display device (12), wherein in the method by a control device (13) a Display position (15) of the graphical object (14) on the AR display device (12) is updated at successive predetermined dispensing times (28), the display position (15) being based on respective observation values (22). because the real environment (16) and / or a relative position of the AR Display device (12) with respect to environment (16) describe is inserted, characterized in that the observation values (22) are provided with a respective time stamp (23), the respective time stamp (23) having a respective time stamp (23) Observation time at which the associated observation value (22) has been detected, and based on the respective time stamp (23) of at least one of the observation values (22) for the next, future output time (28) at least one future observation value (22 ') and / or the next, future display position (15 ') is extrapolated. A method according to claim 1, wherein the respective output time (28) is determined by a V-sync signal of the AR display (12), by which an image structure of the AR Method according to one of the preceding claims, wherein a respective timer unit (27) at least one observation device (B), by which at least some of the Beobachtungswer te (22) are generated, and a timer unit (27 ) of the Steuerein direction (13) with each other by means of synchronization data (26) are synchronized in time. A method according to claim 3, wherein the synchronization data (26) is generated and / or transmitted by means of a time-precision protocol. 5. The method according to one of the preceding claims, wherein the observation values (22) together with their respective Zeitstempei (23) via a Data network (25) and / or via a gateway device (25) to the control device (13) are transmitted, wherein the carry over a transmission latency 6. Method according to claim 1, wherein a respective position and / or speed and / or acceleration of at least one environment object (17) of the environment (16) and / or the AR display device (12) is determined as the respective observation value (22). for setting the future display position (15 ') 7. Method according to one of the preceding claims, wherein sensor values (20) from at least one sensor (18) and / or fused sensor values (21) are used as observation values (22). 8. Method according to one of the preceding claims, wherein the extrapolation is carried out by means of a Kalman filter 9 motor vehicle (11) with an AR display device (12) and with a Control device (13) for the AR display device (12) and with at least one observation device (B) for generating Be obachtungswerten (22), each having a real environment (16) of the motor vehicle (11) and / or a relative position of the AR Describe the display device (12) to the environment (16),  characterized in that  the motor vehicle (11) is adapted to carry out a method according to one of the preceding claims.