[go: up one dir, main page]

WO2020263423A1 - Utilisation d'un suivi oculaire pour cacher des changements de scène de réalité virtuelle dans une vue en clair - Google Patents

Utilisation d'un suivi oculaire pour cacher des changements de scène de réalité virtuelle dans une vue en clair Download PDF

Info

Publication number
WO2020263423A1
WO2020263423A1 PCT/US2020/031549 US2020031549W WO2020263423A1 WO 2020263423 A1 WO2020263423 A1 WO 2020263423A1 US 2020031549 W US2020031549 W US 2020031549W WO 2020263423 A1 WO2020263423 A1 WO 2020263423A1
Authority
WO
WIPO (PCT)
Prior art keywords
virtual scene
determined
focal regions
user
virtual
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/US2020/031549
Other languages
English (en)
Inventor
Andrew D. Wilson
Sebastian Lennard MARWECKI
Eyal Ofek
Christian Holz
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Technology Licensing LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Technology Licensing LLC filed Critical Microsoft Technology Licensing LLC
Priority to EP20729353.1A priority Critical patent/EP3991012A1/fr
Publication of WO2020263423A1 publication Critical patent/WO2020263423A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0179—Display position adjusting means not related to the information to be displayed
    • G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2016—Rotation, translation, scaling

Definitions

  • Virtual reality technology employs specialized computing hardware and software to provide users immersive virtual environments to interact with and explore.
  • Virtual reality technologies can place users into virtual, computer-generated environments, where they can perceive and interact with virtual objects rendered therein.
  • An essential part of any media production is creating and arranging scenery, or staging, which is an important part of audience engagement and enjoyment.
  • the computing device can determine a set of focal regions of the displayed virtual scene, such as a foveal region, a parafoveal region, and/or a perifoveal region, among other things. As one or more sets of focal regions are determined, the computing device can calculate probabilities, analyze timing, determine intent, or process other factors to determine whether and/or when to modify or change a portion of the virtual scene with an increased likelihood that a wearer or user of the HMD does not notice the modification or change.
  • a set of focal regions of the displayed virtual scene such as a foveal region, a parafoveal region, and/or a perifoveal region, among other things.
  • the computing device can calculate probabilities, analyze timing, determine intent, or process other factors to determine whether and/or when to modify or change a portion of the virtual scene with an increased likelihood that a wearer or user of the HMD does not notice the modification or change.
  • FIG. 1 is a block diagram of an exemplary operating environment for dynamically modifying a virtual environment, in accordance with some embodiments of the present disclosure
  • FIG. 2 is a block diagram of an exemplary reality scene modification device for dynamically modifying a virtual environment utilizing eye-tracking, in accordance with some embodiments of the present disclosure
  • FIGs. 3A-3C depict a variety of illustrations showing exemplary implementations for dynamically modifying virtual environments utilizing eye-tracking, in accordance with some embodiments of the present disclosure
  • FIGs. 4A-4B depict a variety of illustrations showing further exemplary implementations for dynamically modifying virtual environments utilizing eye-tracking, in accordance with some embodiments of the present disclosure
  • FIG. 5A-5B depict a variety of illustrations showing even further exemplary implementations for dynamically modifying virtual environments utilizing eye tracking, in accordance with some embodiments of the present disclosure
  • FIG. 6A-6B depict a variety of illustrations showing even further exemplary implementations for dynamically modifying virtual environments utilizing eye tracking, in accordance with some embodiments of the present disclosure
  • FIG. 7A-7C depict a variety of illustrations showing even further exemplary implementations for dynamically modifying virtual environments utilizing eye tracking, in accordance with some embodiments of the present disclosure
  • FIG. 8A-8B depict a variety of illustrations showing even further exemplary implementations for dynamically modifying virtual environments utilizing eye tracking, in accordance with some embodiments of the present disclosure
  • FIG. 9 is a flow chart depicting an exemplary process flow for dynamically modifying virtual environments utilizing eye-tracking, in accordance with some embodiments of the present disclosure.
  • FIG. 10 is a block diagram of an exemplary computing environment suitable for use in accordance with some embodiments of the present disclosure.
  • Virtual reality is an immersive technology, where the user of the technology perceives a rendered virtual environment and virtual objects therein as if the perceived visual information was their present reality. While immersed in the virtual environment, typically while wearing a head-mounted display (HMD), the user can interact with the virtual environment.
  • HMD head-mounted display
  • a scene staging generally occurs before or as a production experience unfolds. While a movie or other content playback follows a linear flow having a predetermined timing of events (i.e. viewpoint is predetermined), in contrast, an interactive media (e.g., VR) may require unexpected, or ad-hoc staging, where user input is taken into account.
  • an interactive media e.g., VR
  • changes to a scene during staging are generally designed to be hidden from a user, for example they happen outside of a visible area. For example, they can happen behind an obstacle or obstacle or obstruction (e.g. a door), or simply occur outside of the field of view of a user.
  • dynamic staging can pose certain challenges. For example, a user may freely look around a virtual scene and might observe a change as it occurs, which can take away from a realistic user experience. Additionally, as the field of view of virtual reality and augmented reality headsets become larger, less space becomes available for dynamic staging. To address these drawbacks, and to create opportunities for dynamic staging in VR scenes, embodiments of the technology described herein provide for making modifications or changes to a virtual scene inside a user’s field of view with little to no likelihood of observance.
  • eye-tracking data may be leveraged to make a modification or change to a virtual scene within a user’s field of view so that the modification or change is“hidden,” or in other words, made covertly with a low likelihood of notice.
  • embodiments of the present disclosure describe techniques for modifying or changing portions of a virtual scene by utilizing eye-tracking data.
  • embodiments of the present disclosure can facilitate modifications or changes to a virtual scene, for example in unattended areas of a user’s field of view, that are unnoticed by a user.
  • embodiments of the present disclosure generally provide systems and methods for dynamically modifying a virtual environment or“scene” based on detected eye-tracking data. Accordingly, various embodiments of the technology disclosed herein track, measure, or otherwise capture data corresponding to a user’s eyes that can be utilized to modify a virtual environment, for instance to covertly insert changes into a virtual scene. More specifically, a computing device coupled to a head-mounted display (HMD) receives sensor data from one or more sensors.
  • HMD head-mounted display
  • the sensors generate sensor data that includes eye-tracking data, for instance data corresponding to a user’s point of gaze, the motion of an eye relative to a user’s head, a pupil size, and/or reactivity of a user’s eye, among other things.
  • the eye-tracking data is employed by the computing device to determine if and/or when to modify one or more portions of a virtual scene that is currently displayed by the HMD, for example to hide changes inside a user’s field of view but applying them outside a determined foveal region of the virtual scene, parafoveal region of the virtual scene, or perifoveal region of the virtual scene.
  • the computing device can apply covert changes to a virtual scene that can occur inside a user’s field of view by processing eye tracking data on two layers.
  • the computing device can determine when to change or modify a portion of a virtual scene.
  • the computing device can process the generated eye-tracking data to determine a user’s attention with respect to a potential modification to a portion of a virtual scene, i.e. the likelihood a user will notice the change or modification.
  • the computing device can determine the probability of detection corresponding to the change or modification. If the determined probability of detection falls below a pre-determined threshold the computing device can determine that the change or modification is ready to be made to the virtual scene.
  • the computing device can determine if the change or modification should be made. As such, in addition to ensuring that a user does not notice the change or modification, the computing device can determine that a user will not anticipate the change or modification. For example, restricting a change or modification soon after a user performs a specific action or a series of actions. In this way covert changes or modification to a virtual scene can be caused by covert triggers, either using random triggers or through derivation of user understanding and intention through the leveraging of eye-tracking data. In one instance the computing device can determine scan paths to define triggers. For example, the computing device can compare a generated gaze pattern of a user and compare it to a predefined gaze pattern corresponding to a particular displayed scene to determine user understanding and, in turn, determine if a change or modification should be made in a given virtual scene.
  • the computing device can receive sensor data from a set of eye-tracking sensors coupled to a HMD as a user interacts with a virtual scene provided for display on the HMD coupled to the computing device. Based on the eye tracking data, the computing device can determine a set of focal regions of the displayed virtual scene, for example the focal regions can include a foveal region, a parafoveal region, and/or a perifoveal region of a virtual scene displayed on an HMD. The virtual scene can then be modified by the computing device by determining that a portion of the virtual scene to be modified is outside the determined set of focal regions.
  • the computing device can process the generated eye-tracking data with a focus on a user’s attention as they interact with a virtual scene.
  • the computing device can base such processing on one or more high-level functions of a user’s perception, for example attention and memory and causal inference.
  • the computing device can derive a probability that a user will detect a modification or change in a virtual scene.
  • One or more probabilities can be accumulated by the computing device and a single detection value can be derived to be utilized by the computing device as a detection rate to make a determination of when a change or modification can be made to a virtual scene perceived by a user.
  • the computing device can cause changes or modifications to a virtual scene that is currently displayed by the HMD by utilizing gaze point to determine the current attention of a user. In some embodiments this is accomplished through determining focal regions of a displayed virtual scene based on eye tracking data associated with a user. It will be appreciated that changes or modification to virtual objects that appear outside of a determined foveal, parafoveal, and/or perifoveal region of the displayed virtual scene are less likely to be detected, and as such gaze point can be leveraged to make changes or modification to the virtual scene. In this instance, the computing device can compute the convex hull of a virtual object’s mesh data points with respect to the view plane.
  • the virtual object is considered by the computing device to be in the user’s focus (e.g. the user is looking at the virtual object within the virtual scene).
  • the minimum angular distance threshold can be set to 7 degrees, such that the threshold is above a 5 degree angle of a foveal region. In other embodiments, a higher angular distance can be used as a user’s central vision can extend to 30 degrees.
  • the computing device can cause changes or modifications to a virtual scene with that is displayed by the HMD, by utilizing and processing gaze dwell times on various objects in the virtual scene.
  • spatial memory can be considered with respect to a user’s attention when changing or modifying a virtual scene, thus a user may be prevented from using visual recall to notice a change or modification of the virtual scene.
  • the computing device can determine one or more different sets of focal regions of a displayed virtual scene based on different sensor data received from the set of eye-tracking sensors.
  • the different sets of focal regions can include corresponding different foveal, parafoveal, and/or perifoveal regions of the displayed virtual scene.
  • the computing device can calculate a weight associated with various portions or virtual objects of the virtual scene, for instance a calculated weight can be based on determined time durations between the determined set of focal regions and the one or more different sets of focal regions. Based on the calculated weight a portion of the virtual scene can be modified or changed.
  • a weighted directed graph can be implemented by the computing device with nodes representing all relevant objects in a virtual scene and edges connecting the nodes. Weights can be determined for theses nodes that can for example represent an internalized allocentric spatial relation between pairs of nodes.
  • the user can also be represented by a node, and the edges from the user node can represent the internalized egocentric spatial relation between the user and one or more virtual objects.
  • the weights can be adjusted (i.e. incremented or decremented) based on determined sets of focal regions of the virtual scene. For example, the weights can increment when a user looks at a virtual object or at multiple virtual objects in a row, which in some instances can be order dependent. The weights can decrement when a user has not looked at a virtual object for a given time duration.
  • the computing device can determine the probability of a user noticing the changed spatial configuration. For instance, the determined probability can be the maximum of the difference of all spatial relations before and after the change or modification that can further be normalized by a baseline distance and a baseline rotation.
  • the computing device can cause changes or modifications to a virtual scene that is displayed by an HMD utilizing pupilometry as it relates to cognitive load.
  • the computing device can determine a pupilometry based on, for example, a detected pupil diameter included in received sensor data from the set of eye tracking sensors. From the pupilometry, a cognitive load value can be determined and the virtual scene can be changed or modified based on the cognitive load value.
  • an index of pupillary activity can be implemented as a measure of cognitive load.
  • a pupil dilation’s correlation with detection rate of changes to a virtual scene can be utilized to predict a user’s attention.
  • pupil dilation can be measured over varying illumination levels of a viewing plane to determine a baseline. From on this baseline, an assumed pupil diameter can be derived over which fast wavelet transforms can be run to count local peaks of that value, which can be normalized.
  • the computing device can cause changes or modifications to a virtual scene that is currently displayed by an HMD utilizing saccades, for instance saccades may be used to approximate visual saliency.
  • the computing device can determine a number of saccades over a defined period of time based on the received sensor data, and change or modify a virtual scene based on a determination that the number of saccades for a given time frame is below a threshold value.
  • modifications or changes to a virtual scene can be made at a time between a measured saccade and a given time after it, as this assumes a user would not focus on objects outside of the determined focal region during this time frame (i.e. covert attention of a user is not possible during the time duration after a measured saccade).
  • covert attention can be prevented by implementing a task requiring a user’s visual attention (e.g. reading).
  • the computing device can cause changes or modifications to a virtual scene that is displayed by an HMD by taking into account measured user intent and/or measured user understanding as they relate to a virtual scene.
  • user intent can be measured by capturing and thresholding dwell times on objects or sets of objects.
  • the computing device can determine a time duration associated with a determined foveal region as compared to a different portion of a displayed virtual scene.
  • the virtual scene can be changed or modified based on a comparison of the determined duration to a defined dwell threshold duration.
  • user understanding can be measured by using a probabilistic comparison of a user’s gaze pattern to an intended gaze pattern associated with a virtual scene.
  • the computing device can determine a plurality of different sets of focal regions of the displayed virtual scene, the plurality of different sets of focal regions defining a gaze pattern.
  • the computing device can change or modify the virtual scene based on a determination that the defined gaze pattern corresponds to at least one predetermined or known gaze pattern.
  • the computing device can utilize additional masking techniques when changing or modifying a virtual scene, for example by modifying a virtual scene based on another modification to the virtual scene, such as the insertion of a virtual object into the virtual scene that is positioned at a distance away from the portion to be modified. Additionally, other modifications can be made to the virtual scene that is a threshold distance away from the portion to be modified, for example creating distractions, or reducing frame rates in portions of the virtual scene.
  • FIG. 1 a block diagram is provided showing an example operating environment 100 in which some embodiments of the present disclosure may be employed. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, and groupings of functions) can be used in addition to or instead of those shown, and some elements may be omitted altogether for the sake of clarity. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by one or more entities may be carried out by hardware, firmware, and/or software. For instance, some functions may be carried out by a processor executing instructions stored in memory.
  • example operating environment 100 includes a head-mounted display (HMD) coupled to a set of sensors, such as HMD 110 and sensor(s) 120.
  • the HMD 110 can include a stereoscopic display to facilitate a stereoscopic presentation of three-dimensional virtual environments and virtual objects that can be perceived by a user wearing the HMD 110.
  • the HMD 110 can incorporate a virtual scene modification component 115, that can be integrated into the HMD, or stored separately, for example on a remote computing device or server device 140.
  • the set of sensors can include tracking sensors 120
  • the sensor data can be used to determine a user’s point of gaze, the motion of an eye relative to a user’s head, or pupil size, and/or reactivity of a user’s eye.
  • virtual scene data can be hosted by a remote server device, such as server device 140.
  • the server device 140 can be accessed by HMD 110 via a network 130, which can include a LAN, WAN, PAN, or the Internet, by way of example.
  • the server device 140 can be coupled to a database 150 that can store, among other things, one or more applications comprising virtual scene data that can be retrieved and provided for display on HMD 110.
  • FIG. 2 a block diagram 200 is provided, illustrating an exemplary virtual scene modification component 210, such as virtual scene modification component 115 of FIG. 1, for dynamically modifying or changing a virtual scene based on received sensor data, such as data received from sensors 120.
  • a virtual scene modification component 210 provided in accordance with some described embodiments, in an example, can hide or covertly insert modifications or changes to a virtual scene that is displayed, or currently displayed, by an HMD (e.g. HMD 110 of FIG. 1), among other things. It should be understood that this and other arrangements described herein are set forth only as examples.
  • the virtual scene modification component 210 is an example of a suitable architecture for implementing certain aspects of the present disclosure. It should be understood that any number of user devices, hardware, modules, or components within the scope of the present disclosure can be employed to perform the functions described in associated with the virtual scene modification component 210. As each of the described components are depicted as being included in the virtual scene modification component 210, it is contemplated that any component depicted therein is not limited to the illustrated embodiment, and can be distributed among a plurality of computing devices, modules, or hardware devices, or in some instances, may be conflated into a single hardware device or module, such as a processor or hardware device.
  • any one or more of the described components can be completely removed from the virtual scene modification component 210, so long as one or more operations described in correspondence to a removed component can be compensated for by one or more other components, or a third-party resource, remote computing device, or hardware device, among other things.
  • the virtual scene modification component 210 can include an eye-tracking component 220 that receives sensor data from the sensor(s) 120, e.g. eye-tracking sensor(s).
  • the eye-tracking component 220 can determine, among other things, a user’s point of gaze, the motion of an eye relative to a user’s head, or pupil size, and/or reactivity of a user’s eye based on the received eye-tracking sensor data.
  • the eye-tracking sensor data can include electronic information that corresponds to fixations, saccades, and/or reactivity of a user’s eye and its movement, among other things.
  • the virtual scene modification component 210 can further include a focal region defining component 230.
  • the focal region defining component 230 can generate, or otherwise define, a set of focal regions of the displayed virtual scene based on the eye-tracking sensor data, for example provided by eye-tracking component 220.
  • the set of focal regions can include one or more of a foveal region, a parafoveal region, and/or a perifoveal region of a displayed virtual scene. As will be appreciated, these regions can correspond to how information, such as a scene, is processed by a user’s eye.
  • the virtual scene modification component 210 can further include an attention tracking component 240.
  • the attention tracking component 240 can determine that a portion of a virtual scene can be modified based on the determined set of focal regions (e.g. foveal region, a parafoveal region, and/or a perifoveal region of the displayed virtual scene).
  • attention tracking component 240 processes the determined set of focal regions to determine when to modify or change a virtual scene.
  • attention tracking component 240 can provide an estimation and/or a determination of a user’s attention with respect to the portion of the virtual scene to be modified (i.e. the likelihood of a user noticing the change in the field of view).
  • an application generating the virtual scene, or another user can suggest, request, or otherwise specify a modification or change (e.g. the movement or changing of one or more virtual objects) to be made to the virtual scene.
  • the request can be sent to and/or received by the virtual scene modification component 220.
  • the attention tracking component 240 can process the focal regions of the virtual scene and subsequently calculate or otherwise determine a probability that a user will detect the change. It will be appreciated that the attention tracking component 240, in essence, determines when a modification or change can be made to a portion of the virtual scene.
  • attention tracking component 240 can determine that a portion of a virtual scene can be modified based on a relative position of the portion to be modified or changed as compared to one or more regions of the determined set of focal regions. In some other embodiments, attention tracking component 240 can determine that a portion of the virtual scene can be modified based on a weight calculated for the portion (e.g. based on a weight calculation derived from a weighted directed graph), a determined pupilometry (e.g.
  • attention tracking component 240 can determine or otherwise calculate one or more attention values which each correspond to a probability of detection, which defines how likely or not a user will notice, or attend to, a modification or change to a portion of the virtual scene. Furthermore, attention tracking component 240 can determine whether a probably of detection falls below a predefined threshold, indicating the modification or change can be made to the virtual scene at that time.
  • the virtual scene modification component 220 can further include a change activation component 250.
  • the change activating component 250 can determine that a portion of a virtual scene can be modified based on the determined set of focal regions (e.g. foveal region, a parafoveal region, and/or a perifoveal region of the displayed virtual scene).
  • change activation component 250 processes the determined set of focal regions to determine if a modification or change can be made to a virtual scene.
  • attention tracking component 240 can determine that a user is not likely to notice or attend to the modification or change
  • change activation component 250 can determine that a user is not likely to anticipate a change.
  • covert modifications or changes to a virtual scene can incorporate covert triggers.
  • the virtual scene modification component 210, and more particularly change activating component 250 can use random triggers to cause a modification or change to a virtual scene (e.g. cross-fading one or more virtual objects into, or out of, the virtual scene at random time intervals), or in some embodiments, user understanding and user intention with respect to one or more features (e.g. one or more virtual objects and their associated visual characteristics) of a virtual scene can be determined or otherwise derived by change activation component 250.
  • one or more predefined scan paths, or gaze patterns can be associated with a virtual scene (e.g. stored on the HMD or in data store 150 of server 140).
  • change activation component 250 can process the focal regions and determine one or more scan paths or gaze patterns for comparison to the one or more predefined scan paths or gaze patterns which can be used to determine if a modification or change to a portion of the virtual scene can occur.
  • change activation component 250 can determine a user intent and/or interest with respect to one or more virtual objects of the virtual scene. For example, user intent with respect to a particular virtual object can be derived by determining that a foveal region of the determined set of focal regions corresponds to the particular virtual object. Additionally it can be determined (e.g.
  • change activation component 250 can determine a user understanding with respect to one or more virtual objects of the virtual scene. For example, user understanding with respect to a virtual scene can be derived by determining a plurality of different sets of focal regions (e.g. by focal region defining component 230) and subsequently defining (e.g. by change activation component 250) one or more gaze patterns or scan paths corresponding to the virtual scene. Change activation component 250 can probabilistically compare the one or more gaze patterns or scan paths to one or more known (e.g.
  • change activation component 250 can generate the one or more gaze patterns or scan paths by implementing a weighted directed graph, where the weights of the nodes and edges are adjusted (incremented and/or decremented) based on a determination that a foveal region corresponds to a particular virtual object for a given amount of time and/or on a determination of a time duration between when a first set of focal regions is determined and when a second set of focal regions is determined.
  • the virtual scene modification component 220 can further include a scene modifying component 260.
  • Scene modifying component 260 can receive input from, for example, attention tracking component 240 and/or change activation component 250 and cause a modification or change to one or more portions of a virtual scene.
  • Scene modifying component 260 can additionally cause other changes to the virtual scene to further mask modifications or changes to the one or more portions of the virtual scene. For example, as described herein, changes or modifications can be made to a portions of a virtual scene in an unattended area of display (i.e. outside a user’s fovea).
  • salient noise or other motion conditions can be injected into the virtual scene that can create distractions to mask a modification or change to a portion of the virtual scene.
  • scene modifying component 260 can distract a user or otherwise reduce the possibility that a user will detect a change or modification made to a portion of a virtual scene.
  • FIGs 3A-3C through FIGs. 8A-8B are now described in relation to some of the foregoing masking techniques.
  • a virtual scene 310 can be provided for display on an HMD (e.g. HMD 110 of FIG. 1) for a user to interact with.
  • HMD e.g. HMD 110 of FIG. 1
  • a user can interact with the virtual scene 310 by being given a task 320 to complete, e.g. solving a puzzle.
  • the virtual reality application generating the scene 310 can make the task 320 easier or more difficult by rearranging, inserting, or removing one or more virtual objects 322a, 322b, 322n in the virtual scene 310.
  • modifications or changes to the virtual scene 310 can be made covertly according to aspects of the technology described herein such that the user does not notice the modification or change within the field of view, for example a modification or change can be made inside the field of view but outside of a focal region.
  • a set of focal regions 330 of the displayed virtual scene 310 can be determined, for example the set of focal regions 330 can include a foveal region 332, a parafoveal region 334, and/or a perifoveal region 336 of a displayed virtual scene 310.
  • these regions can relate to various regions of a user’s eye used to process an image (i.e. image information), for example with respect to visual acuity.
  • Foveal region 332 for example can define a focal region corresponding to sharp, central image processing.
  • Parafoveal region 334 for example, can define an intermediate focal region and can define a greater field of vision than foveal region 332.
  • Perifoveal region 336 for example, can define an outer focal region and defines a greater field of vision than parafoveal region 334.
  • these defined regions can vary in width or degree, and that there can also be additional or fewer defined regions.
  • the focal regions 330 can relatively correspond to a user’s gaze, or in other words, correspond relatively to a location that the user is looking at (e.g., a virtual object 340) in the virtual scene 310.
  • a portion of the virtual scene 310 can be modified or changed based the determined set of focal regions 330. For instance, a modification or change can be made with respect to another virtual object, e.g. virtual object 322a in the virtual scene 310 that is outside of the determined set of focal regions 330. Additionally, other changes to the virtual scene can be made in order to further mask the modification or change made with respect to the another virtual object.
  • salient noise or other motion conditions can be injected into the virtual scene that can create distractions to mask a modification or change to a portion of the virtual scene (e.g. causing virtual object 340 to move or become animated).
  • a gradual fade may be utilized to further mask the modification or change made with respect to the another virtual object (e.g. virtual object 322a may be moved via a gradual fade).
  • contrast adjustments e.g. lowering the contrast
  • a virtual scene 410 can be provided for display on an HMD (e.g. HMD 110 of FIG. 1) for a user to interact with.
  • the virtual scene 410 can include any number of virtual objects 412a, 412b, 412n.
  • a user can interact with the virtual scene 410 via a haptic prop 414 that is located in the physical world.
  • sensor data can be received from a set of eye-tracking sensors (e.g. sensors 120 of FIG.
  • a set of focal regions 414 can be determined.
  • the application generating the virtual scene 410 is enabled to allow the user to pick up a virtual object 412b, 412n utilizing the haptic prop 414.
  • the set of focal regions 414 it can be determined, for example, that the user is interested in the virtual object 412a being looked at. Accordingly, a portion of the virtual scene 410 can be modified or changed based on the determined focal regions 414, and further such that the modification or change occurs outside the set of focal regions 414.
  • similar object 412b can be moved, matched, or otherwise mapped to a position and a rotation of the haptic prop 414.
  • other changes to the virtual scene can be made in order to further mask the modification or change.
  • a gradual fade may be utilized to further mask the modification or change made in the virtual scene with respect to moving, matching, or otherwise mapping similar object 412b (e.g. virtual objects 412b and 412n may be moved via a gradual fade).
  • a virtual scene 510 can be provided for display on an HMD (e.g. HMD 110 of FIG. 1) for a user to interact with.
  • the virtual scene 510 can include any number of virtual objects 512a, 512b, 512n.
  • sensor data can be received from a set of eye-tracking sensors (e.g. sensors 120 of FIG. 1) and a set of focal regions 514 can be determined.
  • a portion of the virtual scene 510 can be modified or changed based on the determined focal regions 514, for example to adapt to a user’s interest, virtual object 512b can be changed, modified, or otherwise switched, e.g. to virtual object 512c.
  • other changes to the virtual scene can be made in order to further mask the modification or change. For example, a gradual fade may be utilized to further mask the modification or change made in the virtual scene (e.g. the change or modification made with respect to virtual objects 512b and 512c may be done via a gradual fade).
  • a virtual scene 610 can be provided for display on an HMD (e.g. HMD 110 of FIG. 1) for a user to interact with.
  • the virtual scene 610 can include any number of virtual objects, for example virtual objects 612a and 612b.
  • sensor data can be received from a set of eye-tracking sensors (e.g. sensors 120 of FIG. 1) and a set of focal regions 614 can be determined.
  • a portion of the virtual scene 610 can be modified or changed based on the determined focal regions 614, for example an effect 616 (e.g. low fidelity effect) can be applied to a portion of the virtual scene 610 outside of the set of focal regions 614, for example an effect 616 can be applied to virtual object 612b, such that when a user shifts their gaze to virtual object 612b they can see the result of the effect 616.
  • an effect 616 e.g. low fidelity effect
  • a virtual scene 710 can be provided for display on an HMD (e.g. HMD 110 of FIG. 1) for a user to interact with.
  • Virtual scene 710 can include any number of virtual objects (e.g. 711, 712, and 716a) that a user can view and/or otherwise interact with.
  • a user can be presented with one or more tasks to complete (e.g. via interaction with virtual object 712) that requires the user to understand one or more features of the virtual scene 710.
  • a set of focal regions 714 of the displayed virtual scene 710 can be determined as the user attempts to complete the one or more tasks without understanding one or more features of the virtual scene 710.
  • one or more virtual objects e.g. 716a, 711
  • virtual object 716a of FIG. 7A can be modified to 716b of FIG. 7B.
  • a plurality of different sets of focal regions 718a, 718b, 718c, 718d, 718e of the displayed virtual scene 710 can be determined as the user attempts to complete the one or more tasks.
  • one or more gaze patterns or scan paths can be defined (e.g. by change activation component 250) and compared against one or more known or otherwise predefined gaze patterns or scan paths to determine a measure of user understanding with respect to the virtual scene 710.
  • a virtual scene 810 can be provided for display on an HMD (e.g. HMD 110 of FIG. 1) for a user to interact with.
  • HMD e.g. HMD 110 of FIG. 1
  • a user can interact with a virtual scene through virtual locomotion, for example a user can jump from one location (e.g. a present location) in a virtual scene 810 to another location 812 in the virtual scene 810.
  • a set of eye-tracking sensors e.g. eye-tracking sensors 120 of FIG.
  • a set of focal regions 814 of the displayed virtual scene 810 can be determined as the user “moves” through the virtual scene 810. Once the set of focal regions 814 are determined, one or more modifications or changes can be made to a portion 816 of the virtual scene 810 where the portion 816 is outside of the determined focal regions 814.
  • the modification or change can include reducing the frame rate corresponding to the portion of the virtual scene 810 outside of the focal regions 814, for example by blending static images.
  • FIG. 9 a flow diagram is provided depicting a method 900 for dynamically modifying virtual environments, or a virtual scene, utilizing eye-tracking.
  • Each block of method 900 and other methods described herein comprises a computing process that may be performed using any combination of hardware, firmware, and/or software. For instance, various functions may be carried out by a module, hardware device, or processor executing instructions stored in memory. Various portions of the methods may also be embodied as computer-usable instructions stored on computer storage media.
  • a head-mounted display (HMD) (e.g., HMD 110 of FIG. 1) is coupled to a computing device.
  • HMD head-mounted display
  • the HMD can be coupled to a set of eye-tracking sensors, such as sensor(s) 120 of FIG. 1.
  • the computing device can provide a virtual scene to an HMD for display.
  • the computing device can receive sensor data (e.g. eye-tracking data) from a set of eye-tracking sensors coupled to the HMD (e.g. sensor(s) 120 of FIGs 1 and 2).
  • the computing device can determine a set of focal regions of the displayed virtual scene based on the received sensor data (e.g. by attention tracking component 240 of FIG. 2).
  • the set of focal regions includes a foveal region, a parafoveal region, and/or a perifoveal region of a displayed virtual scene.
  • gaze point to can be utilized to determine the current attention of a user. In some embodiments this is accomplished through determining focal regions of a displayed virtual scene based on eye-tracking data associated with a user. It will be appreciated that changes or modification to virtual objects that appear outside of a determined foveal, parafoveal, and/or perifoveal region of the displayed virtual scene are less likely to be detected, and as such gaze point can be leveraged to make changes or modification to the virtual scene.
  • the computing device can compute the convex hull of a virtual object’s mesh data points with respect to the view plane. If the angular distance between the computed convex hull and the measured or determined gaze point (e.g. based on the set of focal regions) falls below a threshold value, the virtual object is considered by the computing device to be in the user’s focus (e.g. the user is looking at the virtual object within the virtual scene). In some further embodiments gaze dwell times on various objects in the virtual scene can be utilized to determine attention with respect to a virtual scene.
  • the computing device can determine one or more different sets of focal regions of a displayed virtual scene based on different sensor data received from the set of eye-tracking sensors.
  • the different sets of focal regions can include corresponding different foveal, parafoveal, and/or perifoveal regions of the displayed virtual scene.
  • the computing device can calculate a weight associated with various portions or virtual objects of the virtual scene, for instance a calculated weight can be based on determined time durations between the determined set of focal regions and the one or more different sets of focal regions. Based on the calculated weight a portion of the virtual scene can be modified or changed.
  • a weighted directed graph can be implemented to process gaze dwell times.
  • pupilometry as it relates to cognitive load can be utilized to process attention with respect to a virtual scene.
  • the computing device can determine a pupilometry based on, for example, a detected pupil diameter included in received sensor data from the set of eye-tracking sensors. From the pupilometry, a cognitive load value can be determined and the virtual scene can be changed or modified based on the cognitive load value.
  • an index of pupillary activity can be implemented as a measure of cognitive load. Accordingly, a pupil dilation’s correlation with detection rate of changes to a virtual scene can be utilized to predict a user’s attention.
  • pupil dilation can be measured over varying illumination levels of a viewing plane to determine a baseline. From on this baseline, an assumed pupil diameter can be derived over which fast wavelet transforms can be run to count local peaks of that value, which can be normalized.
  • saccades are utilized in processing user attention with respect to a virtual scene.
  • saccades may be used to approximate visual saliency.
  • the computing device can determine a number of saccades over a defined period of time based on the received sensor data, and change or modify a virtual scene based on a determination that the number of saccades for a given time frame is below a threshold value.
  • modifications or changes to a virtual scene can be made at a time between a measured saccade and a given time after it, as this assumes a user would not focus on objects outside of the determined focal region during this time frame.
  • covert attention can be prevented by implementing a task requiring a user’s visual attention.
  • measured user intent and/or measured user understanding as they relate to a virtual scene can be taken into account to modify or make changes to a virtual scene.
  • user intent can be measured by capturing and threshold dwell times on objects or sets of objects.
  • the computing device can determine a time duration associated with a determined foveal region as compared to a different portion of a displayed virtual scene.
  • the virtual scene can be changed or modified based on a comparison of the determined duration to a defined dwell threshold duration.
  • user understanding can be measured by using a probabilistic comparison of a user’s gaze pattern to an intended gaze pattern associated with a virtual scene.
  • the computing device can determine a plurality of different sets of focal regions of the displayed virtual scene, the plurality of different sets of focal regions defining a gaze pattern.
  • the computing device can change or modify the virtual scene based on a determination that the defined gaze pattern corresponds to at least one predetermined or known gaze pattern.
  • the computing device can utilize additional masking techniques when changing or modifying a virtual scene, for example by modifying a virtual scene based on another modification to the virtual scene, such as the insertion of a virtual object into the virtual scene that is positioned at a distance away from the portion to be modified.
  • modifications can be made to the virtual scene that is a threshold distance away from the portion to be modified, for example creating distractions, or reducing frame rates in portions of the virtual scene, which can mask out peripheral motion, i.e. obfuscates portions of a virtual scene outside of a foveal region.
  • the other modification can additionally comprise the insertion of a virtual object into the virtual scene positioned at least a defined threshold distance away from the portion, or a modification of another portion of the virtual scene positioned at least the defined threshold distance away from the portion.
  • the computing device can modify a portion of the virtual scene
  • the portion of the virtual scene is modified based further in part on a probability that the modification is detected, the probability being calculated based at least in part on the determined set of focal regions.
  • the virtual scene can be modified based on a duration associated with the foveal region and a different portion of the displayed virtual scene. In this instance the portion of the virtual scene is modified based further on a comparison of a determined duration to a defined dwell threshold duration.
  • the virtual scene can be modified based on gaze patterns.
  • a plurality of different sets of focal regions of the displayed virtual scene based on received sensor data can be determined.
  • the plurality of different sets of focal regions can define a gaze pattern, and a portion of the virtual scene is modified based further in part on a determination that the defined gaze pattern corresponds to one of a set of known gaze patterns.
  • the computing device can provide the virtual scene, including the modified portion, to the HMD for display.
  • computing device 1000 includes a bus 1010 that directly or indirectly couples the following devices: memory 1012, one or more processors 1014, one or more presentation components 1016, input/output ports 1018, input/output components 1020, and an illustrative power supply 1022.
  • Bus 1010 represents what may be one or more busses (such as an address bus, data bus, or combination thereof).
  • busses such as an address bus, data bus, or combination thereof.
  • FIG. 10 is merely illustrative of an exemplary computing device that can be used in connection with one or more embodiments of the present invention. Distinction is not made between such categories as “workstation,” “server,” “laptop,” “hand-held device,” etc., as all are contemplated within the scope of FIG. 10 and reference to “computing device.”
  • Computing device 1000 typically includes a variety of computer-readable media.
  • Computer-readable media can be any available media that can be accessed by computing device 1000 and includes both volatile and nonvolatile media, removable and non-removable media.
  • Computer-readable media may comprise computer storage media and communication media.
  • Computer storage media include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.
  • Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device 1000.
  • Computer storage media excludes signals per se.
  • Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
  • modulated data signal means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.
  • communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.
  • Memory 1012 includes computer storage media in the form of volatile and/or nonvolatile memory.
  • the memory may be removable, non-removable, or a combination thereof.
  • Exemplary hardware devices include solid-state memory, hard drives, optical-disc drives, etc.
  • Computing device 1000 includes one or more processors that read data from various entities such as memory 1012 or I/O components 1020.
  • Presentation component(s) 1016 present data indications to a user or other device.
  • Exemplary presentation components include a display device, speaker, printing component, vibrating component, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Optics & Photonics (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • Computer Graphics (AREA)
  • Architecture (AREA)
  • User Interface Of Digital Computer (AREA)
  • Controls And Circuits For Display Device (AREA)

Abstract

Divers modes de réalisation sont décrits dans la présente invention pour modifier une scène virtuelle sur la base d'un suivi oculaire. Un dispositif informatique couplé à un HMD peut fournir une scène virtuelle à afficher sur le HMD. Le dispositif informatique peut recevoir des données de capteur provenant d'un ensemble de capteurs de suivi oculaire couplés au HMD. Sur la base des données de capteur reçues, le dispositif informatique peut déterminer un ensemble de régions focales de la scène virtuelle affichée, comprenant une région périfovéale de la scène virtuelle affichée. Une partie de la scène virtuelle peut ensuite être modifiée sur la base, en partie, de la détermination que la partie est à l'extérieur de la région périfovéale déterminée.
PCT/US2020/031549 2019-06-25 2020-05-06 Utilisation d'un suivi oculaire pour cacher des changements de scène de réalité virtuelle dans une vue en clair Ceased WO2020263423A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP20729353.1A EP3991012A1 (fr) 2019-06-25 2020-05-06 Utilisation d'un suivi oculaire pour cacher des changements de scène de réalité virtuelle dans une vue en clair

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US16/451,132 2019-06-25
US16/451,132 US10976816B2 (en) 2019-06-25 2019-06-25 Using eye tracking to hide virtual reality scene changes in plain sight

Publications (1)

Publication Number Publication Date
WO2020263423A1 true WO2020263423A1 (fr) 2020-12-30

Family

ID=70919045

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2020/031549 Ceased WO2020263423A1 (fr) 2019-06-25 2020-05-06 Utilisation d'un suivi oculaire pour cacher des changements de scène de réalité virtuelle dans une vue en clair

Country Status (3)

Country Link
US (1) US10976816B2 (fr)
EP (1) EP3991012A1 (fr)
WO (1) WO2020263423A1 (fr)

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9332285B1 (en) * 2014-05-28 2016-05-03 Lucasfilm Entertainment Company Ltd. Switching modes of a media content item
US11417228B2 (en) * 2019-09-18 2022-08-16 International Business Machines Corporation Modification of extended reality environments based on learning characteristics
US11614797B2 (en) * 2019-11-05 2023-03-28 Micron Technology, Inc. Rendering enhancement based in part on eye tracking
CN111522437B (zh) * 2020-03-09 2023-05-02 中国美术学院 一种基于眼动数据获取产品原型方法和系统
CN113262464A (zh) * 2021-04-21 2021-08-17 青岛小鸟看看科技有限公司 一种虚拟现实场景的动态改变方法、装置及电子设备
US12216819B2 (en) * 2021-04-28 2025-02-04 The Johns Hopkins University Methods, systems, and related aspects for determining a cognitive load of a sensorized device user
US11808945B2 (en) * 2021-09-07 2023-11-07 Meta Platforms Technologies, Llc Eye data and operation of head mounted device
US20240212301A1 (en) * 2022-12-23 2024-06-27 Beijing Zitiao Network Technology Co., Ltd. Methods, apparatuses, device and storage medium for displaying content in virtual scene
CN115862124B (zh) * 2023-02-16 2023-05-09 南昌虚拟现实研究院股份有限公司 视线估计方法、装置、可读存储介质及电子设备

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017031089A1 (fr) * 2015-08-15 2017-02-23 Eyefluence, Inc. Systèmes et procédés destinés à des signaux oculaires basés sur la biomécanique et permettant d'entrer en interaction avec des objets réels et virtuels
US20180033405A1 (en) * 2016-08-01 2018-02-01 Facebook, Inc. Adaptive parameters in image regions based on eye tracking information
WO2018022523A1 (fr) * 2016-07-25 2018-02-01 Magic Leap, Inc. Modification, affichage et visualisation d'images à l'aide de lunettes de réalité augmentée et virtuelle
US20190005735A1 (en) * 2017-06-30 2019-01-03 Tobii Ab Systems and methods for displaying images in a virtual world environment

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6343808B2 (ja) * 2013-08-08 2018-06-20 パナソニックＩｐマネジメント株式会社 視野算出装置および視野算出方法
JP6550460B2 (ja) * 2014-05-09 2019-07-24 グーグル エルエルシー 眼信号を識別するためのシステムおよび方法、ならびに連続バイオメトリック認証

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017031089A1 (fr) * 2015-08-15 2017-02-23 Eyefluence, Inc. Systèmes et procédés destinés à des signaux oculaires basés sur la biomécanique et permettant d'entrer en interaction avec des objets réels et virtuels
WO2018022523A1 (fr) * 2016-07-25 2018-02-01 Magic Leap, Inc. Modification, affichage et visualisation d'images à l'aide de lunettes de réalité augmentée et virtuelle
US20180033405A1 (en) * 2016-08-01 2018-02-01 Facebook, Inc. Adaptive parameters in image regions based on eye tracking information
US20190005735A1 (en) * 2017-06-30 2019-01-03 Tobii Ab Systems and methods for displaying images in a virtual world environment

Also Published As

Publication number Publication date
US20200409455A1 (en) 2020-12-31
EP3991012A1 (fr) 2022-05-04
US10976816B2 (en) 2021-04-13

Similar Documents

Publication Publication Date Title
US10976816B2 (en) Using eye tracking to hide virtual reality scene changes in plain sight
US10890968B2 (en) Electronic device with foveated display and gaze prediction
US12236011B2 (en) Reducing head mounted display power consumption and heat generation through predictive rendering of content
CN109643162B (zh) 用现实世界内容增强虚拟现实内容
US9135508B2 (en) Enhanced user eye gaze estimation
JP6317765B2 (ja) 複合現実感表示調整
US9041623B2 (en) Total field of view classification for head-mounted display
Kellnhofer et al. GazeStereo3D: Seamless disparity manipulations
CN105393191A (zh) 自适应事件识别
US12189848B2 (en) Managing devices having additive displays
Mantiuk et al. Gaze‐driven object tracking for real time rendering
CN114026603B (zh) 渲染计算机生成现实文本
US11543655B1 (en) Rendering for multi-focus display systems
WO2015088910A1 (fr) Affichage interactif d'images à grande gamme dynamique
Koulieris et al. C‐LOD: Context‐aware Material Level‐of‐Detail applied to Mobile Graphics
US11949949B2 (en) Content generation based on audience engagement
Berkman Eye tracking in virtual reality
US20240112303A1 (en) Context-Based Selection of Perspective Correction Operations
US20250097402A1 (en) Interpolation of reprojected content
US11496723B1 (en) Automatically capturing a moment
US11823343B1 (en) Method and device for modifying content according to various simulation characteristics
Lehtiranta Gaze Prediction in VR
CN120673693A (zh) 显示控制方法、装置、电子设备、存储介质及程序产品
Sun Computational Methods for Immersive Perception
CN118974686A (zh) 针对内容的节能上下文相关处理

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20729353

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020729353

Country of ref document: EP

Effective date: 20220125