[go: up one dir, main page]

EP3549109A4 - VIRTUAL USER CONTROLS IN A MIXED REALITY ENVIRONMENT - Google Patents

VIRTUAL USER CONTROLS IN A MIXED REALITY ENVIRONMENT Download PDF

Info

Publication number
EP3549109A4
EP3549109A4 EP17878799.0A EP17878799A EP3549109A4 EP 3549109 A4 EP3549109 A4 EP 3549109A4 EP 17878799 A EP17878799 A EP 17878799A EP 3549109 A4 EP3549109 A4 EP 3549109A4
Authority
EP
European Patent Office
Prior art keywords
mixed reality
user controls
reality environment
virtual user
virtual
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP17878799.0A
Other languages
German (de)
French (fr)
Other versions
EP3549109A1 (en
EP3549109B1 (en
Inventor
Adrian Kaehler
John Adam CROSTON
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Magic Leap Inc
Original Assignee
Magic Leap Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Magic Leap Inc filed Critical Magic Leap Inc
Priority to EP23166917.7A priority Critical patent/EP4220572B1/en
Publication of EP3549109A1 publication Critical patent/EP3549109A1/en
Publication of EP3549109A4 publication Critical patent/EP3549109A4/en
Application granted granted Critical
Publication of EP3549109B1 publication Critical patent/EP3549109B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/04815—Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
    • G06V10/464—Salient features, e.g. scale invariant feature transforms [SIFT] using a plurality of salient features, e.g. bag-of-words [BoW] representations
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/107—Static hand or arm

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Mathematical Physics (AREA)
  • Evolutionary Computation (AREA)
  • Computer Hardware Design (AREA)
  • Computer Graphics (AREA)
  • Medical Informatics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biophysics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Optics & Photonics (AREA)
  • User Interface Of Digital Computer (AREA)
  • Processing Or Creating Images (AREA)
EP17878799.0A 2016-12-05 2017-12-01 Virtual user input controls in a mixed reality environment Active EP3549109B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP23166917.7A EP4220572B1 (en) 2016-12-05 2017-12-01 Virtual user input controls in a mixed reality environment

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201662430279P 2016-12-05 2016-12-05
US201662430308P 2016-12-05 2016-12-05
PCT/US2017/064311 WO2018106542A1 (en) 2016-12-05 2017-12-01 Virtual user input controls in a mixed reality environment

Related Child Applications (1)

Application Number Title Priority Date Filing Date
EP23166917.7A Division EP4220572B1 (en) 2016-12-05 2017-12-01 Virtual user input controls in a mixed reality environment

Publications (3)

Publication Number Publication Date
EP3549109A1 EP3549109A1 (en) 2019-10-09
EP3549109A4 true EP3549109A4 (en) 2020-07-29
EP3549109B1 EP3549109B1 (en) 2023-04-19

Family

ID=62243916

Family Applications (2)

Application Number Title Priority Date Filing Date
EP17878799.0A Active EP3549109B1 (en) 2016-12-05 2017-12-01 Virtual user input controls in a mixed reality environment
EP23166917.7A Active EP4220572B1 (en) 2016-12-05 2017-12-01 Virtual user input controls in a mixed reality environment

Family Applications After (1)

Application Number Title Priority Date Filing Date
EP23166917.7A Active EP4220572B1 (en) 2016-12-05 2017-12-01 Virtual user input controls in a mixed reality environment

Country Status (9)

Country Link
US (4) US11150777B2 (en)
EP (2) EP3549109B1 (en)
JP (3) JP7112399B2 (en)
KR (3) KR102830052B1 (en)
CN (2) CN110249368B (en)
AU (1) AU2017370555B2 (en)
CA (1) CA3045664A1 (en)
IL (2) IL266888B2 (en)
WO (1) WO2018106542A1 (en)

Families Citing this family (128)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8977255B2 (en) 2007-04-03 2015-03-10 Apple Inc. Method and system for operating a multi-function portable electronic device using voice-activation
US8676904B2 (en) 2008-10-02 2014-03-18 Apple Inc. Electronic devices with voice command and contextual data processing capabilities
EP2954514B1 (en) 2013-02-07 2021-03-31 Apple Inc. Voice trigger for a digital assistant
US9715875B2 (en) 2014-05-30 2017-07-25 Apple Inc. Reducing the need for manual start/end-pointing and trigger phrases
US10170123B2 (en) 2014-05-30 2019-01-01 Apple Inc. Intelligent assistant for home automation
US9338493B2 (en) 2014-06-30 2016-05-10 Apple Inc. Intelligent automated assistant for TV user interactions
US9886953B2 (en) 2015-03-08 2018-02-06 Apple Inc. Virtual assistant activation
US10460227B2 (en) 2015-05-15 2019-10-29 Apple Inc. Virtual assistant in a communication session
US10331312B2 (en) 2015-09-08 2019-06-25 Apple Inc. Intelligent automated assistant in a media environment
US10586535B2 (en) 2016-06-10 2020-03-10 Apple Inc. Intelligent digital assistant in a multi-tasking environment
DK201670540A1 (en) 2016-06-11 2018-01-08 Apple Inc Application integration with a digital assistant
US12197817B2 (en) 2016-06-11 2025-01-14 Apple Inc. Intelligent device arbitration and control
US11269480B2 (en) * 2016-08-23 2022-03-08 Reavire, Inc. Controlling objects using virtual rays
AU2017370555B2 (en) 2016-12-05 2022-12-15 Magic Leap, Inc. Virtual user input controls in a mixed reality environment
US11204787B2 (en) 2017-01-09 2021-12-21 Apple Inc. Application integration with a digital assistant
US10768425B2 (en) * 2017-02-14 2020-09-08 Securiport Llc Augmented reality monitoring of border control systems
US10627895B2 (en) * 2017-03-21 2020-04-21 Lenovo (Singapore) Pte Ltd Providing a virtual control
US20180336275A1 (en) 2017-05-16 2018-11-22 Apple Inc. Intelligent automated assistant for media exploration
EP3659017B1 (en) 2017-07-26 2023-04-05 Magic Leap, Inc. Training a neural network with representations of user interface devices
US10996742B2 (en) * 2017-10-17 2021-05-04 Logitech Europe S.A. Input device for AR/VR applications
WO2019095360A1 (en) * 2017-11-20 2019-05-23 腾讯科技（深圳）有限公司 Method and apparatus for handling menu in virtual scene, and storage medium
US10739861B2 (en) * 2018-01-10 2020-08-11 Facebook Technologies, Llc Long distance interaction with artificial reality objects using a near eye display interface
US10818288B2 (en) 2018-03-26 2020-10-27 Apple Inc. Natural assistant interaction
GB2574372B (en) * 2018-05-21 2021-08-11 Imagination Tech Ltd Implementing Traditional Computer Vision Algorithms As Neural Networks
DK180639B1 (en) 2018-06-01 2021-11-04 Apple Inc DISABILITY OF ATTENTION-ATTENTIVE VIRTUAL ASSISTANT
US10725629B2 (en) * 2018-06-25 2020-07-28 Google Llc Identifying and controlling smart devices
JP7172207B2 (en) * 2018-07-10 2022-11-16 オムロン株式会社 input device
US11348316B2 (en) * 2018-09-11 2022-05-31 Apple Inc. Location-based virtual element modality in three-dimensional content
WO2020068040A1 (en) * 2018-09-24 2020-04-02 Google Llc Controlling a device based on processing of image data that captures the device and/or an installation environment of the device
US11462215B2 (en) 2018-09-28 2022-10-04 Apple Inc. Multi-modal inputs for voice commands
CN112789578B (en) * 2018-09-28 2025-03-18 美国斯耐普公司 Neural network systems for posture, wearing, activity or carrying detection on wearable or mobile devices
US10824244B2 (en) * 2018-11-19 2020-11-03 Facebook Technologies, Llc Systems and methods for transitioning between modes of tracking real-world objects for artificial reality interfaces
US11487351B2 (en) * 2018-11-23 2022-11-01 International Business Machines Corporation Intelligent directing system in an internet of things (IoT) computing environment
US11188148B2 (en) * 2018-12-27 2021-11-30 Facebook Technologies, Llc User interaction in head-mounted display with eye tracking
US10831268B1 (en) * 2019-03-15 2020-11-10 Facebook Technologies, Llc Systems and methods for using eye tracking to improve user interactions with objects in artificial reality
US11348573B2 (en) 2019-03-18 2022-05-31 Apple Inc. Multimodality in digital assistant systems
CN109935206B (en) * 2019-04-15 2021-08-20 京东方科技集团股份有限公司 Display device brightness compensation method, device and equipment based on neural network
DK201970509A1 (en) 2019-05-06 2021-01-15 Apple Inc Spoken notifications
US11307752B2 (en) 2019-05-06 2022-04-19 Apple Inc. User configurable task triggers
US12210436B2 (en) * 2019-05-28 2025-01-28 Trane International Inc. Ranking user interface elements based on physical conditions
US10890983B2 (en) 2019-06-07 2021-01-12 Facebook Technologies, Llc Artificial reality system having a sliding menu
US10942585B2 (en) * 2019-07-22 2021-03-09 Zspace, Inc. Trackability enhancement of a passive stylus
GB2586059B (en) * 2019-08-01 2023-06-07 Sony Interactive Entertainment Inc System and method for generating user inputs for a video game
US11074081B2 (en) * 2019-08-02 2021-07-27 GM Global Technology Operations LLC Architecture and method supporting multiple vision stream using shared server on embedded platform
WO2021029164A1 (en) * 2019-08-09 2021-02-18 ソニー株式会社 Image processing device, image processing method, and program
KR20220045987A (en) 2019-08-19 2022-04-13 라이트 필드 랩 인코포레이티드 Lightfield displays for consumer devices
CN112954422A (en) * 2019-12-11 2021-06-11 天翼智慧家庭科技有限公司 Method for realizing IPTV remote controller on VR equipment
CN111006720B (en) * 2019-12-18 2021-06-11 刘启 Simulation method of measuring instrument
US11687778B2 (en) 2020-01-06 2023-06-27 The Research Foundation For The State University Of New York Fakecatcher: detection of synthetic portrait videos using biological signals
EP3851939A1 (en) * 2020-01-14 2021-07-21 Apple Inc. Positioning a user-controlled spatial selector based on extremity tracking information and eye tracking information
WO2021154437A1 (en) * 2020-01-27 2021-08-05 Magic Leap, Inc. Gaze timer based augmentation of functionality of a user input device
US11830291B2 (en) * 2020-02-10 2023-11-28 University Of Maryland, College Park System and method for multimodal emotion recognition
CN115087905A (en) 2020-02-10 2022-09-20 奇跃公司 Body-centric content positioning relative to three-dimensional containers in a mixed reality environment
CN117827004A (en) 2020-02-14 2024-04-05 奇跃公司 Session manager
JP7681609B2 (en) * 2020-02-14 2025-05-22 マジック リープ， インコーポレイテッド 3D Object Annotation
CN115516364B (en) * 2020-02-14 2024-04-26 奇跃公司 Tool Bridge
CN111459266A (en) * 2020-03-02 2020-07-28 重庆爱奇艺智能科技有限公司 Method and device for operating 2D application in virtual reality 3D scene
KR20210128269A (en) * 2020-04-16 2021-10-26 삼성전자주식회사 Augmented Reality (AR) device and method for predicting pose thereof
US11567571B2 (en) * 2020-05-05 2023-01-31 Autodesk, Inc. Remote control of a device via a virtual interface
US12301635B2 (en) 2020-05-11 2025-05-13 Apple Inc. Digital assistant hardware abstraction
US20210358294A1 (en) * 2020-05-15 2021-11-18 Microsoft Technology Licensing, Llc Holographic device control
CN111538252B (en) * 2020-05-25 2021-09-21 厦门大学 Intelligent home demonstration system applying VR technology
CN111672115B (en) * 2020-06-05 2022-09-23 腾讯科技（深圳）有限公司 Virtual object control method and device, computer equipment and storage medium
US11438683B2 (en) 2020-07-21 2022-09-06 Apple Inc. User identification using headphones
KR102477006B1 (en) * 2020-08-14 2022-12-14 주식회사 브이터치 Method, system and non-transitory computer-readable recording medium for supporting object control
US11474690B2 (en) * 2020-08-14 2022-10-18 VTouch Co., Ltd. Method, system and non-transitory computer-readable recording medium for non-contact control
US20220050432A1 (en) * 2020-08-14 2022-02-17 VTouch Co., Ltd. Method, system and non-transitory computer-readable recording medium for supporting object control
US11467403B2 (en) * 2020-08-20 2022-10-11 Htc Corporation Operating method and electronic system
CN114200895B (en) * 2020-09-17 2024-12-10 北京小米移动软件有限公司 Device control method, device and storage medium
CN112164216A (en) * 2020-09-18 2021-01-01 深圳市欧瑞博科技股份有限公司 Configuration method and device of intelligent remote controller, intelligent gateway equipment and storage medium
WO2022103420A1 (en) * 2020-11-16 2022-05-19 Google Llc Identifying a position of a controllable device using a wearable device
US11914835B2 (en) * 2020-11-16 2024-02-27 Samsung Electronics Co., Ltd. Method for displaying user interface and electronic device therefor
US11423585B2 (en) * 2020-11-30 2022-08-23 Microsoft Technology Licensing, Llc Velocity-based controls
JP2023512122A (en) * 2020-12-30 2023-03-24 広東視場科技有限公司 smart control glasses
US11656690B2 (en) * 2021-03-05 2023-05-23 MediVis, Inc. User input and virtual touch pad in augmented reality for use in surgical settings
US12271658B2 (en) 2021-02-23 2025-04-08 Apple Inc. Digital assistant interactions in copresence sessions
US12353632B2 (en) * 2021-04-08 2025-07-08 Snap Inc. Bimanual interactions between mapped hand regions for controlling virtual and graphical elements
US11250637B1 (en) * 2021-05-14 2022-02-15 Gridraster, Inc. Multimodal 3D deep learning fusion system and method for reducing the need of 3D training dataset of 3D object tracking for enterprise digital twin mixed reality
US12387435B2 (en) 2021-05-14 2025-08-12 Gridraster, Inc. Digital twin sub-millimeter alignment using multimodal 3D deep learning fusion system and method
US20220392170A1 (en) * 2021-06-07 2022-12-08 Citrix Systems, Inc. Interactive Display Devices in Extended Reality Environments
US11579752B1 (en) * 2021-07-20 2023-02-14 Sony Interactive Entertainment Inc. Augmented reality placement for user feedback
EP4409536A1 (en) 2021-09-01 2024-08-07 Snap Inc. Handcrafted augmented reality experiences
US12141363B2 (en) * 2021-09-01 2024-11-12 Snap Inc. Physical action-based augmented reality communication exchanges
EP4396787A1 (en) 2021-09-01 2024-07-10 Snap Inc. Handcrafted augmented reality effort evidence
EP4396649A1 (en) 2021-09-02 2024-07-10 Snap Inc. Augmented reality prop interactions
US12422933B2 (en) 2021-09-03 2025-09-23 Apple Inc. Multi-device gesture control
US12120374B2 (en) * 2021-09-28 2024-10-15 Arris Enterprises Llc Device and method to switch television operation to primary content mode after displaying an important content on secondary input mode
US11836828B2 (en) 2021-10-29 2023-12-05 Meta Platforms Technologies, Llc Controlling interactions with virtual objects
TW202324172A (en) 2021-11-10 2023-06-16 美商元平台技術有限公司 Automatic artificial reality world creation
US20250016231A1 (en) * 2021-11-10 2025-01-09 Drnc Holdings, Inc. CONTEXT AWARE OBJECT RECOGNITION FOR IoT CONTROL
CN118251643A (en) * 2021-11-17 2024-06-25 三星电子株式会社 Electronic device and method for anchoring augmented reality objects
KR20230073869A (en) * 2021-11-19 2023-05-26 삼성전자주식회사 Method of providing user interface supporting interaction and elecronic apparatus performing the same
US20230168786A1 (en) * 2021-11-30 2023-06-01 Verizon Patent And Licensing Inc. Methods and Systems for Location-Based Accessing of Predesignated Data Payloads Using Extended Reality
KR20230106257A (en) * 2022-01-06 2023-07-13 삼성전자주식회사 System, electronic device and method for controlling external electronic device in real world in virtual enviroment
WO2023130435A1 (en) * 2022-01-10 2023-07-13 深圳市闪至科技有限公司 Interaction method, head-mounted display device, and system and storage medium
US12020382B2 (en) 2022-03-24 2024-06-25 Kyndryl, Inc. Physical and virtual environment synchronization
TWI804294B (en) * 2022-04-28 2023-06-01 中華電信股份有限公司 Remote device control system, method and computer readable medium for augmented reality collaboration
US12423917B2 (en) 2022-06-10 2025-09-23 Apple Inc. Extended reality based digital assistant interactions
CN115437589A (en) * 2022-06-13 2022-12-06 北京罗克维尔斯科技有限公司 Controller, method, apparatus, system, medium, and device of display device
US20230409179A1 (en) * 2022-06-15 2023-12-21 Apple Inc. Home automation device control and designation
KR20240011477A (en) 2022-07-19 2024-01-26 주식회사 엘지에너지솔루션 Electrode assembly, battery, and battery pack and vehicle including the same
CN115357114A (en) * 2022-08-04 2022-11-18 闻泰通讯股份有限公司 A device interaction method and an interaction device
US12026325B2 (en) 2022-08-17 2024-07-02 Apple Inc. Handheld input devices with sleeves
US12259560B2 (en) 2022-08-23 2025-03-25 Apple Inc. Handheld controllers with charging and storage systems
KR20240028880A (en) * 2022-08-25 2024-03-05 삼성전자주식회사 Method for controlling external electronic device using remote control device and electronic device for supporting the same
US12045383B2 (en) 2022-09-01 2024-07-23 Snap Inc. Virtual AR interfaces for controlling IoT devices using mobile device orientation sensors
US12073011B2 (en) * 2022-09-01 2024-08-27 Snap Inc. Virtual interfaces for controlling IoT devices
US12148448B2 (en) 2022-09-01 2024-11-19 Snap Inc. Authoring tools for creating interactive AR experiences
US12169598B2 (en) 2022-09-19 2024-12-17 Snap Inc. AR glasses as IoT remote control
US12130972B2 (en) 2022-09-21 2024-10-29 Apple Inc. Tracking devices for handheld controllers
KR20240058665A (en) * 2022-10-26 2024-05-03 삼성전자주식회사 Wearable device for modifying visual object by using data identified by sensor and method thereof
CN115756230A (en) * 2022-11-14 2023-03-07 北京字跳网络技术有限公司 Panel interaction method, device, equipment and storage medium
KR20240084665A (en) * 2022-12-06 2024-06-14 삼성전자주식회사 Wearable device for displaying user interface associated with controlling of external electronic device and method thereof
CN118192793A (en) * 2022-12-14 2024-06-14 北京小米移动软件有限公司 AR glasses interactive display method, device, equipment and storage medium
US11869136B1 (en) * 2022-12-16 2024-01-09 Google Llc User-context aware rendering dataset selection
US20240244285A1 (en) * 2023-01-17 2024-07-18 Piotr Gurgul Ar glasses as iot device for enhanced screen experience
US12387449B1 (en) 2023-02-08 2025-08-12 Meta Platforms Technologies, Llc Facilitating system user interface (UI) interactions in an artificial reality (XR) environment
US12400414B2 (en) 2023-02-08 2025-08-26 Meta Platforms Technologies, Llc Facilitating system user interface (UI) interactions in an artificial reality (XR) environment
US20240281070A1 (en) * 2023-02-16 2024-08-22 Meta Platforms Technologies, Llc Simultaneous Controller and Touch Interactions
WO2024196915A1 (en) * 2023-03-20 2024-09-26 Magic Leap, Inc. Method and system for dynamic depth-based reprojection
US12481514B2 (en) * 2023-06-04 2025-11-25 Apple Inc. Volumetric user interface layers for electronic devices
US12079903B1 (en) * 2023-06-26 2024-09-03 Google Llc Real screens in extended reality
WO2025018514A1 (en) * 2023-07-19 2025-01-23 삼성전자주식회사 Electronic device and method for displaying connection state in virtual environment
US20250036371A1 (en) * 2023-07-25 2025-01-30 simpleAR, Inc. Serializing and deserializing mixed reality experiences or portions thereof
US20250094016A1 (en) * 2023-09-18 2025-03-20 Apple Inc. Systems and methods of motion-based user interfaces
KR102854751B1 (en) * 2024-02-26 2025-09-03 (주)비엔티 Large Capacity Data Processing System For Generating Images From Web Service-Enabled
CN118644706B (en) * 2024-05-29 2024-11-15 孚链艺统科技有限公司 Mobile cultural relic login method based on AID authentication technology
US12468381B1 (en) * 2024-09-06 2025-11-11 Snap Inc. Dynamic interaction zone system for accurate 3D button selection in augmented reality environments

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140049558A1 (en) * 2012-08-14 2014-02-20 Aaron Krauss Augmented reality overlay for control devices
US20160313902A1 (en) * 2015-04-27 2016-10-27 David M. Hill Mixed environment display of attached control elements

Family Cites Families (114)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5291560A (en) 1991-07-15 1994-03-01 Iri Scan Incorporated Biometric personal identification system based on iris analysis
US6222525B1 (en) 1992-03-05 2001-04-24 Brad A. Armstrong Image controllers with sheet connected sensors
US5583795A (en) 1995-03-17 1996-12-10 The United States Of America As Represented By The Secretary Of The Army Apparatus for measuring eye gaze and fixation duration, and method therefor
US5670988A (en) 1995-09-05 1997-09-23 Interlink Electronics, Inc. Trigger operated electronic device
JPH10200971A (en) * 1997-01-13 1998-07-31 Mitsubishi Electric Corp Signal generator for wireless remote control by personal computer
US7227526B2 (en) * 2000-07-24 2007-06-05 Gesturetek, Inc. Video-based image control system
WO2003077549A1 (en) 2002-03-13 2003-09-18 Imax Corporation Systems and methods for digitally re-mastering or otherwise modifying motion pictures or other image sequences data
US8098901B2 (en) 2005-01-26 2012-01-17 Honeywell International Inc. Standoff iris recognition system
KR20050025927A (en) 2003-09-08 2005-03-14 유웅덕 The pupil detection method and shape descriptor extraction method for a iris recognition, iris feature extraction apparatus and method, and iris recognition system and method using its
USD514570S1 (en) 2004-06-24 2006-02-07 Microsoft Corporation Region of a fingerprint scanning device with an illuminated ring
US7248720B2 (en) 2004-10-21 2007-07-24 Retica Systems, Inc. Method and system for generating a combined retina/iris pattern biometric
JP2006146803A (en) 2004-11-24 2006-06-08 Olympus Corp Operation device, and remote operation system
US20070081123A1 (en) 2005-10-07 2007-04-12 Lewis Scott W Digital eyewear
US8696113B2 (en) 2005-10-07 2014-04-15 Percept Technologies Inc. Enhanced optical and perceptual digital eyewear
US11428937B2 (en) 2005-10-07 2022-08-30 Percept Technologies Enhanced optical and perceptual digital eyewear
JP4824420B2 (en) 2006-02-07 2011-11-30 アイテック株式会社 Gaze vector detection method and apparatus
US7970179B2 (en) 2006-09-25 2011-06-28 Identix Incorporated Iris data extraction
US8363783B2 (en) 2007-06-04 2013-01-29 Oraya Therapeutics, Inc. Method and device for ocular alignment and coupling of ocular structures
KR20090034089A (en) * 2007-10-02 2009-04-07 삼성전자주식회사 Remote control device and method
US8098891B2 (en) 2007-11-29 2012-01-17 Nec Laboratories America, Inc. Efficient multi-hypothesis multi-human 3D tracking in crowded scenes
KR100927009B1 (en) 2008-02-04 2009-11-16 광주과학기술원 Haptic interaction method and system in augmented reality
JP5478618B2 (en) 2008-07-03 2014-04-23 エヌイーシー ラボラトリーズ アメリカ インク Epithelial layer detector and related method
US20100131947A1 (en) * 2008-11-24 2010-05-27 Disney Enterprises, Inc. System and method for enabling a local user of a real-life simulation environment to interact with a remote user of a corresponding virtual environment
JP2010134629A (en) * 2008-12-03 2010-06-17 Sony Corp Information processing apparatus and method
JP5262681B2 (en) * 2008-12-22 2013-08-14 ブラザー工業株式会社 Head mounted display and program thereof
US8768014B2 (en) 2009-01-14 2014-07-01 Indiana University Research And Technology Corp. System and method for identifying a person with reference to a sclera image
WO2011091326A1 (en) 2010-01-22 2011-07-28 Optimedica Corporation Apparatus for automated placement of scanned laser capsulorhexis incisions
US8345984B2 (en) 2010-01-28 2013-01-01 Nec Laboratories America, Inc. 3D convolutional neural networks for automatic human action recognition
JP5114795B2 (en) * 2010-01-29 2013-01-09 島根県 Image recognition apparatus, operation determination method, and program
US20150309316A1 (en) 2011-04-06 2015-10-29 Microsoft Technology Licensing, Llc Ar glasses with predictive control of external device based on event input
JP5743416B2 (en) 2010-03-29 2015-07-01 ソニー株式会社 Information processing apparatus, information processing method, and program
US9977496B2 (en) * 2010-07-23 2018-05-22 Telepatheye Inc. Eye-wearable device user interface and augmented reality method
WO2012027597A2 (en) * 2010-08-27 2012-03-01 Intel Corporation Capture and recall of home entertainment system session
JP5257437B2 (en) 2010-10-20 2013-08-07 コニカミノルタビジネステクノロジーズ株式会社 Method for operating portable terminal and processing device
US8683086B2 (en) * 2010-11-17 2014-03-25 Flextronics Ap, Llc. Universal remote control with automated setup
US9304319B2 (en) 2010-11-18 2016-04-05 Microsoft Technology Licensing, Llc Automatic focus improvement for augmented reality displays
US9348143B2 (en) 2010-12-24 2016-05-24 Magic Leap, Inc. Ergonomic head mounted display device and optical system
US10156722B2 (en) 2010-12-24 2018-12-18 Magic Leap, Inc. Methods and systems for displaying stereoscopy with a freeform optical system with addressable focus for virtual and augmented reality
CA3035118C (en) 2011-05-06 2022-01-04 Magic Leap, Inc. Massive simultaneous remote digital presence world
JP5765133B2 (en) * 2011-08-16 2015-08-19 富士通株式会社 Input device, input control method, and input control program
WO2013049861A1 (en) 2011-09-29 2013-04-04 Magic Leap, Inc. Tactile glove for human-computer interaction
US20130104085A1 (en) 2011-10-21 2013-04-25 Digital Artforms, Inc. Systems and methods for human-computer interaction using a two handed interface
EP2771877B1 (en) 2011-10-28 2017-10-11 Magic Leap, Inc. System and method for augmented and virtual reality
KR102440195B1 (en) 2011-11-23 2022-09-02 매직 립, 인코포레이티드 Three dimensional virtual and augmented reality display system
US9497501B2 (en) * 2011-12-06 2016-11-15 Microsoft Technology Licensing, Llc Augmented reality virtual monitor
US20130147686A1 (en) * 2011-12-12 2013-06-13 John Clavin Connecting Head Mounted Displays To External Displays And Other Communication Networks
JP2013172432A (en) 2012-02-23 2013-09-02 Panasonic Corp Equipment control apparatus, equipment control method, equipment control program, and integrated circuit
AU2013243380B2 (en) 2012-04-05 2017-04-20 Magic Leap, Inc. Wide-field of view (FOV) imaging devices with active foveation capability
US9671566B2 (en) 2012-06-11 2017-06-06 Magic Leap, Inc. Planar waveguide apparatus with diffraction element(s) and system employing same
CN107817555A (en) 2012-06-11 2018-03-20 奇跃公司 Use more depth plane three dimensional displays of the waveguided reflector arrays projector
US9141916B1 (en) 2012-06-29 2015-09-22 Google Inc. Using embedding functions with a deep network
JP5467303B1 (en) 2012-07-31 2014-04-09 独立行政法人科学技術振興機構 Gaze point detection device, gaze point detection method, personal parameter calculation device, personal parameter calculation method, program, and computer-readable recording medium
US8369595B1 (en) 2012-08-10 2013-02-05 EyeVerify LLC Texture features for biometric authentication
US9740006B2 (en) 2012-09-11 2017-08-22 Magic Leap, Inc. Ergonomic head mounted display device and optical system
JP2014072576A (en) * 2012-09-27 2014-04-21 Kyocera Corp Display device and control method
US20140152558A1 (en) * 2012-11-30 2014-06-05 Tom Salter Direct hologram manipulation using imu
KR20150103723A (en) 2013-01-03 2015-09-11 메타 컴퍼니 Extramissive spatial imaging digital eye glass for virtual or augmediated vision
EP2946236B1 (en) 2013-01-15 2021-06-16 Magic Leap, Inc. Ultra-high resolution scanning fiber display
JP2014137774A (en) * 2013-01-18 2014-07-28 Denso Corp Display operation device
IL308285B2 (en) 2013-03-11 2024-11-01 Magic Leap Inc System and method for augmented and virtual reality
US9147154B2 (en) 2013-03-13 2015-09-29 Google Inc. Classifying resources using a deep network
KR102271719B1 (en) 2013-03-15 2021-06-30 매직 립, 인코포레이티드 Display system and method
JP6165485B2 (en) 2013-03-28 2017-07-19 国立大学法人埼玉大学 AR gesture user interface system for mobile terminals
WO2014182769A1 (en) 2013-05-07 2014-11-13 The Johns Hopkins University Automated and non-mydriatic fundus-perimetry camera for irreversible eye diseases
US20140341441A1 (en) 2013-05-20 2014-11-20 Motorola Mobility Llc Wearable device user authentication
US9275308B2 (en) 2013-05-31 2016-03-01 Google Inc. Object detection using deep neural networks
US10262462B2 (en) 2014-04-18 2019-04-16 Magic Leap, Inc. Systems and methods for augmented and virtual reality
US9874749B2 (en) 2013-11-27 2018-01-23 Magic Leap, Inc. Virtual and augmented reality systems and methods
US20140380249A1 (en) 2013-06-25 2014-12-25 Apple Inc. Visual recognition of gestures
CN103431840B (en) 2013-07-31 2016-01-20 北京智谷睿拓技术服务有限公司 Eye optical parameter detecting system and method
CN105829953B (en) 2013-10-16 2019-05-28 奇跃公司 Virtual or augmented reality helmet with adjustable interpupillary distance
US9202144B2 (en) 2013-10-30 2015-12-01 Nec Laboratories America, Inc. Regionlets with shift invariant neural patterns for object detection
US10095917B2 (en) 2013-11-04 2018-10-09 Facebook, Inc. Systems and methods for facial representation
JP6236296B2 (en) 2013-11-14 2017-11-22 株式会社デンソーアイティーラボラトリ Learning device, learning program, and learning method
CN109445095B (en) 2013-11-27 2021-11-23 奇跃公司 Virtual and augmented reality systems and methods
US9857591B2 (en) 2014-05-30 2018-01-02 Magic Leap, Inc. Methods and system for creating focal planes in virtual and augmented reality
KR20150073378A (en) * 2013-12-23 2015-07-01 삼성전자주식회사 A device and method for displaying a user interface(ui) of virtual input device based on motion rocognition
CN104750234B (en) 2013-12-27 2018-12-21 中芯国际集成电路制造(北京)有限公司 The interactive approach of wearable smart machine and wearable smart machine
WO2015111778A1 (en) 2014-01-27 2015-07-30 엘지전자 주식회사 Terminal of eye-glass type and method for controlling terminal of eye-glass type
US9430829B2 (en) 2014-01-30 2016-08-30 Case Western Reserve University Automatic detection of mitosis using handcrafted and convolutional neural network features
WO2015117043A1 (en) 2014-01-31 2015-08-06 Magic Leap, Inc. Multi-focal display system and method
CN106233189B (en) 2014-01-31 2020-06-26 奇跃公司 Multifocal Display System and Method
US10203762B2 (en) 2014-03-11 2019-02-12 Magic Leap, Inc. Methods and systems for creating virtual and augmented reality
IL231862A (en) 2014-04-01 2015-04-30 Superfish Ltd Neural network image representation
WO2015164807A1 (en) 2014-04-25 2015-10-29 Texas State University Detection of brain injury and subject state with eye movement biometrics
JP6500477B2 (en) 2015-02-12 2019-04-17 セイコーエプソン株式会社 Head-mounted display device, control system, control method of head-mounted display device, and computer program
US9766715B2 (en) 2014-05-01 2017-09-19 Seiko Epson Corporation Head-mount type display device, control system, method of controlling head-mount type display device, and computer program
US20150324568A1 (en) 2014-05-09 2015-11-12 Eyefluence, Inc. Systems and methods for using eye signals with secure mobile communications
USD759657S1 (en) 2014-05-19 2016-06-21 Microsoft Corporation Connector with illumination region
EP3149539B1 (en) 2014-05-30 2025-04-30 Magic Leap, Inc. Virtual or augmented reality apparatus
USD752529S1 (en) 2014-06-09 2016-03-29 Comcast Cable Communications, Llc Electronic housing with illuminated region
US9766806B2 (en) * 2014-07-15 2017-09-19 Microsoft Technology Licensing, Llc Holographic keyboard display
US9536293B2 (en) 2014-07-30 2017-01-03 Adobe Systems Incorporated Image assessment using deep convolutional neural networks
US20160034811A1 (en) 2014-07-31 2016-02-04 Apple Inc. Efficient generation of complementary acoustic models for performing automatic speech recognition system combination
US9659384B2 (en) 2014-10-03 2017-05-23 EyeEm Mobile GmbH. Systems, methods, and computer program products for searching and sorting images by aesthetic quality
CN105917354A (en) 2014-10-09 2016-08-31 微软技术许可有限责任公司 Spatial pyramid pooling networks for image processing
WO2016054802A1 (en) 2014-10-10 2016-04-14 Beijing Kuangshi Technology Co., Ltd. Hierarchical interlinked multi-scale convolutional network for image parsing
US9900541B2 (en) * 2014-12-03 2018-02-20 Vizio Inc Augmented reality remote control
KR102276339B1 (en) 2014-12-09 2021-07-12 삼성전자주식회사 Apparatus and method for training convolutional neural network for approximation of convolutional neural network
WO2016100717A1 (en) 2014-12-17 2016-06-23 Google Inc. Generating numeric embeddings of images
US9678664B2 (en) 2015-04-10 2017-06-13 Google Inc. Neural network for keyboard input decoding
USD758367S1 (en) 2015-05-14 2016-06-07 Magic Leap, Inc. Virtual reality headset
EP3337385A4 (en) 2015-08-21 2019-04-03 Magic Leap, Inc. ESTIMATING THE SHAPE OF A EYEWEAR USING THE MEASUREMENT OF THE POSITION OF THE EYE
USD805734S1 (en) 2016-03-04 2017-12-26 Nike, Inc. Shirt
USD794288S1 (en) 2016-03-11 2017-08-15 Nike, Inc. Shoe with illuminable sole light sequence
CN114495249A (en) 2016-07-14 2022-05-13 奇跃公司 Iris boundary estimation using corneal curvature
WO2018013200A1 (en) 2016-07-14 2018-01-18 Magic Leap, Inc. Deep neural network for iris identification
KR102529137B1 (en) 2016-08-22 2023-05-03 매직 립, 인코포레이티드 Augmented reality display device with deep learning sensors
RU2016138608A (en) 2016-09-29 2018-03-30 Мэджик Лип, Инк. NEURAL NETWORK FOR SEGMENTING THE EYE IMAGE AND ASSESSING THE QUALITY OF THE IMAGE
IL293688B2 (en) 2016-10-04 2024-02-01 Magic Leap Inc Efficient data layouts for convolutional neural networks
KR102762372B1 (en) 2016-11-15 2025-02-04 매직 립, 인코포레이티드 Deep learning system for cuboid detection
AU2017370555B2 (en) 2016-12-05 2022-12-15 Magic Leap, Inc. Virtual user input controls in a mixed reality environment
US10803616B1 (en) 2017-04-13 2020-10-13 Facebook Technologies, Llc Hand calibration using single depth camera
EP3659017B1 (en) 2017-07-26 2023-04-05 Magic Leap, Inc. Training a neural network with representations of user interface devices

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140049558A1 (en) * 2012-08-14 2014-02-20 Aaron Krauss Augmented reality overlay for control devices
US20160313902A1 (en) * 2015-04-27 2016-10-27 David M. Hill Mixed environment display of attached control elements

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
VALENTIN HEUN ET AL: "Smarter objects", 20130427; 1077952576 - 1077952576, 27 April 2013 (2013-04-27), pages 961 - 966, XP058015943, ISBN: 978-1-4503-1952-2, DOI: 10.1145/2468356.2468528 *

Also Published As

Publication number Publication date
JP7112399B2 (en) 2022-08-03
KR20190094381A (en) 2019-08-13
US20180157398A1 (en) 2018-06-07
KR20230070318A (en) 2023-05-22
US20220083198A1 (en) 2022-03-17
EP3549109A1 (en) 2019-10-09
IL266888B1 (en) 2024-01-01
EP4220572B1 (en) 2025-05-14
US20250068292A1 (en) 2025-02-27
US20230315250A1 (en) 2023-10-05
US11720223B2 (en) 2023-08-08
EP4220572A1 (en) 2023-08-02
KR20220088962A (en) 2022-06-28
CA3045664A1 (en) 2018-06-14
KR102531542B1 (en) 2023-05-10
US11150777B2 (en) 2021-10-19
CN110249368A (en) 2019-09-17
AU2017370555A1 (en) 2019-06-13
JP2022097726A (en) 2022-06-30
CN110249368B (en) 2023-07-07
WO2018106542A1 (en) 2018-06-14
US12175054B2 (en) 2024-12-24
JP2020513618A (en) 2020-05-14
KR102830052B1 (en) 2025-07-03
JP7720364B2 (en) 2025-08-07
AU2017370555B2 (en) 2022-12-15
CN116883628A (en) 2023-10-13
IL266888A (en) 2019-07-31
JP2023153357A (en) 2023-10-17
IL308490A (en) 2024-01-01
JP7339392B2 (en) 2023-09-05
EP3549109B1 (en) 2023-04-19
IL266888B2 (en) 2024-05-01
KR102413561B1 (en) 2022-06-24

Similar Documents

Publication Publication Date Title
EP3549109A4 (en) VIRTUAL USER CONTROLS IN A MIXED REALITY ENVIRONMENT
EP3646309C0 (en) LAPAROSCOPIC TOOLS WITH VIRTUAL REALITY
IL268397A (en) Drawing a virtual and real object on a mixed reality device
IL265756A (en) Controls and interfaces for user interactions in virtual spaces
GB201702693D0 (en) Virtual reality
PL3392827T3 (en) Collaborative multi-user virtual reality
SG10202108555YA (en) Three-dimensional graphical user interface for informational input in virtual reality environment
EP3677999C0 (en) MANIPULATION OF USER INTERFACE OBJECTS IN A USER INTERFACE
DK3038002T3 (en) INTERACTIVE USER INTERFACES
GB2579789B (en) Runtime parameter selection in simulations
SG11201703011VA (en) Virtual reality immersion with an architectural design software application
GB2524269B (en) Virtual reality
GB2560004B (en) Virtual reality
GB201807762D0 (en) Location globe in virtual reality
GB201515631D0 (en) A method and system for communicating with a user immersed in a virtual reality environment
EP3223245C0 (en) VIEWPOINT SELECTION IN VIRTUAL 3D ENVIRONMENT
EP3368003A4 (en) COSMETIC COMPOSITION COMPRISING A POLYURETHANE
EP3149578C0 (en) GROUPING VIRTUAL MACHINES IN A CLOUD APPLICATION
GB201620985D0 (en) Virtual reality
PT3037964T (en) Virtual application manager in a cloud computing environment
EP3606726A4 (en) SEGMENTS IN VIRTUAL BUILDING VOLUMES
EP3704630A4 (en) ALIGNED VIEWS IN VIRTUAL REALITY
EP3469474A4 (en) DISPENSER CONTROL USER INTERFACE
GB2558284B (en) Virtual reality
EP3398094A4 (en) ENLARGEMENT OF VIRTUAL MODELS

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20190531

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20200626

RIC1 Information provided on ipc code assigned before grant

Ipc: G06T 19/00 20110101AFI20200622BHEP

Ipc: G06F 3/01 20060101ALI20200622BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20210726

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: G06V 40/10 20220101ALI20221018BHEP

Ipc: G06V 20/20 20220101ALI20221018BHEP

Ipc: G06V 10/46 20220101ALI20221018BHEP

Ipc: G06N 20/00 20190101ALI20221018BHEP

Ipc: G06F 3/04815 20220101ALI20221018BHEP

Ipc: G06F 3/01 20060101ALI20221018BHEP

Ipc: G06T 19/00 20110101AFI20221018BHEP

INTG Intention to grant announced

Effective date: 20221107

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602017067940

Country of ref document: DE

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1561769

Country of ref document: AT

Kind code of ref document: T

Effective date: 20230515

REG Reference to a national code

Ref country code: NL

Ref legal event code: FP

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230719

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1561769

Country of ref document: AT

Kind code of ref document: T

Effective date: 20230419

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230821

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230719

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230819

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230720

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602017067940

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20240122

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231201

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231201

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231201

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231231

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231201

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20231231

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230419

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20241121

Year of fee payment: 8

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20241121

Year of fee payment: 8

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: BE

Payment date: 20241121

Year of fee payment: 8

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20241122

Year of fee payment: 8

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20241121

Year of fee payment: 8

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20171201

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20171201