[go: up one dir, main page]

WO2023009853A1 - Interface utilisateur pour sélectionner un champ de vision d'une caméra dans des lunettes intelligentes - Google Patents

Interface utilisateur pour sélectionner un champ de vision d'une caméra dans des lunettes intelligentes Download PDF

Info

Publication number
WO2023009853A1
WO2023009853A1 PCT/US2022/038933 US2022038933W WO2023009853A1 WO 2023009853 A1 WO2023009853 A1 WO 2023009853A1 US 2022038933 W US2022038933 W US 2022038933W WO 2023009853 A1 WO2023009853 A1 WO 2023009853A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
interest
view
field
region
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/US2022/038933
Other languages
English (en)
Inventor
Sebastian Sztuk
Sapna Shroff
Jun Hu
Johana Gabriela Coyoc ESCUDERO
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Meta Platforms Technologies LLC
Original Assignee
Meta Platforms Technologies LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US17/859,808 external-priority patent/US12132983B2/en
Application filed by Meta Platforms Technologies LLC filed Critical Meta Platforms Technologies LLC
Priority to CN202280051850.7A priority Critical patent/CN117769694A/zh
Priority to EP22769424.7A priority patent/EP4377771A1/fr
Publication of WO2023009853A1 publication Critical patent/WO2023009853A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
    • G01C21/34—Route searching; Route guidance
    • G01C21/36—Input/output arrangements for on-board computers
    • G01C21/3626—Details of the output of route guidance instructions
    • G01C21/3629—Guidance using speech or audio output, e.g. text-to-speech
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B27/0172—Head mounted characterised by optical features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B2027/0178—Eyeglass type
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/02—Services making use of location information
    • H04W4/021—Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences

Definitions

  • the present disclosure is related to user interfaces in smart glass devices that include one or more cameras for recording images and video. More specifically, the present disclosure is related to methods for identifying a region of interest in a user field of view using eye tracking tools and providing non-invasive feedback to the user to make the user’s region of interest consistent with the field of view of the one or more cameras in the smart glasses.
  • Many electronic appliances today include, embedded within a wearable framework, one or more cameras that the user may activate to collect pictures or videos. However, in many cases, the user collects images and videos that do not match the region of interest of the user’s field of view at the time of collection. This happens because the angle and field of view of the one or more cameras is generally not the same as that of the user. In certain types of electronic appliances, this inconsistency may be cured by providing direct feedback to the user, on a display, of the field of view of the one or more camera accessories. The user can then manually modify camera position and optical configuration (magnification, aperture size, and the like) to make it consistent with a region of interest.
  • camera position and optical configuration magnification, aperture size, and the like
  • a device in a first embodiment, includes: a frame including an eyepiece to provide a forward-image to a user, a first forward-looking camera mounted on the frame; a processor configured to identify a region of interest within the field of view of the user, and an interface device to indicate to the user that the field of view of the first forward-looking camera is misaligned with the region of interest.
  • a device comprising: a frame including an eyepiece to provide a forward-image to a user; a first forward-looking camera mounted on the frame; a processor configured to identify a region of interest within the forward-image; and an interface device to indicate the user that a field of view of the first forward-looking camera is misaligned with the region of interest.
  • the device may further comprise a sensing device facing the user for tracking a pupil position of the user, wherein the processor identifies the region of interest based on the pupil position of the user.
  • the processor may identify the region of interest based on a gaze direction of the user within the field of view.
  • the device may further comprise a memory storing instructions which, when executed by the processor, cause the device to find a gazing vector indicative of a center of the region of interest within the field of view of the user.
  • the interface device may include an audio device to provide a signal for the user to adjust a head pose and align the field of view of the first forward-looking camera with the region of interest.
  • the interface device may comprise a vibrating component configured to indicate the user to nudge a head position in a desired direction to align the field of view of the first forward-looking camera with the region of interest.
  • the eyepiece may include a liquid crystal configured to obscure a vertical portion of the eyepiece, or a horizontal portion of the eyepiece, when activated, to induce the user to move a head position horizontally or vertically, respectively, to center the field of view of the first forward-looking camera with the region of interest.
  • the device may further comprise an actuator configured to modify a color of a portion of the field of view of the eyepiece corresponding to the region of interest.
  • the device may further comprise a microphone to receive a voice command from the user identifying an object of interest within the field of view.
  • the device may further comprise a second forward-looking camera, wherein the interface device is configured to provide the user an option to activate the second forward-looking camera when the field of view of the first forward-looking camera is misaligned with the region of interest.
  • a computer-implemented method includes: receiving from a user of a smart glass, an indication of an object of interest within a forward-image viewed by the user, comparing a region of interest centered on the object of interest with a field of view of a first forward-looking camera mounted on a frame of the smart glass, and providing, via an interface actuator mounted on the smart glass, a feedback to the user based on a relative position between the region of interest and the field of view of the first forward-looking camera, optionally wherein, the feedback is intended to have the user adjust a head position relative to the object of interest.
  • receiving the indication of an object of interest within the field of view of the user may include tracking a location of a pupil of the user and identifying the object of interest based on the location of the pupil of the user.
  • receiving the indication of an object of interest within the field of view of the user may include receiving an eye position for a user of a smart glass, the eye position indicative of a position of the object of interest within the field of view of the user.
  • receiving the indication of an object of interest within the field of view of the user may include determining a gaze direction of the user based on a position of two pupils of the user.
  • receiving the indication of an object of interest within the field of view of the user may include receiving a position of two pupils of the user, and determining a gaze direction comprises determining a vergence of the position of two pupils of the user.
  • the computer-implemented method may further comprise determining the region of interest having an area of the field of view of the first forward-looking camera centered on the object of interest.
  • the smart glass may include a second forward-looking camera, further comprising receiving, from the user, and instruction to turn the second forward-looking camera on when the region of interest is misaligned with the field of view of the first forward- looking camera.
  • the smart glass may include a second forward-looking camera, and wherein providing a feedback to the user comprises determining a relative position of the region of interest and a field of view of the second forward-looking camera.
  • the smart glass may include a speaker, and providing a feedback to the user comprises providing an instruction to the user to move the smart glass in a given direction.
  • the smart glass may include a light emitter, and providing a feedback to the user comprises indicating a direction to move a gaze of the user with the light emitter.
  • a computer-implemented method for a camera interface in a smart glass includes identifying a mismatch between a region of interest centered in an object of interest for the user and the field of view of a first camera mounted on the smart glass.
  • the computer-implemented method includes automatically updating a position, direction or other setting of the first camera, or selecting a second camera on the smart glass that is better positioned relative to the region of interest, without notifying the user.
  • a system in another embodiment, includes one or more processors and a memory storing instructions which, when executed by the one or more processors, cause the system to perform a method.
  • the method includes receiving from a user of a smart glass, an indication of an object of interest within a field of view of the user, comparing a region of interest centered on the object of interest with a field of view of a first forward-looking camera mounted on a frame of the smart glass, and providing, via an interface actuator mounted on the smart glass, a feedback to the user based on a relative position between the region of interest and the field of view of the first forward-looking camera, the feedback intended to have the user adjust a head position relative to the object of interest.
  • a system in yet another embodiment, includes a first means to store instructions, and a second means to execute the instructions and cause the system to perform a method.
  • the method includes receiving from a user of a smart glass, an indication of an object of interest within a field of view of the user, comparing a region of interest centered on the object of interest with a field of view of a first forward-looking camera mounted on a frame of the smart glass, and providing, via an interface actuator mounted on the smart glass, a feedback to the user based on a relative position between the region of interest and the field of view of the first forward-looking camera.
  • FIG. 1 illustrates an architecture including one or more wearable devices coupled to one another, to a mobile device, a remote server and to a database, according to some embodiments.
  • FIGS. 2A-2C illustrate a scene viewed by a user wearing a smart glass, according to some embodiments.
  • FIG. 3 illustrates a visual indicator in the eyepiece of a smart glass to help the user adjust a camera field of view based on a region of interest for image capture, according to some embodiments.
  • FIG. 4 illustrates forward-looking cameras in a smart glass, and an interference on a field of view of one of the forward-looking cameras, according to some embodiments.
  • FIG. 5 illustrates a smart glass having a glass shading capability, according to some embodiments.
  • FIGS. 6A-6B illustrate examples of a spatial modulation of the clear state and the shaded state across the eyepieces, according to some embodiments.
  • FIGS. 7A-7B illustrate a smart glass with the two eyepieces in a single transmissive state facing a scene, according to some embodiments.
  • FIG. 8 is a flow chart illustrating steps in a method for matching a region of interest with a field of view of a camera in a smart glass, according to some embodiments.
  • FIG. 9 is a block diagram illustrating an exemplary computer system with which a headset and methods for use of the same can be implemented, according to some embodiments.
  • elements having the same or similar label number share the same or similar features, unless stated explicitly otherwise.
  • a system includes a smart glass with a built-in or attached camera.
  • the system may or may not have a display for the user. If a display exists, it may be unavailable for the camera field of view, or it may be preferably devoted for other utilities.
  • a system includes a smart glass with a built-in or attached camera.
  • a smart glass device as disclosed herein includes a frame including eyepieces to provide a field of view to a user, a forward-looking camera mounted on the frame, a sensing device facing the user for tracking a pupil position of the user, a processing circuit configured to identify a region of interest (ROI) within the field of view of the user, and an interface device to indicate to the user that a field of view of the forward-looking camera is misaligned with the region of interest.
  • a memory may store instructions which, when executed by the processor, cause the smart glass to perform one or more steps in a method consistent with the present disclosure.
  • the memory may also store data, such as calibration data for the position and orientation of the forward-looking camera relative to the user field of view.
  • Embodiments as disclosed herein use eye tracking tools to identify a region of interest in a scene viewed through a smart glass as disclosed above.
  • embodiments as disclosed herein read user gesture to identify a region of interest in a scene viewed through a smart glass as disclosed above.
  • a system is designed such that the camera captures the scene, including some example hand gestures from the user.
  • the frames including hand gestures are run through a gesture recognition system to recognize the moment when the appropriate gesture is presented by the user, and identify the region of interest based on a reading of the hand gesture from the user.
  • FIG. 1 illustrates an architecture 10 including one or more wearable devices 100-1 (e.g., a smart glass) and 100-2 (e.g., a smart watch) coupled to one another (hereinafter, collectively referred to as “wearable devices 100”), to a mobile device 110, a remote server 130, and to a database 152, according to some embodiments.
  • Smart glass 100-1 may be configured for AR/VR applications, and mobile device 110 may be a smart phone, all of which may communicate with one another via wireless communications and exchange a first dataset 103- 1.
  • Dataset 103-1 may include a recorded video, audio, or some other file or streaming media.
  • a user 101 of wearable devices 100 is also the owner or is associated with mobile device 110.
  • the smart glass may directly communicate with the remote server, the database, or any other client device (e.g. , a smart phone of a different user, and the like) via the network.
  • the mobile device may be communicatively coupled with a remote server and a database via a network 150, and transmit/share information, files, and the like with one another, e.g., dataset 103-2 and dataset 103-3 (hereinafter, collectively referred to as “datasets 103”).
  • Network 150 may include, for example, any one or more of a local area network (LAN), a wide area network (WAN), the Internet, and the like.
  • the network can include, but is not limited to, any one or more of the following network topologies, including a bus network, a star network, a ring network, a mesh network, a star-bus network, tree or hierarchical network, and the like.
  • Smart glass 100-1 may include a frame 105 including eyepieces 107 to provide a forward-image to user 101.
  • a camera 115 e.g., forward-looking
  • a sensing device 128 facing the user is configured to track a pupil position of the user.
  • Processor 112 is configured to identify a region of interest (ROI) of user 101.
  • An interface device 129 indicates to user 101 that an FOV of camera 115 is misaligned with the ROI.
  • smart glass 100-1 may automatically identify a misalignment between the ROI and the FOV of camera 115 by applying algorithms such as object recognition or scene understanding.
  • smart glass 100-1 may also include a haptic actuator 125 to recreate a sense of touch to the user, for a VR/AR application, and a speaker 127 to communicate to user 101 voice or sound signals indicative of adjusting a gaze direction for improving the FOV of camera 115.
  • haptic actuator 125 may include a vibrating component to indicate to the user to nudge their head position in a desired direction to align the FOV of forward-looking camera 115 with the ROI (e.g. , apply a vibration on the left temple of the user to shift the user gaze slightly to the left, and so forth).
  • smart glass 100-1 may include multiple sensors 121 such as IMUs, gyroscopes, microphones, and capacitive sensors configured as touch interfaces for the user.
  • sensors 121 such as IMUs, gyroscopes, microphones, and capacitive sensors configured as touch interfaces for the user.
  • Other touch sensors may include a pressure sensor, a thermometer, and the like.
  • wearable devices 100 may include a memory circuit 120 storing instructions, and a processor circuit 112 configured to execute the instructions to cause smart glass 100-1 to perform, at least partially, some of the steps in methods consistent with the present disclosure.
  • Memory circuit 120 may also store data, such as calibration data for the position and orientation of camera 115 relative to the FOV of the user.
  • smart glass 100-1, mobile device 110, server 130, and/or database 152 may further include a communications module 118 enabling the device to wirelessly communicate with remote server 130 via network 150. Smart glass 100-1 may thus download a multimedia online content (e.g., dataset 103-1) from remote server 130, to perform at least partially some of the operations in methods as disclosed herein.
  • memory 120 may include instructions to cause processor 112 to receive and combine signals from sensors 121, avoid false positives, and better assess user intentions and commands when an input signal is received from a user interface.
  • smart watch, or wrist band device 100-2 may include contact and motion sensors that capture a hand gesture of user 101.
  • the hand gesture may be to increase, expand, pan, or shift an FOV of one or more of cameras 115 according to an ROI of user 101.
  • processor 112 may be configured to identify the hand gestures of the user as provided by any one of the sensors in wearable devices 100, and adjust the FOV of camera 115 accordingly.
  • FIGS. 2A-2C illustrate a scene 210 viewed by a user wearing a smart glass 200, according to some embodiments.
  • a user gaze e.g., gaze vector 229 A
  • the user may be glancing towards the picture frame (e.g., ROI 238).
  • the center of the ROI 238 is selected as the vergence point of gaze vector 229 A.
  • gaze vector 229A is aligned with the FOV of a first forward-looking camera 215-1 or a second forward -looking camera 215-2 (hereinafter, collectively referred to as “forward looking cameras 215”), shown here with a box with yellow dotted lines, according to some embodiments.
  • a prior calibration setting stored in a memory may allow the system to estimate a desirable framing to capture an object of interest within ROI 238.
  • the system could provide feedback to the user indicating that ROI 238 is outside the FOV of one of forward-looking cameras 215.
  • This indication could be provided, for example, using audio cues from a microphone 227 audible to the user of smart glass 200 or haptics on the right side of the spectacles, or some other means.
  • the user may then choose to move their head with eyepieces 207 and forward-looking cameras 215, thereby bringing ROI 238 into the field of view of forward-looking cameras 215.
  • the system may give a “green light” notification to indicate that ROI 238 has been communicated to the capture system.
  • ROI 238 for a user of a smart glass is selected according to some embodiments.
  • the system may capture/trigger still capture, or video capture.
  • the system may apply optimization techniques to the region of interest (e.g., exposure optimization, camera focus optimization, and the like), or some other module/event that benefits from the region of interest information.
  • FIG. 3 illustrates a visual indicator 350 in an eyepiece 307 of a smart glass 300 to help the user adjust an FOV 338 of a first forward-looking camera 315-1 based on a region of interest for image capture, according to some embodiments.
  • the scene 310 may be captured also by a second forward-looking camera 315-2 (hereinafter, first and second forward-looking cameras 315-1 and 315-2 will be referred to, hereinafter, as “cameras 315”).
  • the system may identify that gaze vector 329 indicates an object outside of FOV 338.
  • the system may include an eye tracking sensor 328. Accordingly, the system may either indicate to the user to move their head slightly in a given direction ( e.g . , to the right and slightly up), or may determine that an FOV of second forward-looking camera 315-2 is better aligned with the ROI of the user. Accordingly, the system may switch second forward- looking camera 315-2 on and first forward-looking camera 315-1 off.
  • FIG. 4 illustrates forward-looking cameras 415-1 and 415-2 (hereinafter, collectively referred to as “forward-looking cameras 415”) in a smart glass 400, and an interference 410 on an FOV 438 of at least one of forward-looking cameras 415, according to some embodiments.
  • Smart glass 400 includes eyepieces 407.
  • interference 410 may be the visor of a cap the user is wearing, which inadvertently gets in the way of FOV 438.
  • the system may automatically crop interference 410 out of the image and advise the user that auto-crop has been used.
  • a processor 412 may identify the interference and warn the user via a speaker 427 that a cap or “a foreign object” is in the way of FOV 438.
  • the user may in fact have collected multiple images with camera 415-1 including interference 410. Accordingly, the images may be uploaded to a mobile device via a wireless data transfer ( cf mobile device 110 and dataset 103-1). An algorithm running on a processor in mobile device 110 may identify interference 410 and provide the voice advisory to smart glass 400 via microphone 427.
  • a gaze vector 429 determined using an eye tracking sensor 428 may have selected FOV 438 from first forward-looking camera 415-1.
  • eye tracking sensor 428 may not identify or even sense interference 410, which may also block a portion of an FOV of second forward-looking camera 415-2.
  • the user may adjust the cap, or remove it completely.
  • FOV 438 is free from interference 410, the system may issue a clearance notification indicating to the user that a picture can now be taken with either one of forward-looking cameras 415.
  • FIG. 5 illustrates a smart glass 500 having a glass shading capability, according to some embodiments.
  • active liquid crystals having polarizing components can be switched to be more or less transmissive to light passing through the lens or eyepiece of the glasses. Accordingly, a clear state 501A (high transparency) and a shaded state 501B (low transparency, “sunglass” state) may be provided to eyepieces 507. Spatially modulating this property across eyepieces 507 can be used to provide feedback to the user as to the relative position of the camera field of view with the forward-image viewed by the user.
  • each of eyepieces 507 includes an actuator configured to modify the optical transparency of at least a portion of the eyepiece.
  • Smart glass also includes a camera 515 mounted on frame 505. Camera 515 faces forward towards the user FOV.
  • smart glass 500 also includes a processor 512 configured to compare an FOV of camera 515 with the forward-image viewed by the user. Processor 512 may be configured to indicate to the user that the FOV of the camera is missing a portion of the forward-image viewed by the user by adjusting the optical transparency of at least a portion of the eyepieces 507.
  • a memory 520 stores instructions which, when executed by processor 512, cause smart glass 500 to perform steps in methods as disclosed herein.
  • FIGS. 6A-6B illustrate examples of a spatial modulation of a partial vertical clear state 601 A and a partial horizontal clear state 60 IB (hereinafter, collectively referred to as “clear states 601”) and a partial vertical shaded state 602A and a partial horizontal clear state 602B (hereinafter, collectively referred to as “shaded states 602”) across eyepieces 607, according to some embodiments.
  • Smart glass 600 may include one or two forward-looking cameras 615-1 and 615-2 (hereinafter, collectively referred to as “cameras 615”). Either one of cameras 615, or both, may be active at any point in time, to ensure a better capture of a forward image in view of a user ROI.
  • the system may be configured to keep only one of cameras 615 ‘on’ at any given time.
  • a suitably designed liquid crystal may be configured to switch in a localized manner within each eyepiece 607.
  • vertical sections of eyepieces 607 can be shaded, and made more or less transmissive, to provide feedback to the user.
  • the feedback may be indicative of an FOV of one of cameras 615 that is active. For example, when shading 601 A of eyepieces 607 provides a clear vertical stripe in the middle of the forward-image viewed by the user, this may be an indication that the camera FOV is properly centered in the horizontal direction.
  • shifting the partial clear state 601 A left or right along eyepieces 607 may give a clear indication to the user that the frustum of one of active cameras 615 is misaligned in the horizontal direction and that the head pose might need adjustment prior to collecting an image or video by moving left or right, respectively, or that the use should activate the other one of cameras 615.
  • a clear, horizontal stripe 601B across the middle of the user field of view might be an indication that the field of view of one of cameras 615 is properly centered in the vertical direction.
  • Horizontal stripe 601B is delimited above and below by semi transparent or “shaded” stripes 602B .
  • shifting the clear stripe 60 IB up or down the eyepieces may give a clear indication to the user that the field of view of one of cameras 615 is misaligned in the vertical direction and that the head pose might need adjustment prior to collecting an image or video.
  • eyepieces 607 with such a localized transmission switching state can be used to provide the user some broad level of feedback as to what is seen by cameras 615 mounted on smart glass 600.
  • an eye tracking sensor 628 may be used to identify an ROI for the user, in which case shading 601 is adjusted accordingly.
  • some embodiments may include side and top/bottom bars in photo/video capture mode, the user will automatically understand how the camera is framing, even without partially shading the display in the eyepieces.
  • FIGS. 7A-7B illustrate a smart glass 700 with two eyepieces 707 in a single transmissive state facing a scene, according to some embodiments.
  • a field of view 738 of an entire scene 710 viewed by the user of smart glass 700 is captured by a camera 715 that is mounted on a frame 705 of smart glass 700.
  • embodiments as disclosed herein help the user have a sense of the extent of the camera coverage 748 of the scene at the time of capture without the need for a viewfmder/display that shows the camera’s perspective, as shown below.
  • eyepieces 707 have a localized switchable transmission property, so a (vertical) portion at the outer edges of the eyepieces is in a darker transmission state 701B, according to some embodiments (cf smart glass 700). And a clearer portion 701 A overlaps portion 738 (e.g, the field of view of camera 715). Accordingly, portion 701A will appear in the picture, and portion 701B will be cropped from the camera capture.
  • the position and size of the portion 701 A could be calibrated to the camera’s perspective, based on the position and orientation of the camera relative to the eyepieces. Hence, the user could be informed that the portion of the view that is clearly visible will be captured by the camera.
  • camera FOV 738 covers the user’s ROI
  • the user may activate the image collection.
  • camera FOV 738 does not cover the user’s ROI
  • the user may move the head to adjust camera FOV 738.
  • the user may select a different camera to use for image capture, and the system may indicate to the user the size and position of the field of view of the selected camera, prior to collecting an image.
  • FIG. 8 is a flow chart illustrating steps in a method 800 for matching a region of interest with a field of view of a camera in a smart glass, according to some embodiments (e.g., smart glasses 100-1, 200, 300, 400, 500, 600, and 700, and cameras 115, 215, 315, 415, 515, 615, and 715).
  • the smart glass may include, in addition to the camera, an eye tracking sensor, a memory storing instructions and a processor configured to execute the instructions to perform at least partially one or more steps in method 800 (e.g., sensing devices 128, 228, 328, and 428, memories 120 and 520, and processors 112 and 512).
  • a method consistent with the present disclosure may include at least one of the steps in method 800, or two or more steps in method 800 performed in a different order, simultaneously, quasi-simultaneously, or overlapping in time.
  • Step 802 includes receiving, from a user of a smart glass, an instruction indicative of an object of interest within a field of view of the user.
  • step 802 includes tracking a location of a pupil of the user and identifying the object of interest based on the location of the pupil of the user.
  • step 802 includes receiving an eye position for a user of a smart glass, the eye position indicative of a position of the object of interest within the field of view of the user.
  • step 802 includes determining a gaze direction of the user based on a position of two pupils of the user.
  • step 802 includes receiving a position of two pupils of the user and determining a gaze direction comprises determining a vergence of the position of two pupils of the user.
  • step 802 may include displaying, in the eyepieces of the smart glass, two converging lines indicative of the gaze direction, converging on the apparent object of interest of the user, illustrating the user what the new crop is for the camera image.
  • step 802 includes determining the region of interest having an area of the field of view of the first forward-looking camera centered on the object of interest.
  • the smart glass includes a second forward-looking camera, and step 802 further includes receiving, from the user, an instruction to turn the second forward-looking camera on when the region of interest is misaligned with the field of view of the first forward-looking camera.
  • Step 804 includes comparing a region of interest centered on the object of interest with a field of view of a first forward-looking camera mounted on a frame of the smart glass.
  • Step 806 includes providing, via an interface actuator mounted on the smart glass device, a feedback to the user based on a relative position between the region of interest and the field of view of the first forward-looking camera.
  • the smart glass includes a second forward-looking camera, and step 806 includes determining a relative position of the region of interest and a field of view of the second forward-looking camera.
  • the smart glass includes a speaker, and step 806 includes providing an instruction to the user to move the smart glass in a given direction.
  • the smart glass includes a light emitter, and step 806 includes indicating a direction to move a gaze of the user with the light emitter.
  • FIG. 9 is a block diagram illustrating an exemplary computer system 900 with which headset 100 of FIG. 1, and method 800 can be implemented, according to some embodiments.
  • computer system 900 may be implemented using hardware or a combination of software and hardware, either in a dedicated server, or integrated into another entity, or distributed across multiple entities.
  • Computer system 900 may include a desktop computer, a laptop computer, a tablet, a phablet, a smartphone, a feature phone, a server computer, or otherwise.
  • a server computer may be located remotely in a data center or be stored locally.
  • Computer system 900 includes a bus 908 or other communication mechanism for communicating information, and a processor 902 (e.g., processor 112) coupled with bus 908 for processing information.
  • processor 902 e.g., processor 112
  • the computer system 900 may be implemented with one or more processors 902.
  • Processor 902 may be a general-purpose microprocessor, a microcontroller, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), a controller, a state machine, gated logic, discrete hardware components, or any other suitable entity that can perform calculations or other manipulations of information.
  • DSP Digital Signal Processor
  • ASIC Application Specific Integrated Circuit
  • FPGA Field Programmable Gate Array
  • PLD Programmable Logic Device
  • controller a state machine, gated logic, discrete hardware components, or any other suitable entity that can perform calculations or other manipulations of information.
  • Computer system 900 can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them stored in an included memory 904 (e.g., memory 120), such as a Random Access Memory (RAM), a flash memory, a Read-Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable PROM (EPROM), registers, a hard disk, a removable disk, a CD-ROM, a DVD, or any other suitable storage device, coupled with bus 908 for storing information and instructions to be executed by processor 902.
  • the processor 902 and the memory 904 can be supplemented by, or incorporated in, special purpose logic circuitry.
  • the instructions may be stored in the memory 904 and implemented in one or more computer program products, e.g., one or more modules of computer program instructions encoded on a computer-readable medium for execution by, or to control the operation of, the computer system 900, and according to any method well known to those of skill in the art, including, but not limited to, computer languages such as data-oriented languages (e.g., SQL, dBase), system languages (e.g., C, Objective-C, C++, Assembly), architectural languages (e.g., Java, .NET), and application languages (e.g., PHP, Ruby, Perl, Python).
  • data-oriented languages e.g., SQL, dBase
  • system languages e.g., C, Objective-C, C++, Assembly
  • architectural languages e.g., Java, .NET
  • application languages e.g., PHP, Ruby, Perl, Python.
  • Instructions may also be implemented in computer languages such as array languages, aspect-oriented languages, assembly languages, authoring languages, command line interface languages, compiled languages, concurrent languages, curly-bracket languages, dataflow languages, data-structured languages, declarative languages, esoteric languages, extension languages, fourth-generation languages, functional languages, interactive mode languages, interpreted languages, iterative languages, list-based languages, little languages, logic-based languages, machine languages, macro languages, metaprogramming languages, multiparadigm languages, numerical analysis, non-English-based languages, object-oriented class-based languages, object-oriented prototype-based languages, off-side rule languages, procedural languages, reflective languages, rule-based languages, scripting languages, stack-based languages, synchronous languages, syntax handling languages, visual languages, wirth languages, and xml-based languages.
  • Memory 904 may also be used for storing temporary variable or other intermediate information during execution of instructions to be executed by processor 902.
  • a computer program as discussed herein does not necessarily correspond to a file in a file system.
  • a program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subprograms, or portions of code).
  • a computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
  • the processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output.
  • Computer system 900 further includes a data storage device 906 such as a magnetic disk or optical disk, coupled with bus 908 for storing information and instructions.
  • Computer system 900 may be coupled via input/output module 910 to various devices.
  • Input/output module 910 can be any input/output module.
  • Exemplary input/output modules 910 include data ports such as USB ports.
  • the input/output module 910 is configured to connect to a communications module 912.
  • Exemplary communications modules 912 include networking interface cards, such as Ethernet cards and modems.
  • input/output module 910 is configured to connect to a plurality of devices, such as an input device 914 and/or an output device 916.
  • Exemplary input devices 914 include a keyboard and a pointing device, e.g., a mouse or a trackball, by which a consumer can provide input to the computer system 900.
  • Other kinds of input devices 914 can be used to provide for interaction with a consumer as well, such as a tactile input device, visual input device, audio input device, or brain-computer interface device.
  • feedback provided to the consumer can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the consumer can be received in any form, including acoustic, speech, tactile, or brain wave input.
  • Exemplary output devices 916 include display devices, such as an LCD (liquid crystal display) monitor, for displaying information to the consumer.
  • wearable devices 100 can be implemented, at least partially, using a computer system 900 in response to processor 902 executing one or more sequences of one or more instructions contained in memory 904. Such instructions may be read into memory 904 from another machine-readable medium, such as data storage device 906. Execution of the sequences of instructions contained in main memory 904 causes processor 902 to perform the process steps described herein. One or more processors in a multi-processing arrangement may also be employed to execute the sequences of instructions contained in memory 904. In alternative aspects, hard-wired circuitry may be used in place of or in combination with software instructions to implement various aspects of the present disclosure. Thus, aspects of the present disclosure are not limited to any specific combination of hardware circuitry and software.
  • a computing system that includes a back end component, e.g., a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical consumer interface or a Web browser through which a consumer can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components.
  • the components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network.
  • the communication network can include, for example, any one or more of a LAN, a WAN, the Internet, and the like. Further, the communication network can include, but is not limited to, for example, any one or more of the following network topologies, including a bus network, a star network, a ring network, a mesh network, a star-bus network, tree or hierarchical network, or the like.
  • the communications modules can be, for example, modems or Ethernet cards.
  • Computer system 900 can include clients and servers.
  • a client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
  • Computer system 900 can be, for example, and without limitation, a desktop computer, laptop computer, or tablet computer. Computer system 900 can also be embedded in another device, for example, and without limitation, a mobile telephone, a PDA, a mobile audio player, a Global Positioning System (GPS) receiver, a video game console, and/or a television set top box.
  • GPS Global Positioning System
  • machine-readable storage medium or “computer-readable medium” as used herein refers to any medium or media that participates in providing instructions to processor 902 for execution. Such a medium may take many forms, including, but not limited to, non volatile media, volatile media, and transmission media.
  • Non-volatile media include, for example, optical or magnetic disks, such as data storage device 906.
  • Volatile media include dynamic memory, such as memory 904.
  • Transmission media include coaxial cables, copper wire, and fiber optics, including the wires forming bus 908.
  • Machine-readable media include, for example, floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, an EPROM, a FLASH EPROM, any other memory chip or cartridge, or any other medium from which a computer can read.
  • the machine-readable storage medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter affecting a machine-readable propagated signal, or a combination of one or more of them.
  • phrases “at least one of’ does not require selection of at least one item; rather, the phrase allows a meaning that includes at least one of any one of the items, and/or at least one of any combination of the items, and/or at least one of each of the items.
  • the phrases “at least one of A, B, and C” or “at least one of A, B, or C” each refer to only A, only B, or only C; any combination of A, B, and C; and/or at least one of each of A, B, and C.
  • exemplary is used herein to mean “serving as an example, instance, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments. Phrases such as an aspect, the aspect, another aspect, some aspects, one or more aspects, an implementation, the implementation, another implementation, some implementations, one or more implementations, an embodiment, the embodiment, another embodiment, some embodiments, one or more embodiments, a configuration, the configuration, another configuration, some configurations, one or more configurations, the subject technology, the disclosure, the present disclosure, and other variations thereof and alike are for convenience and do not imply that a disclosure relating to such phrase(s) is essential to the subject technology or that such disclosure applies to all configurations of the subject technology.
  • a disclosure relating to such phrase(s) may apply to all configurations, or one or more configurations.
  • a disclosure relating to such phrase(s) may provide one or more examples.
  • a phrase such as an aspect or some aspects may refer to one or more aspects and vice versa, and this applies similarly to other foregoing phrases.
  • a reference to an element in the singular is not intended to mean “one and only one” unless specifically stated, but rather “one or more.”
  • the term “some” refers to one or more.
  • Underlined and/or italicized headings and subheadings are used for convenience only, do not limit the subject technology, and are not referred to in connection with the interpretation of the description of the subject technology. Relational terms such as first and second and the like may be used to distinguish one entity or action from another without necessarily requiring or implying any actual such relationship or order between such entities or actions.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Human Computer Interaction (AREA)
  • Optics & Photonics (AREA)
  • Automation & Control Theory (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • General Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

L'invention concerne un dispositif portable destiné à être utilisé dans des applications de réalité immersive. Le dispositif portable comprend des oculaires pour fournir une image directe à un utilisateur, une première caméra orientée vers l'avant montée sur la monture et ayant un champ de vision, un processeur configuré pour identifier une région d'intérêt à l'intérieur de l'image avant, et un dispositif d'interface pour indiquer à l'utilisateur qu'un champ de vision de la première caméra orientée vers l'avant est aligné avec la région d'intérêt. L'invention concerne également des procédés d'utilisation du dispositif, une mémoire stockant des instructions et un processeur pour exécuter les instructions pour amener le dispositif à effectuer les procédés d'utilisation.
PCT/US2022/038933 2021-07-29 2022-07-29 Interface utilisateur pour sélectionner un champ de vision d'une caméra dans des lunettes intelligentes Ceased WO2023009853A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202280051850.7A CN117769694A (zh) 2021-07-29 2022-07-29 用于选择智能眼镜中的摄像头的视场的用户接口
EP22769424.7A EP4377771A1 (fr) 2021-07-29 2022-07-29 Interface utilisateur pour sélectionner un champ de vision d'une caméra dans des lunettes intelligentes

Applications Claiming Priority (8)

Application Number Priority Date Filing Date Title
US202163227238P 2021-07-29 2021-07-29
US202163227240P 2021-07-29 2021-07-29
US202163227231P 2021-07-29 2021-07-29
US63/227,240 2021-07-29
US63/227,238 2021-07-29
US63/227,231 2021-07-29
US17/859,808 US12132983B2 (en) 2021-07-29 2022-07-07 User interface to select field of view of a camera in a smart glass
US17/859,808 2022-07-07

Publications (1)

Publication Number Publication Date
WO2023009853A1 true WO2023009853A1 (fr) 2023-02-02

Family

ID=83283300

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2022/038933 Ceased WO2023009853A1 (fr) 2021-07-29 2022-07-29 Interface utilisateur pour sélectionner un champ de vision d'une caméra dans des lunettes intelligentes

Country Status (1)

Country Link
WO (1) WO2023009853A1 (fr)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180157045A1 (en) * 2016-12-06 2018-06-07 Google Llc Ocular Video Stabilization
US20190011703A1 (en) * 2016-07-25 2019-01-10 Magic Leap, Inc. Imaging modification, display and visualization using augmented and virtual reality eyewear
US20210081047A1 (en) * 2019-09-13 2021-03-18 Apple Inc. Head-Mounted Display With Haptic Output
US11047693B1 (en) * 2015-09-11 2021-06-29 Philip Raymond Schaefer System and method for sensing walked position

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11047693B1 (en) * 2015-09-11 2021-06-29 Philip Raymond Schaefer System and method for sensing walked position
US20190011703A1 (en) * 2016-07-25 2019-01-10 Magic Leap, Inc. Imaging modification, display and visualization using augmented and virtual reality eyewear
US20180157045A1 (en) * 2016-12-06 2018-06-07 Google Llc Ocular Video Stabilization
US20210081047A1 (en) * 2019-09-13 2021-03-18 Apple Inc. Head-Mounted Display With Haptic Output

Similar Documents

Publication Publication Date Title
US12132983B2 (en) User interface to select field of view of a camera in a smart glass
CN103475893B (zh) 三维显示中对象的拾取装置及三维显示中对象的拾取方法
US11102413B2 (en) Camera area locking
US20180103194A1 (en) Image capture systems, devices, and methods that autofocus based on eye-tracking
US11736679B2 (en) Reverse pass-through glasses for augmented reality and virtual reality devices
CN102402379A (zh) 移动终端及其控制方法
EP3574408A1 (fr) Structure d'antémémoire sans requêtes infructueuses pour transformations d'image en temps réel
US12166955B2 (en) Reverse pass-through glasses for augmented reality and virtual reality devices
CN117769694A (zh) 用于选择智能眼镜中的摄像头的视场的用户接口
WO2023009853A1 (fr) Interface utilisateur pour sélectionner un champ de vision d'une caméra dans des lunettes intelligentes
US12101549B2 (en) Camera control using system sensor data
US12501150B2 (en) User interface to select field of view of a camera in a smart glass
WO2023009806A1 (fr) Interface utilisateur pour sélectionner un champ de vision d'une caméra dans des lunettes intelligentes
WO2023283323A1 (fr) Commande de caméra à l'aide de données de capteur de système
CN120730108A (zh) 画面显示方法、装置、设备及存储介质
WO2025165347A1 (fr) Recommandations de capture d'image de réalité étendue à l'aide d'avatars

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22769424

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202280051850.7

Country of ref document: CN

WWE Wipo information: entry into national phase

Ref document number: 2022769424

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2022769424

Country of ref document: EP

Effective date: 20240229