[go: up one dir, main page]

WO2011025450A1 - Procédés et systèmes pour interaction visuelle - Google Patents

Procédés et systèmes pour interaction visuelle Download PDF

Info

Publication number
WO2011025450A1
WO2011025450A1 PCT/SE2010/050916 SE2010050916W WO2011025450A1 WO 2011025450 A1 WO2011025450 A1 WO 2011025450A1 SE 2010050916 W SE2010050916 W SE 2010050916W WO 2011025450 A1 WO2011025450 A1 WO 2011025450A1
Authority
WO
WIPO (PCT)
Prior art keywords
expert
image
user
camera
unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/SE2010/050916
Other languages
English (en)
Inventor
Per Carleberg
Torbjörn GUSTAFSSON
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
XMREALITY RESEARCH AB
Original Assignee
XMREALITY RESEARCH AB
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from SE0950609A external-priority patent/SE537143C2/sv
Application filed by XMREALITY RESEARCH AB filed Critical XMREALITY RESEARCH AB
Publication of WO2011025450A1 publication Critical patent/WO2011025450A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking

Definitions

  • the present document relates to systems and methods for remote visual interaction between, for example, a user and an expert.
  • the user who generally has less knowledge than the expert about a specific piece of equipment, carries on a visual and verbal discussion with the expert, who is located at a distance, as to how certain actions are to be performed.
  • apparatus has a tendency to become more and more complicated and complex.
  • apparatus usually- some form of equipment, system or product of a technical nature is meant here.
  • a biological being (construction) is also contained within the term “apparatus”, such as a human being (e.g., a patient in a medical consultation) .
  • Apparatus most often requires, for its maintenance and repair, a person who is well trained and very familiar with how such apparatus functions and how it should be handled in various situations. The person should also have practical experience with the handling of the apparatus. Such a person, with the aforementioned skills, is designated here by the term "expert".
  • a person with more general knowledge about the apparatus is designated here by the term "user".
  • the user can in many cases, using his general knowledge and experience, repair and maintain the apparatus. In some cases, however, the user's knowledge is not enough, and the user must then turn to the expert for guidance.
  • the apparatus can exist in several units and these can be geographically spread out. Most often, an apparatus is repaired and maintained by persons (users) who are situated at the actual location of the apparatus, or by persons (users) whom a firm or organization sends out to the actual location.
  • the expert should be present in some way at a central location in a company or organization so that as many persons as possible can quickly obtain help from him or them. It can be both costly and impractical to send out an expert on different assignments in order to help a user. Through modern technology, however, it is possible to let the expert be virtually present at the user's location and to guide him visually and verbally.
  • FIG. 1 One example of such a known system and method, which has been shown at exhibitions and described in report form, consists of two items of headborne microdisplay equipment (presentation device) with an integrated video camera.
  • the user wears the one item of equipment and the expert the other.
  • the user's and expert's equipment are interconnected by means of computers and transmission channels.
  • the expert's hand, arm, or some other object which the expert has placed on a work surface can be extracted from the image from the expert's video camera and superimposed in the image that is seen by the user's camera and describes what is in front of the user.
  • the principle of the known system is shown in Fig. 1, where:
  • • 102 is a video camera that is integrated in a user' s presentation device
  • • 103 is a computer for a user to use
  • • 202 is a video camera that is integrated in an expert's presentation device • 203 is a computer for an expert to use
  • • 300 is a transmission channel between a user and an expert.
  • User and expert also each have a unit for verbal communication, which includes a microphone and loudspeaker device. This unit is not shown in the figure.
  • the transmission channel's delay can further make the visual interaction difficult, in the same way as with head movements.
  • the expert may be guiding several users at the same time, i.e., switching between different users, and is then forced to spend relatively long periods of time wearing the headborne equipment. The expert may then find his equipment inconvenient and fatiguing.
  • one object of the present document is to provide systems and methods which facilitate - A - remote visual interaction between, for example, an expert and at least one user in an interaction system.
  • One particular object is to provide a method and system for visual interaction where the expert and the user can interact in as natural a way as possible.
  • a method for remote visual interaction between an expert and at least one user in an interaction system comprising: at least one workstation comprising a presentation device and a user camera for use by the user, wherein said workstation is operatively connected to a support station comprising an expert presentation device and an expert camera unit for use by the expert.
  • the method comprises using the user camera to capture the user' s field of vision in order to form a user image, presenting the user image to the expert by means of the expert presentation device, using the expert camera unit to depict an object at the support station as an expert object image, mixing together the expert object image with the user image to form a mixed image, and presenting the mixed image to the user via the user presentation device.
  • the method also comprises using the expert camera unit to depict the object when this is at least partly located essentially between the expert camera unit and the expert presentation device, creating an image of the object, constituting a subset of the expert object image, and creating the mixed image based on the user image and the depiction of the object .
  • the expert can give the user instructions in a natural and simple way by pointing directly in the image that the user is seeing.
  • the depicting of the object can be achieved by means of an object camera and a mask camera.
  • the mask camera can be adapted to a wavelength range that differs from a wavelength range to which the object camera is adapeted.
  • the mask camera can be a near IR camera.
  • a system for remote visual interaction between an expert and at least one user in an interaction system.
  • the system comprises at least one workstation comprising a presentation device and a user camera for use by the user, wherein said workstation is operatively connected to a support station comprising an expert presentation device and an expert camera unit for use by the expert, wherein the user camera is configured to capture the user' s field of vision, which is presented to the expert by means of the expert presentation device as a user image and the expert camera unit is configured to depict an object at the support station as an expert object image and mix together the expert object image with the user image and present these to the user via the user presentation device as a mixed image.
  • the expert camera unit is turned toward the expert presentation device in order to depict the object when this is at least partly located in a field of vision between the expert camera unit and the expert presentation device.
  • the expert camera unit is arranged to achieve a depiction of the object, constituting a subset of the expert object image.
  • the expert camera unit is arranged to achievel the mixed image based on the user image and the depiction of the object.
  • a system for remote visual interaction between an expert and at least one user in an interaction system.
  • the system comprises at least one workstation comprising a presentation device and a user camera for use by the user, wherein said workstation is operatively connected to a support station comprising an expert presentation device and an expert camera unit for use by the expert, wherein the user camera is configured to capture the user's field of vision, which is presented to the expert by means of the expert presentation device as a user image and the expert camera unit is configured to depict an object at the support station as an expert object image and mix together the expert object image with the user image and present these to the user via the user presentation device as a mixed image.
  • This system is characterized in that images in remote visual interaction between an expert and at least one user are synchronized by the system comprising a user computer designed to define a fixed point in the user image, an expert computer designed to define a fixed point in the expert object image, and a synchronizing unit designed to synchronize the user image and the expert image by placing the user fixed point and the expert fixed point on top of each other in the mixed image in a predetermined way.
  • a method in which images in remote visual interaction between an expert and at least one user are synchronized by the method comprising the steps of defining a fixed point in the user image, defining a fixed point in the expert object image, and synchronizing the user image and the expert object image by placing the user fixed point and the expert fixed point on top of each other in the mixed image in a predetermined way.
  • a mobile unit comprising a presentation device and a camera unit which is turned toward the presentation device in a working position in order to depict objects on or in front of the presentation device, a processing unit which is arranged to present a first image on the presentation device and to receive a second image from the camera unit, and a communication unit which is arranged to receive the first image from a remotely situated unit and send at least a portion of the second image to a remotely situated unit.
  • the presentation device is arranged in a housing and the camera unit is moveable relative to the housing between the working position and a transport position.
  • mobile unit is meant a computer of laptop type, an e-reader or the like.
  • the housing can be formed by the display screen of a laptop computer.
  • a mobile unit that can function as a mobile and compact expert station in an interaction system.
  • the processing unit can be designed to mix together the first and second image in order to form a third image, wherein the third image is sent to the remotely- situated unit.
  • the camera unit in the transport position, can be drawn into or toward the housing.
  • the camera unit can be received in a recess in the housing or otherwise conform to the shape of the housing so that the mobile unit receives an advantageous form from the transport perspective.
  • the camera unit In the working position, the camera unit can protrude from the housing.
  • the camera unit in the transport position, can be drawn into or toward an intermediate housing connected to the housing.
  • a housing can be, for example, a keyboard component of a laptop computer.
  • the camera unit In the working position, the camera unit can protrude from the intermediate housing.
  • the camera unit can comprise an object camera and a mask camera, as described below.
  • the portable unit can comprise a regenerable power source, such as a rechargeable or replaceable battery.
  • the portable unit can have a mass of less than around 20 kg, less than around 10 kg, less than around 5 kg or less than around 1 kg.
  • a station for remote visual interaction with a second station, comprising a first presentation device, arranged to present a first image, a camera unit which is arranged to depict objects on or in front of the first presentation device as a second image, a second presentation device, arranged to present a third image, and a processing unit which is arranged to form the third image by mixing together the first image and the second image.
  • the first and second presentation devices are placed adjacent to each other, and the second presentation device has a greater angle to a horizontal plane than does the first presentation device.
  • adjacent to is meant that the presentation devices are arranged in immediate proximity to each other, either integrated with each other or in the form of separate units.
  • the presentation devices can be arranged in line with each other and with a definite position for the user.
  • the first presentation device can make an angle of less than around 45 degrees with the horizontal plane, less than 30 degrees with the horizontal plane, or less than around 15 degrees with the horizontal plane.
  • the second presentation device can make an angle of more than around 45 degrees with the horizontal plane, more than around 60 degrees with the horizontal plane, or more than around 75 degrees with the horizontal plane.
  • the camera unit can comprise an object camera and a mask camera.
  • the station can be arranged to receive the first image from the second station.
  • the station can be arranged to send the third image to the second station.
  • a mobile communication unit comprising a processing unit, a communication unit for wireless communication, a presentation device comprising a microdisplay and a magnifying optics, and a camera unit, which faces in the same direction as the presentation device.
  • the processing unit is arranged to receive a first image from the camera unit and a second image via the communication unit, to mix together the first and second image in order to form a third image, and to present the third image to the user via the presentation device.
  • mobile communication unit is meant a cell phone, “smart phone”, communication radio or the like.
  • At least one of the processing unit and the communication unit can be arranged at least partly in a first housing, and the presentation unit can be arranged at least partly in a second housing, which can move relative to the first housing between a resting position and a working position.
  • the second housing can fold out, spread out, or be pulled out relative to the first housing.
  • the camera unit can be arranged at least partly in the second housing.
  • the camera unit can comprise an object camera and a mask camera.
  • a system for remote visual interaction between an expert and at least one user, comprising: a support station, which consists of a portable unit as described above or a station as described above, and a workstation comprising a user presentation device and a user camera unit, which is arranged to depict at least one portion of a user' s field of vision, wherein the user camera is arranged to generate the first image and the user presentation device is arranged to present the second image.
  • the workstation can comprise a mobile communication unit as described above.
  • the workstation can be arranged to define a fixed point in a user image from the workstation' s camera unit and the support station can be set up to define a fixed point in an expert object image from the support station's camera unit; and a synchronizing unit is set up to synchronize the user image and the expert image by placing the user fixed point and the expert fixed point on top of each other in a mixed image in a predetermined way.
  • Fig. 1 shows an example of a previously known system for remote interaction
  • Figs. 2a-2e show relevant parts of the user's equipment
  • Figs. 3-9 show relevant parts of the expert's equipment in various configurations
  • Fig. 10 shows relevant parts of the method for visual interaction
  • Figs. 11-12 show relevant parts of the method for spatial synchronization
  • Fig. 13 shows examples of image compression
  • Fig. 14 is a flow chart showing the steps of the method
  • Figs. 15a-15b are schematic images of an item of expert equipment.
  • Fig. 16 is a schematic image of another item of expert equipment .
  • Fig. 17 is a schematic image of an item of user equipment .
  • an item of headborne (helmetborne) or handheld equipment consisting of a visual presentation device, a camera unit and a fitting for attachment to the head or helmet, or handle. Note, however, that the user should be able to remove and set aside this equipment, yet secure it in some way in front of him so that the user can still receive a visual experience at the same time as the user is performing some particular work.
  • the presentation unit can be, for example:
  • a display (monitor screen) of the type found in computers, cell phones, etc.
  • the camera unit can be, for example:
  • a video camera sensitive in the visual range 2.
  • the interactive tool can be, for example:
  • buttons mechanical control by using a part of the body
  • a physical unit such as a block or plate, with patterns thereon which a camera can detect.
  • Various types of virtual objects can be coupled into the pattern, such as 3D objects, pictures, text, etc. These virtual objects can be mixed into the image of the user and/or expert.
  • the verbal communication unit can be, for example:
  • Synchronization can be accomplished, for example, in the same way as for video chat.
  • • 102 is a video camera that is integrated in a user' s presentation device
  • • 104 is a fitting for attaching the presentation device to the head or a helmet
  • 105 is a user' s handheld presentation device (microdisplay system)
  • • 106 is a fitting for a handle • 107 shows a user's handheld presentation device in the form of a monitor screen of the type found in computers, cell phones, etc.
  • • 108 shows a user' s presentation device in the form of a headborne video projector
  • • 109 shows a user's presentation device in the form of a handheld video projector.
  • the user also has a unit for verbal communication, which includes a microphone and loudspeaker device. This unit is not shown in Fig. 2.
  • the expert sits or stands at a device which can resemble a permanent installation, but it should also be able to be mobile.
  • the difference between this equipment and the user's equipment is that, for the most part, the expert's equipment is not carried by the expert in so obvious a way as the user does.
  • the expert's device consists of:
  • an interactive tool which can be, for example:
  • buttons mechanical control by using a part of the body
  • a physical unit such as a block or plate, with patterns thereon which a camera can detect.
  • Various types of virtual objects can be coupled into the pattern, such as 3D objects, pictures, text, etc. These virtual objects can be mixed into the image which the expert sees, and so the expert can for example interact and point to these virtual objects as if they were real objects.
  • the "mask camera” and "object camera” see these virtual objects virtually (see below) .
  • mask camera • another video camera, "mask camera", whose task is to mask out the object which is located on the work surface from the image being projected onto the work surface.
  • object camera can also function as the mask camera.
  • the mask camera can be placed above or below the work surface.
  • a horizontal monitor screen (such as a flat TV or flat computer monitor screen of a relatively large kind) , on which a transparent and durable glass or plastic sheet is placed.
  • An optical filter can be placed between the monitor screen and the sheet.
  • the filter can be transparent (for the most part) to visual radiation (it lets through the image on the screen so that the eye can perceive it) , but opaque (for the most part) to radiation other than visual, such as near IR
  • the filter could also be of the type that blocks certain intervals within the visible wavelength band.
  • a "back projection" surface (cloth + sheet (glass/plastic) ) on which a projector projects an image from below.
  • a filter can be placed in front of the projector's lens that only lets through visual radiation (for the most part) .
  • the filter could also be of the type that blocks certain intervals within the visible wavelength band.
  • a sheet (white or with certain patterns) above which a projector is placed ("front projection").
  • a filter can be placed in front of the projector's lens that only lets through visual radiation (for the most part) .
  • the filter could also be of the type that blocks certain intervals within the visible wavelength band.
  • V A microdisplay system which is placed between the expert's head and the expert's work surface.
  • • 206 is a horizontal monitor screen (such as a flat TV or flat computer monitor screen of a relatively large size)
  • • 207 is a video camera, "object camera”
  • optical beam splitter such as a "hot mirror” or a “cold mirror” (beam splitters with different reflecting and transmitting properties depending on wavelength interval)
  • • 210 is an optical filter, in front of the mask camera, which blocks certain wavelengths, such as the visual wavelengths
  • • 211 is a video projector
  • • 212 is an optical filter, in front of the video projector, which blocks certain wavelengths, such as near IR
  • • 213 is a beam splitter for visual wavelengths • 214 is a cloth/sheet for "back” or "front projection"
  • • 215 is a cloth/sheet acting as an optical filter which blocks certain wavelengths, such as near IR
  • • 216 is a video screen (computer screen) , of flat or other configuration
  • • 218 is a mechanical attachment device, such as a mechanical arm.
  • Figs. 3-6 The expert also has a unit for verbal communication that includes a microphone and loudspeaker device. This unit is not shown in Figs. 3- 9. Also note that in Figs. 7-9 the expert sees an image similar to the user, i.e., his own object mixed into the image that the expert sees.
  • Fig. 3 the expert, 200, stands or sits in front of a work surface, 204, on which the expert can place objects, such as a hand or a tool.
  • a work surface 204
  • objects such as a hand or a tool.
  • a flat monitor screen 206
  • an optical filter 215, which blocks near IR radiation.
  • the object camera 207 and the mask camera 208 At a distance of 50-100 cm above the work surface are placed two cameras, the object camera 207 and the mask camera 208.
  • an optical filter, 210 In front of the mask camera is placed an optical filter, 210, which blocks visual radiation but lets through near IR radiation.
  • object camera and mask camera are arranged so that they divide the optical axis by a beam splitter, a "hot mirror” which is transparent to visual radiation and reflective to near IR radiation.
  • the right-hand dashed ring of the figure shows an alternative solution with object and mask camera, where these are placed alongside each other.
  • the mask camera extracts this object out against the known (in this case) background on the work surface. The mask camera is not disturbed by the variable image from the monitor screen, since the image does not reach the mask camera through the optical filter.
  • Fig. 4 the expert, 200, stands or sits in front of a work surface, 204, on which the expert can place objects, such as a hand or a tool. Under the work surface there is a "film cloth” or “film sheet”, 214, with "back projection” properties. A projector, 211, projects an image on the "film cloth”. An optical filter, 212, which is placed in front of the projector's lens, blocks near IR radiation from the projector, if such happens to be present. At a distance of 50-100 cm above the work surface are placed two cameras, the object camera 207 and the mask camera 208. In front of the mask camera is placed an optical filter, 210, which blocks visual radiation but lets through near IR radiation.
  • object camera and mask camera are arranged so that they divide the optical axis by a beam splitter, a "hot mirror” which is transparent to visual radiation and reflective to near IR radiation.
  • the right-hand dashed ring of the figure shows an alternative solution with object and mask camera, where these are placed alongside each other.
  • the mask camera extracts this object out against the known (in this case) background on the work surface. The mask camera is not disturbed by the variable image from the monitor screen, since the image does not reach the mask camera through the optical filter.
  • Fig. 5 the expert, 200, stands or sits in front of a work surface, 204, on which the expert can place objects, such as a hand or a tool.
  • a work surface 204
  • objects such as a hand or a tool.
  • Integrated in the work surface there is a "film cloth” or “film sheet”, 214, with "front projection” properties.
  • a projector, 211 which is placed at a distance of 50-100 cm, projects an image on the "film cloth”.
  • An optical filter, 212 which is placed in front of the projector's lens, blocks near IR radiation from the projector, if such happens to be present.
  • the object camera, 207 is placed near the projector.
  • object camera and projector are arranged so that they divide the optical axis by a beam splitter, 213.
  • the right-hand dashed ring of the figure shows an alternative solution with object camera and projector, where these are placed alongside each other.
  • the mask camera is placed under the work surface at a distance of 25-100 cm.
  • an optical filter, 210 which blocks visual radiation but lets through near IR radiation.
  • the mask camera extracts this object out against the known (in this case) background on the work surface.
  • the mask camera is not disturbed by the variable image from the monitor screen, since the image does not reach the mask camera through the optical filters.
  • On the surfaces in the image that the mask camera masks out are then placed corresponding surfaces from the image from the object camera, and in this way one gets a new image with a known and well-defined background and the object in question. This image is sent to the user, whose computer can extract out the object in question, and then mix this object into the user's image or view.
  • Fig. 6 the expert, 200, stands or sits in front of a work surface, 204, on which the expert can place objects, such as a hand or a tool.
  • a work surface 204
  • objects such as a hand or a tool.
  • Integrated in the work surface there is a "film cloth” or “film sheet”, 214, with "front projection” properties.
  • a projector, 211 which is placed at a distance of 50-100 cm, projects an image on the "film cloth”.
  • An optical filter, 212 which is placed in front of the projector's lens, blocks near IR radiation from the projector, if such happens to be present.
  • the object camera, 207, and the mask camera, 208 are placed near the projector.
  • object camera and mask camera are arranged so that they divide the optical axis by a beam splitter, a "hot mirror” which is transparent to visual radiation and reflective to near IR radiation.
  • the right dashed ring of the figure shows an alternative solution with object camera and mask camera, where these are placed alongside each other.
  • an optical filter, 210 In front of the mask camera is placed an optical filter, 210, which blocks visual radiation but lets through near IR radiation.
  • the mask camera extracts this object out against the known (in this case) background on the work surface. The mask camera is not disturbed by the variable image from the monitor screen, since the image does not reach the mask camera through the optical filters.
  • the mask camera masks out are then placed corresponding surfaces from the image from the object camera, and in this way one gets a new image with a known and well-defined background and the object in question.
  • This image is sent to the user, whose computer can extract out the object in question, and then mix this object into the user's image or view.
  • the expert camera can first take an image of the background against which the expert is working and thereafter mask out future objects, such as the expert's hand or a marker.
  • Fig. 7 the expert, 200, stands or sits in front of a work surface, 204, on which the expert can place objects, such as a hand or a tool.
  • a monitor screen, 216 is placed between the expert's head and the work surface. Placed near or on this monitor screen is a camera which functions both as object camera and mask camera, 207/208. Monitor screen and camera are held in place by a mechanical device, 218.
  • the work surface can have a color or pattern allowing an object to be masked out from the work surface.
  • the work surface can be blue or green, for example, and the masking can then be performed by the "chroma-key" technique. In this way one gets a new image with a known and well-defined background and the object in question. This image is sent to the user, whose computer can extract out the object in question, and then mix this object into the user's image or view.
  • Fig. 8 the expert, 200, stands or sits in front of a work surface, 204, on which the expert can place objects, such as a hand or a tool.
  • a microdisplay system with magnifying optics, 217 is placed between the expert's head and the work surface. Placed near or on this monitor screen is a camera which functions both as object camera and mask camera, 207/208. Monitor screen and camera are held in place by a mechanical device, 218.
  • the work surface can have a color or pattern allowing an object to be masked out from the work surface.
  • the work surface can be blue or green, for example, and the masking can then be performed by the "chroma-key" technique. In this way one gets a new image with a known and well-defined background and the object in question. This image is sent to the user, whose computer can extract out the object in question, and then mix this object into the user's image or view.
  • Fig. 9 the expert, 200, stands or sits in front of a work surface, 204, on which the expert can place objects, such as a hand or a tool.
  • the work surface can have a color or pattern allowing an object to be masked out from the work surface.
  • the work surface can be blue or green, for example, and the masking can then be performed by the "chroma-key" technique. In this way one gets a new image with a known and well-defined background and the object in question. This image is sent to the user, whose computer can extract out the object in question, and then mix this object into the user's image or view.
  • a method for visual interaction must be used.
  • the method is based on moving images being transmitted (and mixed) between user and expert, and vice versa.
  • Such a method can function in the following way:
  • An image (image sequence) describing the user's field of vision, the "user image”, is transmitted, on the one hand, to the user's computer (calculating unit) and, on the other hand, via a transmission channel to the expert's computer (calculating unit) .
  • the "user image” is shown to the expert by the expert's presentation device.
  • the expert places an object, such as a hand, on his work surface, and this object is extracted out with the aid of the mask camera and the object camera, and this object is thus described as a subset in the "expert object image”. Note that the expert in this case obtains a mixed visual experience of an actual object being placed on a presentation device which shows the "user image”.
  • the "expert object image” is transmitted via a transmission channel to the user's computer where the "expert object image” is superposed on (mixed with) the "user image” (the instantaneous one) .
  • the image that has been mixed is here called the "mixed image”.
  • the “mixed image” is shown to the user by the user's presentation device. Note that the “expert object image” (the sequence) is slightly delayed in relation to the “user image” (the sequence) .
  • • 301 is a one-way transmission channel between a user and an expert
  • • 220 is an expert's computer (calculating unit) • 110 is a user's presentation device
  • • 122 is a user's view/object.
  • the expert's presentation device is largely independent of how the expert moves his head.
  • a fixed point the "expert fixed point” can be obtained, for example at the center of the image. This can be predefined in terms of size (z position), position (x,y) and twist.
  • the expert's presentation device can be movable, for example in the form of a cell phone or other user-borne unit.
  • synchronization between the expert and the user can be achieved by defining the expert fixed point in the same way as is described below for the user fixed point.
  • the user and/or the expert can "freeze” an image from the user, from which a number of "features" are detected.
  • the user's computer (system) and/or the expert's computer (system) can also automatically “freeze” and detect “features” according to a time plan.
  • Spatial synchronization here means that, when the images are mixed together, the system takes into consideration how the views of the expert and of the user are synchronized in the (x,y) direction, in the z direction and in terms of twist. Regarding synchronization in the z direction, this means that an image has to be enlarged or reduced to permit complete spatial synchronization. Spatial synchronization also means that, when the images are mixed together, the system also takes into consideration twisting of the views and turns the images correctly, i.e. synchronizes them.
  • the spatial synchronization can be defined with the aid of the "features" or "markers” that the system detects. These of course change in terms of size and mutual relationship when the user approaches or moves away from them or holds his head at an angle.
  • • 511 is a field in the expert's image that does not have continuous updating from the user.
  • the field can be filled with image information from the "memory" where image information of the field can be stored as it was previously in the image
  • Fig. 12 This figures shows how a synchronization proceeds when a user moves.
  • the expert fixed point, 501 does not change position, size and twist.
  • the user fixed point, 507 which was defined according to Fig. 11, changes position, size and twist. Therefore, the synchronization, 508, is complicated, because the user image and the expert object image have to be adapted in such a way that the two fixed points are placed correctly over each other, 509.
  • the synchronization entails certain sequences as regards the images that the user and the expert have in their fields of vision.
  • the image the user sees 512, certain surfaces from the expert, 513, are not continuously updated.
  • the expert sees 510, certain surfaces from the expert, 511, are not continuously updated.
  • the image quality and sound quality may need to be adjusted depending on the transmission capacity (bit/time unit) and delay (time to transmit one bit) of the transmission channel that is used.
  • the image quality is dependent on which compression method is used and on the degree of compression.
  • the relationship between transmission capacity and image quality should in principle be kept constant and at such a level that stable interactivity is achieved, both as regards image and also sound.
  • the delay on the transmission channel is shorter in time than the time it takes to transmit an image, the image quality can be allowed to deteriorate in order to maintain interactivity.
  • Compression methods for high image quality normally produce a greater delay than other methods, since it takes a longer time for compression and decompression.
  • the time for compression and decompression can be reduced.
  • each row (or column) in the image should contain a number of start and stop values for relevant image data, i.e. contain data from some "object".
  • Figure 13 is a diagram showing an example of a compression method in which:
  • • 600 is a color camera that generates a "Bayern Pattern" signal (compressed color signal).
  • the camera could, for example, generate an RGB image or YUV image, in which case parts of the figure would be able to be omitted
  • 601 is a module that converts the "Bayern Pattern" image into three images, an R image (red) , a G image (green) and a B image (blue)
  • 602 shows three compression modules in which a respective image is compressed, e.g. with JPEG2000
  • 603 is a transmission channel in which the compressed images are transmitted, for example one after another, i.e. sequentially,
  • 605 is a module that combines the decompressed images into one color image
  • 606 is a presentation device for the user/expert.
  • a method is made available for synchronizing images in remote visual interaction between an expert and at least one user in an interaction system comprising at least one work station comprising a presentation device and a camera for use by the user, said work station being operatively connected to a support station comprising a presentation device and a camera for use by the expert, the user camera being configured to capture the user's field of vision, which is presented to the expert via the expert presentation device as a user image, and the expert camera being configured to depict an object at the support station as an expert object image and mix the expert object image together with the user image and present these to the user via the user presentation device as a mixed image.
  • the method comprises the steps of:
  • Figs. 15a and 15b show a variant of an expert station.
  • the expert station can comprise a first monitor screen 801, on which a user image is presented to the expert and to which the expert points with a hand 803 or pointer device.
  • the expert station can also be provided with a lighting source 804, which generates visible and/or near IR light, and a camera unit 805, which can comprise a visible IR and a near IR camera and can be designed in accordance with what has been described with reference to Figs. 3-6.
  • the camera unit produces an image which can be mixed with the image shown on the first monitor screen, and the mixed image is shown on a second monitor screen 802, which is placed in a manner that gives an ergonomically advantageous viewing angle.
  • the monitor screens 801, 802 can be placed on a common stand 806.
  • the camera unit's near IR camera generates a mask, which is used to determine the position and shape of the hand 803. With the aid of the mask, those parts of an image from the visible IR camera of the camera unit corresponding to the hand can be extracted to form an image that shows only the expert's hand. This image can be mixed and combined with the image that is shown on the first monitor screen.
  • a predefined "skin pattern" or a predefined image of a hand with the desired appearance can be placed on the masked area.
  • the shape of the predefined hand can be adapted (twisted) to match the masked area.
  • Fig. 15b shows a marker 807 (a detail with predetermined shape or pattern) which the computer recognizes via the camera module.
  • the computer "couples" the image or 3D object to the position and twist of the marker.
  • the image and/or 3D object can be shown to the expert on the second monitor screen 802 and to the user in the latter 's presentation system.
  • Fig. 16 shows an expert station in the form of a laptop computer, which has a keyboard component 901 and a monitor screen part 902, and a stand 905 supporting a camera unit 906 that can be designed in the manner described with reference to Figs. 3-6.
  • the expert can point to the monitor screen, his hand 903 being extracted and mixed with the image shown on the monitor screen and being sent to a user located at a distance.
  • the camera unit 906 can comprise a light source for visible and/or near IR light.
  • Fig. 17 shows a variant of a user unit in the form of a cell phone or similar device, which can have a processing unit, user interface (at least loudspeaker and set of buttons/touch screen) and communication unit, and which has a first housing 1001, which accommodates at least one of the processing unit, the user interface and the communication unit.
  • the user unit also comprises a second housing 1002, which is movable relative to the first housing 1001 and which accommodates a microdisplay 1005 with magnifying optics and a camera unit 1007, which is arranged in the same direction 1004 as the microdisplay and the user's eye 1006.
  • the camera unit and the microdisplay can be designed in the manner described with reference to Figs. 1-2.
  • the first and second housings can pivot or slide relative to each other, such that the second housing can be moved between a (collapsed/retracted) rest position and a working position, in which the microdisplay is visible to the user.
  • the user unit can be designed such that a loudspeaker is placed to be audible from one ear of the user, while at the same time the microdisplay is visible to one eye of the user.
  • the pivoting/sliding between the housings can be such as to permit a certain degree of individual adaptation of the user unit, such that loudspeaker and display are positioned in the most advantageous way possible for a respective user.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

La présente invention concerne des systèmes et des procédés destinés à une interaction visuelle à distance entre au moins un utilisateur et un expert. L'invention concerne un procédé d'interaction visuelle à distance entre un expert (200) et au moins un utilisateur (100). Le procédé est utilisé dans un système d'interaction comprenant au moins une station de travail comprenant un dispositif de présentation (101) et une caméra utilisateur (102) utilisée par un utilisateur (100); la station de travail étant opérationnellement connectée à une station support comprenant un dispositif de présentation expert (201) et un bloc caméra expert (202) conçu pour être utilisé par l'expert (200). Le procédé consiste à utiliser la caméra utilisateur (102) pour capture le champ de vision de l'utilisateur (100) afin de former une image utilisateur (505), à présenter l'image utilisateur (505) à l'expert (200) au moyen du dispositif de présentation expert (201), à utiliser le bloc caméra expert (202) afin d'illustrer un objet (502) au niveau de la station support en tant qu'image d'objet expert (500), à mélanger l'image d'objet expert (500) avec l'image utilisateur (505) afin de former une image mixte, puis à présenter l'image mixte à l'utilisateur par l'intermédiaire du dispositif de présentation utilisateur (101). Le bloc caméra expert (202) est utilisé pour présenter l'objet (502) lorsque celui-ci est au moins en partie essentiellement situé entre le bloc caméra expert et le dispositif de présentation expert.
PCT/SE2010/050916 2009-08-25 2010-08-25 Procédés et systèmes pour interaction visuelle Ceased WO2011025450A1 (fr)

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US23664309P 2009-08-25 2009-08-25
US61/236,643 2009-08-25
SE0950609A SE537143C2 (sv) 2009-08-25 2009-08-25 System och metod för visuell interaktion
SE0950609-8 2009-08-25
SE1050496 2010-05-19
SE1050496-7 2010-05-19

Publications (1)

Publication Number Publication Date
WO2011025450A1 true WO2011025450A1 (fr) 2011-03-03

Family

ID=43628259

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/SE2010/050916 Ceased WO2011025450A1 (fr) 2009-08-25 2010-08-25 Procédés et systèmes pour interaction visuelle

Country Status (1)

Country Link
WO (1) WO2011025450A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2920683A1 (fr) * 2012-11-15 2015-09-23 Iversen, Steen Svendstorp Procédé de fourniture d'une instruction visuelle représentée numériquement d'un spécialiste à un utilisateur ayant besoin de ladite instruction visuelle, et système associé
DE102016003074A1 (de) * 2016-03-12 2017-09-14 Audi Ag Verfahren zum Betreiben eines Virtual-Reality-Systems und Virtual-Reality-System

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1435737A1 (fr) * 2002-12-30 2004-07-07 Abb Research Ltd. Système et méthode de réalité augmentée
WO2005124429A1 (fr) * 2004-06-18 2005-12-29 Totalförsvarets Forskningsinstitut Procede interactif de presentation d'une information dans une image
US20060176242A1 (en) * 2005-02-08 2006-08-10 Blue Belt Technologies, Inc. Augmented reality device and method
EP1710002A1 (fr) * 2005-04-08 2006-10-11 Canon Kabushiki Kaisha Procédé et appareil de traitement d'informations
EP1739642A1 (fr) * 2004-03-26 2007-01-03 Atsushi Takahashi Systeme de verre grossissant numerique d'entite 3d ayant une fonction d'instruction visuelle 3d
WO2009036782A1 (fr) * 2007-09-18 2009-03-26 Vrmedia S.R.L. Dispositif et procédé de traitement d'informations pour une assistance technique à distance

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1435737A1 (fr) * 2002-12-30 2004-07-07 Abb Research Ltd. Système et méthode de réalité augmentée
EP1739642A1 (fr) * 2004-03-26 2007-01-03 Atsushi Takahashi Systeme de verre grossissant numerique d'entite 3d ayant une fonction d'instruction visuelle 3d
WO2005124429A1 (fr) * 2004-06-18 2005-12-29 Totalförsvarets Forskningsinstitut Procede interactif de presentation d'une information dans une image
US20060176242A1 (en) * 2005-02-08 2006-08-10 Blue Belt Technologies, Inc. Augmented reality device and method
EP1710002A1 (fr) * 2005-04-08 2006-10-11 Canon Kabushiki Kaisha Procédé et appareil de traitement d'informations
WO2009036782A1 (fr) * 2007-09-18 2009-03-26 Vrmedia S.R.L. Dispositif et procédé de traitement d'informations pour une assistance technique à distance

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2920683A1 (fr) * 2012-11-15 2015-09-23 Iversen, Steen Svendstorp Procédé de fourniture d'une instruction visuelle représentée numériquement d'un spécialiste à un utilisateur ayant besoin de ladite instruction visuelle, et système associé
DE102016003074A1 (de) * 2016-03-12 2017-09-14 Audi Ag Verfahren zum Betreiben eines Virtual-Reality-Systems und Virtual-Reality-System
US10497181B1 (en) 2016-03-12 2019-12-03 Audi Ag Method for operating a virtual reality system, and virtual reality system
DE102016003074B4 (de) 2016-03-12 2025-01-02 Audi Ag Verfahren zum Betreiben eines Virtual-Reality-Systems und Virtual-Reality-System

Similar Documents

Publication Publication Date Title
US12461368B2 (en) Sedentary virtual reality method and systems
US10642564B2 (en) Display system, display device, information display method, and program
US6046712A (en) Head mounted communication system for providing interactive visual communications with a remote system
US20160269685A1 (en) Video interaction between physical locations
CN110275602A (zh) 人工现实系统和头戴式显示器
US6650305B1 (en) Wireless electronic display
CN108446011A (zh) 一种基于增强现实的医疗辅助方法及设备
CN108427194A (zh) 一种基于增强现实的显示方法及设备
WO2011025450A1 (fr) Procédés et systèmes pour interaction visuelle
JP6733401B2 (ja) 表示システム、表示装置、情報表示方法、及び、プログラム
JP6540426B2 (ja) 表示システム、表示装置、情報表示方法、及び、プログラム
JP2007121600A (ja) 画像情報表示装置
CN108427195A (zh) 一种基于增强现实的信息处理方法及设备
McKay et al. Membrane-mirror-based autostereoscopic display for tele-operation and teleprescence applications
CN213903982U (zh) 一种新型智能眼镜及远程可视化系统
CN208588947U (zh) 一种具有手势识别功能的显示系统
CN108696740A (zh) 一种基于增强现实的直播方法及设备
SE0950609A1 (sv) System och metod för visuell interaktion
CN207184672U (zh) 一种基于vr显示的视频传输系统
CN106303467B (zh) 一种智能穿戴设备及数据传输方法
JP2004282462A (ja) ３次元映像情報配信システム
WO2024233110A1 (fr) Dispositifs, procédés et interfaces utilisateur graphiques permettant le basculement entre plusieurs modes de fonctionnement
KR20110134574A (ko) 영상 입출력 기기를 통한 증강현실 시스템 및 그 제공 방법
JP2019075684A (ja) 通信システムおよび通信方法
JP2000354234A (ja) テレビ電話機

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 10812409

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 10812409

Country of ref document: EP

Kind code of ref document: A1