[go: up one dir, main page]

EP2188701B1 - Détection tactile multipoint par le biais d'une réflexion interne totale frustrée - Google Patents

Détection tactile multipoint par le biais d'une réflexion interne totale frustrée Download PDF

Info

Publication number
EP2188701B1
EP2188701B1 EP08782614.5A EP08782614A EP2188701B1 EP 2188701 B1 EP2188701 B1 EP 2188701B1 EP 08782614 A EP08782614 A EP 08782614A EP 2188701 B1 EP2188701 B1 EP 2188701B1
Authority
EP
European Patent Office
Prior art keywords
optical waveguide
compliant layer
compliant
light
layer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Not-in-force
Application number
EP08782614.5A
Other languages
German (de)
English (en)
Other versions
EP2188701A4 (fr
EP2188701A2 (fr
Inventor
Jefferson Y. Han
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Technology Licensing LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US11/833,908 external-priority patent/US8441467B2/en
Application filed by Microsoft Technology Licensing LLC filed Critical Microsoft Technology Licensing LLC
Publication of EP2188701A2 publication Critical patent/EP2188701A2/fr
Publication of EP2188701A4 publication Critical patent/EP2188701A4/fr
Application granted granted Critical
Publication of EP2188701B1 publication Critical patent/EP2188701B1/fr
Not-in-force legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
    • G06F3/0425—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/041—Indexing scheme relating to G06F3/041 - G06F3/045
    • G06F2203/04104—Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/041—Indexing scheme relating to G06F3/041 - G06F3/045
    • G06F2203/04108—Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/041—Indexing scheme relating to G06F3/041 - G06F3/045
    • G06F2203/04109—FTIR in optical digitiser, i.e. touch detection by frustrating the total internal reflection within an optical waveguide due to changes of optical properties or deformation at the touch location

Definitions

  • the present invention relates to techniques for enabling high-resolution multi-touch sensing based on frustrated total internal reflection.
  • Touch sensing is commonplace for single points of contact.
  • One group of touch sensing techniques is to employ frustrated total internal reflection (FTIR).
  • FTIR frustrated total internal reflection
  • TIR total internal reflection
  • Fiber optics, light pipes, and other optical waveguides rely on this phenomenon to transport light efficiently with very little loss.
  • another material at the interface can frustrate this total internal reflection, causing light to escape the waveguide there instead.
  • Frustrated total internal reflection is well known and has been used in the biometrics community to image fingerprint ridges since at least the 1960s.
  • U.S. Patent 3,200,701 to White issued in 1965 and describes using FTIR to optically detect the ridge pattern of a skin surface.
  • U.S. Patent 4,346,376 to Mallos discloses a CRT-based touch sensor, which replaced the bulky prism with a thin platen waveguide and operates by detecting the light scattered away by an object in optical contact. More recent fingerprint sensors use this approach, as disclosed in Fujieda, I., Haga, H., "Fingerprint Input based on Scattered-Light Detection," Applied Optics-IP, 36, 35, 9152-9156 (1997 ).
  • Multi-touch sensing may be achieved by employing a passive matrix of sensing elements, such as force-sensitive-resistors (FSRs), as discussed in Hillis, W. D., "A High Resolution Imaging Touch Sensor," International Journal of Robotics Research, pages 1, 2, 33-44 (1982 ).
  • FSRs force-sensitive-resistors
  • U.S. Patent 4,134,063 to Nicol et al. discloses the use of capacitive electrodes for this purpose.
  • SmartSkin An Infrastructure for Freehand Manipulation on Interactive Surfaces
  • the Fingerworks iGesturePad is another example of a device that employs a passive matrix of capacitance sensors.
  • U.S. Patent 6,323,846 to Westerman et al. discloses additional examples of using such an array in a multi-touch surface system.
  • Such systems while less complex than systems that employ multiple active sensors, still entail numerous electrical connections and thus disadvantageously limit their application to uses that require relatively low resolution (e.g., under 100 x 100). Furthermore, such systems are visually opaque and thus require the use of top-projection if to be integrated with a graphic display.
  • DE 100 47 920 A1 discloses a device that has a light source with a light conducting body with an upper side approximately parallel to the control field that totally reflects light; touching or pressing the field alters the reflection of light.
  • a detector has a detection surface on which an image of the upper side of the body corresponding to the control field is formed; light intensities detected at points in the detection surface are evaluated to detect the contact position.
  • US 4 484 179 A discloses a touch sensitive device that is arranged with substantially parallel surfaces which are at least partially transparent to light.
  • a flexible membrane is supported above the top surface of the screen and when the overlay is flexed into contact with the screen, light which had passed through the screen, is trapped between the screen surfaces by total internal reflection. The edges of the screen are fitted with light detectors for gathering the trapped light.
  • US 2007/152985 A1 discloses devices, systems, and methods for directing a beam of light into a display such that the beam of light undergoes internal reflection within the display and capturing a reflected light beam are disclosed.
  • US 6 700 129 B1 discloses an invention that transmits, to an electronic board controller, click signals generated upon manipulating multiple switches provided on a coordinates-inputting pen, timing signals upon detecting infrared scanning beams, and the like without using a connecting wire.
  • light-emitting elements are provided which can emit visible or infrared light to the coordinates-inputting pen.
  • a timing signal upon detecting the infrared scanning beam with the pen and a click signal generated upon manipulating the switch on the pen are sent out on a modulated light to be received by a light receiver arranged at a fixed position and transmitted to the electronic board controller.
  • US 4 668 861 A discloses a tactile sensing device for use in robotics and medical prosthetics which includes a transparent sheet-like element and a second resilient sheet-like element positioned adjacent the first transparent element.
  • a light detection and imaging means is positioned to observe the interface between the two elements.
  • a light source is provided to illuminate the interior of one of the two elements. Any object pressing against the resilient element deforms the same into contact with the transparent element. Areas of contact caused by the pressing object produce a lighted area that can be detected by the light detecting means.
  • a multi-touch sensing system comprises an optical waveguide having an index of refraction sufficient to cause light of at least one wavelength (e.g., narrow band wavelength) received in a predetermined direction to undergo total internal reflection within the optical waveguide.
  • the system includes a compliant layer having a first surface disposed adjacent to and spaced from a first side of the optical waveguide, and the compliant layer is sufficiently deformable upon depression to allow portions of the first surface of the compliant layer to contact the optical waveguide.
  • the first surface of the compliant layer has an index of refraction sufficiently similar to the index of refraction of the optical waveguide to cause, upon the first surface of the compliant layer contacting the optical waveguide, some of the light undergoing total internal reflection to scatter and to escape the optical waveguide.
  • An image sensor detects some of the light that escapes from the optical waveguide.
  • a multi-touch sensing system comprises an optical waveguide, and a compliant layer directly coupled to the first side of the optical waveguide, where the compliant layer and the optical waveguide having substantially similar indexes of refraction so that light of at least one wavelength received by the optical waveguide in a predetermined direction undergoes total internal reflection within the optical waveguide and the compliant layer.
  • the system also includes a cladding layer directly coupled to the compliant layer.
  • the cladding layer has an index of refraction sufficiently lower than the index of refraction of the compliant layer so that deformation of the cladding layer causes some of the light undergoing total internal reflection to scatter and to escape the compliant layer and the optical waveguide.
  • An image sensor detects some of the light that escapes the compliant layer and the optical waveguide.
  • a multi-touch sensing system comprises a compliant optical waveguide having an index of refraction sufficient to cause light of at least one wavelength received in a predetermined direction to undergo total internal reflection within the compliant optical waveguide.
  • the system also includes a cladding layer directly coupled to the first side of the compliant optical waveguide.
  • the cladding layer has an index of refraction sufficiently lower than the index of refraction of the compliant optical waveguide so that deformation of the cladding layer causes some of the light undergoing total internal reflection to scatter and to escape the compliant optical waveguide.
  • An image sensor detects some of the light that escapes the compliant optical waveguide.
  • the present invention includes a number of aspects, features and variations. Some of these aspect and features are summarized below.
  • the compliant layer has an associated light filter for filtering light of said at least one wavelength.
  • the compliant layer is deformable by depressing a second surface of the compliant layer by a finger of the user of the multi-touch sensing system.
  • the image sensor is optically disposed to receive the light that escapes from the second side of the optical waveguide.
  • a band pass filter optically disposed between the second side of the optical waveguide and the image sensor allows substantially only the light at said wavelength to pass from the optical waveguide to the image sensor.
  • the image sensor is a flat image sensor such as a TFT (thin-film transistor) image sensor.
  • the flat image sensor is a wedge-optic camera.
  • the system includes a light source disposed directly against an edge of the optical waveguide for emitting the light that is received by the optical waveguide.
  • the optical waveguide may be a compound component.
  • the optical waveguide is comprised of a non-compliant optical waveguide treated with a compliant layer.
  • the system includes a rear projector for projecting a video image towards the second side of the optical waveguide.
  • the compliant layer and the optical waveguide are substantially transparent to visible light so that the displayed video image is viewable through these layers.
  • a diffuser disposed on the compliant layer diffuses the projected video image.
  • the compliant layer itself is designed to diffuse the projected video image.
  • the system includes a film opaque to the light at said wavelength that is disposed on the compliant layer.
  • a front projector projects a video image onto the film.
  • the system includes a liquid crystal display (LCD) panel disposed on the second side of the optical waveguide for displaying a video image in a direction towards the compliant layer, and the compliant layer and the optical waveguide are substantially transparent to visible light so that the displayed video image is viewable through these layers.
  • LCD liquid crystal display
  • multiple emissive sources provide backlight to the LCD panel and a backlight diffuser is disposed between the LCD panel and the emissive sources.
  • a thin-profile wedge backlight unit provides backlight to the LCD panel.
  • the image sensor is a wedge-optic camera.
  • multiple light sources are provided as backlight to the LCD panel, and a backlight diffuser is disposed between the LCD panel and the light sources, and the image sensor is a wedge-optic camera.
  • the LCD panel is disposed within an optical path of the light escaping from the optical waveguide to the image sensor, and the LCD panel is transparent to light at said wavelength so as to not prevent the light escaping the optical waveguide from reaching the image sensor.
  • the system includes a liquid crystal display (LCD)/optical sensing panel disposed on the second side of the optical waveguide for displaying a video image in a direction towards the compliant layer, and the image sensor is the optical sensing elements of the LCD/optical sensing panel.
  • LCD liquid crystal display
  • the image sensor is the optical sensing elements of the LCD/optical sensing panel.
  • the system includes an active matrix OLED with integrated sensors disposed on the second side of the optical waveguide for displaying a video image in a direction towards the compliant layer, the image sensor is the integrated sensors of the active matrix OLED with integrated sensors.
  • the system may include a flexible OLED (FOLED) coupled to the compliant layer for displaying a video image in a direction towards a user of the multi-touch sensing system.
  • FOLED flexible OLED
  • the FOLED is sufficiently flexible to deform the compliant layer, upon depression of the FOLED, so that the compliant layer contacts the optical waveguide.
  • the system includes a double-sided FOLED disposed on the compliant layer and that has an emissive layer for displaying a video image in a direction towards a user of the multi-touch sensing system and a sensing layer for sensing some of the light that escapes from the optical waveguide in a direction towards the user.
  • the double-sided FOLED is sufficiently flexible to deform the compliant layer, upon depression of the double-sided FOLED, so that the compliant layer contacts the optical waveguide.
  • the image sensor corresponds to the sensing layer of the FOLED.
  • the system includes a combination flexible LCD and a flexible backlight coupled to the compliant layer for displaying a video image in a direction towards a user of the multi-touch sensing system.
  • the combination of the flexible LCD and the flexible backlight are sufficiently flexible to deform the compliant layer, upon depression of the combination, so that the compliant layer contacts the optical waveguide.
  • the system includes a flexible LCD coupled to the compliant layer for displaying a video image in a direction towards a user of the multi-touch sensing system.
  • a light source directs visible light through the compliant layer toward the flexible LCD, and the compliant layer diffuses the visible light.
  • the flexible LCD includes integrated sensors for sensing at least some of the light that escapes from the optical waveguide in a direction towards the user, and the image sensor corresponds to the sensors integrated in the flexible LCD.
  • light at said wavelength is infrared light and the image sensor detects infrared light.
  • the system further includes a second image sensor for detecting visual information through the optical waveguide and the compliant layer.
  • the second image sensor is equipped to not detect light escaping from the optical waveguide that is intended for the first sensor.
  • the second image sensor is capable of sensing an area beyond the compliant layer.
  • the system incorporates an auto-calibration system.
  • the auto-calibration system registers the infrared touch image coordinate system precisely to the graphical (displayed) coordinate system.
  • a method of multi-touch sensing comprises totally internally reflecting at least some light of at least one wavelength within an optical waveguide, sufficiently depressing a compliant layer having a first surface disposed adjacent to and spaced from a first side of the optical waveguide to cause portions of the first surface of the compliant layer to contact the first side of the optical waveguide, the first surface of the compliant layer having an index of refraction sufficiently similar to an index of refraction of the optical waveguide to cause, upon the first surface of the compliant layer contacting the optical waveguide, some of the light undergoing total internal reflection to scatter and to escape the optical waveguide, and sensing at least some of the light escaping the optical waveguide.
  • a method of multi-touch sensing comprises totally internally reflecting at least some light of at least one wavelength within an optical waveguide and a compliant layer that is directly coupled to a first side of the optical waveguide, deforming a cladding layer directly coupled to the compliant layer and having an index of refraction sufficiently lower than an index of refraction of the compliant layer to cause some of the light undergoing total internal reflection to scatter and to escape the compliant layer and the optical waveguide, and sensing at least some of the light escaping the optical waveguide and the compliant layer.
  • a method of multi-touch sensing comprises totally internally reflecting at least some light of at least one wavelength within a compliant optical waveguide, deforming a cladding layer directly coupled to the compliant optical waveguide and having an index of refraction sufficiently lower than an index of refraction of the compliant optical waveguide to cause some of the light undergoing total internal reflection to scatter and to escape the compliant optical waveguide, and sensing at least some of the light escaping the compliant optical waveguide.
  • Multi-touch sensing enables a user to interact with a system with more than one finger at a time, as in chording and bi-manual operations. Multi-touch sensing may accommodate multiple users simultaneously, which is especially useful for larger shared-display systems such as interactive walls and tabletops.
  • Figures 1A through 1D of the drawings show several simple examples of multi-touch sensing of the present invention.
  • Multi-touch sensing in accordance with the present invention is based on frustrated total internal reflection (FTIR).
  • FTIR sensing techniques can acquire true touch image information at high spatial and temporal resolutions, is scalable to large installations, and is well suited for use with many display technologies.
  • FTIR sensing techniques can acquire true touch image information at high spatial and temporal resolutions, is scalable to large installations, and is well suited for use with many display technologies.
  • TIR total internal reflection
  • the concept of FTIR is well known and understood in the art, further technical description of FTIR is omitted herein except where necessary for an understanding of the present invention.
  • the terms “one embodiment,” “another embodiment,” “second embodiment,” “third embodiment,” “one variation,” “another variation,” and the like are used for convenience only to distinguish one structure, process, system, etc., from another structure, process, system, etc., and are not used to designate relative importance or amount of difference or other sort of quantification or comparative trait, unless otherwise expressly stated herein.
  • the terms “first”, “second”, “primary,” “secondary” and the like are used to distinguish one component, element, object, step, process, activity or thing from another, and are not used to designate relative position or arrangement in time, unless otherwise expressly stated herein.
  • Non-Diffusive, Micro-Structured Index-Matched Compliant Layer for an FTIR Touch Sensor (Compliant Layer Design Type 1)
  • FIG. 2 of the drawings is a simplified schematic illustration of a multi-touch sensing system 10 in accordance with a first embodiment of the present invention.
  • multi-touch sensing system 10 (or, for convenience, “sensor 10” or “system 10") includes an optical waveguide 12, a light source 14, a compliant layer 16, and an image sensor 20 (also sometimes referred to as an "imaging sensor” or “imaging camera”), along with other components to be described.
  • Light source 14 preferably is an infrared (IR) emitter, such as an IR emitter comprised of multiple high-power infrared LEDs.
  • IR infrared
  • Light source 14 is placed directly against a polished edge of optical waveguide 12 so as to maximize coupling into total internal reflection.
  • light source 14 may provide an optical output of 460mW at 880nm, although other optical outputs can be employed.
  • Optical waveguide 12 may be a sheet of acrylic whose edges have been polished clear, but other suitable materials may be utilized. In one example of a manufactured prototype, a one-quarter inch (6.4mm) thick acrylic sheet having the dimensions of 16 inches by 12 inches (406mm x 305mm) is utilized as the optical waveguide. Common glass generally is not preferred due to its poor optical transmittance. However, clearer glass formulations (e.g. "water white", BK-7 borosilicate glass, etc.) may be employed. Though more expensive, such glass is structurally stiffer and is far less easily scratched than acrylic.
  • clearer glass formulations e.g. "water white", BK-7 borosilicate glass, etc.
  • multi-touch sensing system 10 includes compliant layer 16 disposed on one side of waveguide 12.
  • a small airgap 24 is maintained precisely between the two components due to the microstructure of the compliant layer.
  • Compliant layer 16 may be custom molded or etched onto a layer of compliant plastic or other material or made in any other suitable manner.
  • Compliant layer 16 or the material at its surface that is adjacent to waveguide 12 has an index of refraction sufficiently similar (“substantially similar") to the index of refraction of waveguide 12 to maximize coupling when pressure is applied (e.g., by finger 26 shown in Figure 2 ) to compliant layer 16. Minute areas 16a of compliant layer 16 are in contact with waveguide 12 to maintain airgap 24.
  • areas 16a are small enough and sparse enough so as to not swamp the desired signal (i.e., minimize the amount of scattered light during a static, non-depressed state of compliant layer 16).
  • the compliant layer is chosen or engineered with a pattern such that the amount of FTIR response is dependant on the amount of force or pressure acting on it.
  • a wavelength-selective filter film layer 18 (also called "IR filter” herein) preferably is disposed on compliant layer 16, as shown in Figure 2 , to remove or, at least, minimize effects of background illumination.
  • IR filter 18 filters at least frequencies of light emitted by light source 14.
  • the stack shown in Figure 2 comprising compliant layer 16 and waveguide 12 along with IR filter 18, is substantially transparent to visible light and, thus, is ideal for utilization with a direct view display (e.g. LCD), but also may be used with rear projection by treating the outer surface of the compliant layer to be diffusive or by adding an appropriate diffuser, such as discussed in certain embodiments described below.
  • a direct view display e.g. LCD
  • an appropriate diffuser such as discussed in certain embodiments described below.
  • an infrared image sensor 20 having a suitable lens, such as a wide-angle or shift lens, is mounted orthogonally relative to the waveguide and detects the light scattered through the waveguide.
  • the imaging sensor is equipped with a band-pass filter 22 that is matched to the output of light source 14, to minimize optical noise.
  • Image sensor 20 may be of different types, e.g., CCD, CMOS.
  • the scattered light also may be sensed by a thin or flat image sensor 30, such as schematically shown in Figure 2A .
  • a wedge-optic camera such as the camera described in the publication Boucal, et al. 2006, "Wedge Displays as Cameras,” SID International Symposium Digest of Technical Papers, 37 (2). pp. 1999-2002 ("Boucal '06”) may also be employed.
  • Wedge-optic camera 32 is schematically illustrated in Figure 2B .
  • a TFT (thin film transistor) photodiode/phototransistor array may be utilized.
  • suitable additional optics may be employed to redirect the scattered light to enable one or more of the herein-described sensing devices to be disposed at another location relative to the waveguide (e.g., adjacent the IR emitter).
  • Figure 2C schematically illustrates an optical waveguide that is a compound component in accordance with the present invention and, as shown, is comprised of non-compliant optical waveguide layer 36 treated with a compliant optical waveguide layer 34.
  • the two layers 34 and 36 collectively operate as the optical waveguide.
  • Layer 34 may, in another variation, be non-compliant but have other beneficial characteristics.
  • employing a bonded compliant layer over the optical waveguide may assist with FTIR.
  • Other components of each of the various embodiments, such as compliant layer 16 ( Figure 2 ), may also be compound components.
  • image sensor 20 may be supplied to a suitable computer (not shown) or other electronic device capable of handling various well-known image processing operations, such as rectification, background subtraction, noise removal, and analysis for each video frame.
  • a suitable computer not shown
  • image processing operations such as rectification, background subtraction, noise removal, and analysis for each video frame.
  • Well known vision tracking techniques then may be employed to translate the video sequences into discrete touch events and strokes.
  • An image camera that captures the light at 8-bit monochrome at 60 frames per second at a resolution of 640x480 (corresponding to greater than 1mm 2 precision) is suitable for many multi-touch sense applications.
  • an imaging camera having greater resolution, a different frame capture rate and/or other characteristics may be employed. Processing may be carried out by any suitable computing system.
  • Multi-touch sensing in accordance with the present invention provides full imaging touch information without occlusion or ambiguity issues.
  • the touch sense is zero-force and true, that is, it accurately discriminates touch from a very slight hover.
  • the multi-touch sensing system of the invention is capable of sampling at both high temporal and spatial resolutions.
  • the multi-touch sensing system is scalable to relatively large surfaces, such as a wall-sized touch display, although various factors including camera resolution and amount of illumination should be taken into account for the multi-touch sensing system to cover relatively large areas.
  • Figure 3 shows a simplified schematic illustration of a multi-touch sensing system 40 in another example.
  • a clear compliant material e.g. optical adhesive
  • compliant layer 42 which is index-matched and directly applied to waveguide 44.
  • a cladding layer 46 of a lower index of refraction is disposed on compliant layer 42 at the interaction surface. The deformation of the cladding layer causes light to reflect and scatter out from the stack.
  • different types of sensors such as a flat image sensor or a wedge camera, may be employed.
  • FIG. 4 shows a simplified schematic illustration of a multi-touch sensing system 40 in a further example.
  • a compliant waveguide 52 operates as both the waveguide and the compliant layer.
  • Compliant waveguide 52 is supported by a substrate 54 and is made of a clear material with refractive index higher than the substrate.
  • a cladding layer 56 disposed at the interaction surface of compliant waveguide 52 has a lower index of refraction (e.g., PET) than the waveguide.
  • cladding layer 56 is deformed, such as shown in Figure 4 , light is reflected and scattered out from the stack.
  • substrate 54 does not need to operate as a waveguide for the light emitted by the light source (IR emitter 58) and, thus, substrate 54 may be made of a wide variety of materials and have a wide variety of optical characteristics.
  • substrate 54 may be part of an already existing display assembly, such as an LCD display or a storefront window in a retail environment.
  • compliant waveguide 52 and cladding layer 56 may be applied in-situ to the already existing substrate 54.
  • Figures 2 , 3 and 4 schematically illustrate different embodiments/variations of multi-touch sensing systems.
  • the embodiment shown in Figure 2 is referred to hereinafter as Compliant Layer Design Type 1
  • the variation shown in Figure 3 is referred to hereinafter as Compliant Layer Design Type 2
  • the variation shown in Figure 4 is referred to hereinafter as Compliant Layer Design Type 3.
  • different types of sensing systems/processes may be employed.
  • Figures 2, 2A and 2B schematically illustrate three different types of image sensors: (1) image sensor 20 employing an appropriate lens; (2) a flat image sensor 30; and (3) a wedge-optic camera 32.
  • Other types of sensing systems/processes may be employed, including a TFT photodiode array or other types of suitable flat image sensor not identified herein.
  • each of the variations employ various components including at least an optical waveguide.
  • one or more of such components may be a compound component, and in particular variations, the compliant optical waveguide may be a (non-compliant) optical waveguide treated with a compliant layer collectively functioning as the optical waveguide.
  • an optical waveguide having such a compliant layer assists in FTIR.
  • each of such described embodiments may be used in connection with any one of the herein-described compliant layer design types (i.e., Types 1, 2 or 3). Still further, and unless otherwise indicated, each of such described embodiments may use any type of sensing system/method (e.g., image sensor 20, flat image sensor 30, wedge-optic camera 32, etc.). Still further, and unless otherwise indicated, each of such described embodiments may use any other type of display method, if suitable, or none at all. For convenience only, Figures 5-6 and 8-13 , described below, schematically show various embodiments employing compliant layer design type 1.
  • FIG. 5 shows a simplified schematic illustration of a multi-touch rear projection system 70 in accordance with the present invention.
  • a projector 72 is disposed behind the viewing/contact side of the stack (i.e., the compliant layer, waveguide, etc.) and a diffuser film 74 is disposed on the IR filter 76 (or disposed on compliant layer 78 or on cladding 46 or 56 - Figures 3 and 4 ).
  • diffuser film 74 is omitted and the compliant layer 78 (or compliant waveguide) is engineered to produce desired diffusing characteristics.
  • Employing a rear projection within a multi-touch sensing system is further described in the '691 publication. Due to the use of a rear projector, it is preferable to not employ a TFT image sensor unless such sensor is designed to not interfere with the video projection.
  • FIG. 6 is a schematic illustration of a multi-touch front projection system.
  • System 80 includes, along with the previously described components of the multi-touch sensor (i.e., waveguide, compliant layer), a front projector 82, an infrared-opaque film (barrier 84) that is disposed on the compliant layer 86, and any suitable front-projection layer (projection screen 88) disposed over barrier 84. Since the stack does not need to be visibly transparent due to front projection, an infrared-opaque film is preferred and substantially more effective than an IR filter (e.g., IR filter 76 shown in Figure 5 ). As previously mentioned, different types of image sensors may be employed.
  • FIGS 7A and 7B are schematic illustrations of multi-touch systems that employ compliant layer design types 1 and 3, respectively, and include active matrix LCDs (AMLCD) in accordance with the present invention.
  • multi-touch system 100 employs an active LCD panel 102 disposed behind a waveguide 104.
  • compliant layer design type 1 is employed, which includes a compliant layer 104, along with an airgap 108 disposed between waveguide 104 and compliant layer 106, and an IR filter 110.
  • a diffuser 112 is disposed adjacent to, preferably contacting, the backside of LCD panel 102, and multiple emissive sources (backlights 114), such as LEDs, are distributed evenly behind diffuser 112. Different types of image sensors may be employed.
  • multi-touch system 120 employs an active LCD panel 122 disposed behind a compliant layer design type 3, in which a cladding layer 126 is disposed on a compliant waveguide 124.
  • a diffuser 132 is disposed adjacent to the backside of LCD panel 122 and multiple emissive sources (backlights 134) are distributed evenly behind diffuser 132.
  • the examples shown in Figures 7A and 7B may be applied to typical, existing, off-the-shelf type, generally non-flexible, LCDs.
  • Such LCDs typically are transparent to infrared wavelengths and, thus, the imaging sensor of the present invention is suitably disposed behind the active LCD layer without modification to the LCD layer.
  • the LCD may be suitably disposed within the optical path (of the infrared light) between the optical waveguide and the image sensor.
  • the existing backlight unit (BLU) of the LCD is modified to include the above-mentioned diffuser (e.g., diffuser 112, 134) against the backside of the LCD panel and suitable emissive sources are disposed behind the diffuser.
  • Figures 7C, 7D and 7E show additional examples employing an LCD display.
  • Figure 7C is a schematic illustration of a system 130 that includes an FTIR sensing stack 131 (using any of the described embodiments) that is deposed on an LCD 132a along with a thin-profile wedge backlight unit 133. Image sensing is achieved with a direct view video camera 134a.
  • FIG 7D is a schematic illustration of a system 135 that is a similar to system 130 shown in Figure 7C , but employs a wedge-optic camera 136 for image sensing.
  • Figure 7E shows another system 137 similar to system 130, but instead of utilizing a wedge backlight unit, the LCD is direct backlit using a diffuser 138 and light sources 139.
  • FIG 8 schematically illustrates a multi-touch system 140 that utilizes an active-matrix LCD 142 that includes within the LCD panel additional optical sensing elements (called herein, for convenience, "LCD/optical sensing panel”).
  • LCD panels that include, essentially, a flat camera onto the panel itself, are known.
  • the sensors within the LCD panels ordinarily are used to sense touch visually, typically through shadows. See, for example, " Active Matrix LCD with Integrated Optical Touch Screen” Willem den Boer, Adi Abileah, Pat Green, Terrance Larsson, Scott Robinson, and Tin Nguyen, SID Symposium Digest 34, 1494 (2003 ).
  • the multi-touch system 140 of the present invention shown in Figure 8 does not utilize the sensors to sense touch directly, that is, through shadows or other direct imaging, but instead integrates the herein-described FTIR techniques with the LCD display so that the sensors within the LCD display are used to sense the light that escapes the optical waveguide.
  • a non-diffusive compliant layer 144 is stacked on an IR-pumped waveguide 146, which is disposed on the LCD panel 142.
  • the LCD backlight unit (BLU) 148 is disposed behind the LCD panel 142 without the inclusion of additional image sensors disposed behind (or in front of) the backlight unit.
  • BLU backlight unit
  • the sensing elements on the LCD are individually equipped with IR filters (as part of the LCD color filter array).
  • an active matrix OLED with integrated sensors may be employed in place of the LCD panel shown in Figure. 8 .
  • FIG 9 schematically illustrates a multi-touch system 160 that employs a flexible OLED 162 (also known as "FOLED").
  • Flexible OLEDs generally are fabricated on plastic substrates that can deform substantially under force. Flexible OLEDs are well known, such as discussed in G. Gustafsson, Y. Cao, G. M. Treacy, F. Klavetter, N. Colaneri, A. J. Heeger, Flexible light-emitting diodes made from soluble conducting polymers, Nature 1992, 357, 477 .
  • FOLED 162 is disposed directly on the compliant layer 164. Upon contact, the force is transmitted through the flexible display (i.e., FOLED 162) to the compliant layer 164.
  • the display is opaque to IR so no IR-filter is necessary. If not, an opaque barrier optionally may be disposed behind the display. Similar to various other embodiments, an image sensor 168 is disposed behind the waveguide 166 to detect the scattered light.
  • FOLED 162 preferably is sufficiently compliant to deform under desired touch sensitivity. The example of Figure 9 advantageously has zero disparity between the display and the interaction surface.
  • Various other flexible-type displays may be used in place of a flexible OLED.
  • so-called “eInk” displays which are discussed in Huitema, H. E. A.; Gelinck, G. H.; van der Putten, J. B. P. H.; Kuijk, K. E.; Hart, C. M.; Cantatore, E.; Herwig, P. T.; van Breemen, A. J. J. M.; de Leeuw, D. M. "Plastic transistors in active-matrix displays" Nature 2001, 414, (6864), 599 may be utilized.
  • Other flexible emissive or reflective displays e.g., electrophoretic, electrochromic
  • electrophoretic, electrochromic may be utilized.
  • FIG 10 schematically illustrates a multi-touch system 180 that employs two flexible layers integrated onto the same substrate, or laminated back to back, where both layers are active.
  • one layer may be an emissive FOLED (emissive layer) and the other layer may be a flexible TFT photodiode array (sensing layer).
  • the two flexible layers combined back to back are called herein, for convenience, a double-sided FOLED and is shown as element 182 in Figure 10 .
  • FOLEDs have been commonly fabricated with per-pixel integrated sensing elements, usually to track aging of the emissive component, and such elements are usually stacked beneath the pixel. These sensing elements are typically shielding from optical signals below the substrate.
  • double-sided FOLED 182 has a display side 182a and an optical sensing array 182b fabricated on its opposite side so that it can sense optical signals incident on that side.
  • Double-sided FOLED 182 is disposed onto a compliant surface 184, which is disposed on top of an IR-pumped waveguide 186.
  • FIG 11 schematically illustrates a multi-touch system 200 that employs a flexible LCD 202 that is sufficiently flexible to deform under the touch of a finger.
  • LCDs developed on plastic substrates that are sufficiently flexible are known, such as disclosed in " Active Matrix Displays Made with Printed Polymer Thin Film Transistors," H. Sirringhaus, S.E. Bums, C. Kuhn, K. Jacobs, J.D. MacKenzie, M. Etchells, K. Chalmers, P. Devine, N. Murton, N. Stone, D. Wilson, P. Cain, T. Brown, A.C.Arais, J. Mills, R.H. Friend Plastic Logic Limited .
  • the embodiment of Figure 11 includes a thin flexible backlight 204 that is disposed between flexible LCD 202 and the compliant layer 206 of the FTIR sensor of the present invention. Similar to various other embodiments, an image sensor 210 is disposed behind the waveguide 208 to detect the scattered light.
  • Figure 12 is a schematic illustration of a multi-touch system 220 similar to the embodiment of Figure 11 , having a flexible LCD 222, and an IR filter 223, but the compliant layer 224 operates as both the diffusing backlight unit for the LCD and the compliant surface disposed on the IR-pumped waveguide 226. Visible light output from backlights 228 (e.g., an array of LEDs), which are disposed behind waveguide 226, is directed onto compliant layer 224, which preferably has strongly diffusive properties, which in turn operates as the backlight unit for the flexible LCD 222.
  • backlights 228 e.g., an array of LEDs
  • compliant layer 224 which preferably has strongly diffusive properties, which in turn operates as the backlight unit for the flexible LCD 222.
  • Figure 13 schematically illustrates a multi-touch system 240 that employs a flexible LCD 242 that contains integrated sensors.
  • flexible LCD 242 is disposed on compliant layer 244, which is disposed on top of the IR-pumped waveguide 246.
  • the emissive sources (backlights 248) are distributed behind waveguide 246.
  • Compliant layer 244 when depressed, contacts waveguide 246 and the resulting scattered light (represented by arrows "c" in Figure 13 ) escapes through compliant layer 244 and is sensed by the sensors integrated within flexible LCD 242. No additional image sensor or IR-filter is necessary in the example shown in Figure 13 .
  • FIG. 14 is a schematic illustration of a multi-touch sensing system 260 that employs such an additional sensor 262 (also called “secondary image sensor” herein).
  • Figure 14 further shows the multi-touch sensing system with previously described compliant layer design type 3 (described with reference to Figure 4 ).
  • the other herein-described compliant layer designs may be employed with a secondary image sensor used to detect other visual information.
  • the multi-touch sensing system with additional sensing capability in accordance with the present invention may include or be incorporated within any of the display devices described herein so long as the secondary image sensor is able to image/sense the element (e.g., a finger) or elements above the system's contact surface (i.e. is not filtered by the aforementioned IR filter), and preferably also does not detect the signal from the primary FTIR image sensor.
  • the secondary image sensor is able to image/sense the element (e.g., a finger) or elements above the system's contact surface (i.e. is not filtered by the aforementioned IR filter), and preferably also does not detect the signal from the primary FTIR image sensor.
  • the secondary image sensor 262 such as shown in Figure 14 , preferably is equipped with a bandpass filter of a different wavelength than that used for the FTIR system. Additional illuminants 264, matched to the wavelength of the secondary image sensor's bandpass filter, aid the parallel sensing system to, among other things, sense fingers or objects hovering when touch is not being sensed by the primary sensor.
  • Various applications of the secondary image sensor include determining identity (e.g., of each finger of a hand ... thumb, index finger, etc.) of each touch region, recognizing position/orientation/gesture of the hand or of other objects, and identifying/recognizing other visual elements and/or movements.
  • Multi-touch Sensing System Incorporating an Auto-Calibration System
  • the (primary) image sensor may be equipped with a band-pass filter (e.g., filter 22) that is matched to the output of the IR emitter (e.g., light source 14).
  • a band-pass filter e.g., filter 22
  • such various embodiments may include an auto-calibration system, such as schematically shown as system 280 in Figure 15 .
  • the band-pass filter switchable bandwidth filter 282
  • the band-pass filter is mounted on a mechanical filter holder that is operable under computer control to be moved in or out of (represented by arrows "d") the optical path.
  • the filter is used to prevent the camera (image sensor 284) from seeing the graphical image output from, for example, projector 288, and displayed on display 290.
  • image sensor 284 (along with appropriate signal processing of the output of image sensor 284) is operable in accordance with the present invention to register the infrared touch image coordinate system precisely to the graphical coordinate system. Registration of the graphical coordinate system to the infrared image is performed as follows. The filter is moved out of the optical path, the computer is instructed to send a point (i.e. fiducial) to be graphically output on the display. This point is sensed by the image sensor, and its coordinate noted. The process is repeated for multiple other points on the screen surface (e.g.
  • image sensor 284 may further be utilized to geometrically align multiple overlapping image projections, such as when larger systems incorporating multiple projectors are used.
  • the IR emitter may be pulsed and synchronized to the shutter of the image sensor to beneficially reduce the amount of ambient light received by the image sensor.
  • the light source can be pulsed at a brighter intensity to increase the signal to noise ratio of the system.
  • the "IR" emitter may be chosen to be of near-UV wavelength.
  • components previously suited for IR are replaced with components for this alternate band, especially since UV can be quite degrading.
  • special additional consideration must be taken into account to protect users from stray emissions, due to the harmfulness of UV.
  • the filters associated with compliant layers described in various embodiments herein, used to shield the sensor from background signal may also be utilized to shield the user from UV light scattered from the optical waveguide.
  • multiple image sensors may be employed to sense the FTIR effect caused by touch, generally to allow for the implementation of a relatively large multi-touch system.
  • the image sensors may be spaced apart from one another along a single axis, multiple axes, along a grid system, or other suitable manner.
  • two or more wavelengths of light are output from the light source and multiple image sensors are employed for detecting the different wavelengths.
  • two sets of LEDs are employed to pump the waveguide, one at, for example, 880nm, and the other at, for example, 950nm. Then, each image sensor images a different wavelength. Different wavelength filters may be employed for this purpose.
  • light must be received (at a particular time and location) by both image sensors for the system to deem the occurrence of a depression (i.e., FTIR response) at such time/location.
  • Well known processing methods may be employed to process both video streams in this matter.
  • FTIR response is further discriminated from background light.
  • a live finger can be discriminated from latent residues in the event a compliant surface is not utilized by recognizing the ratios of intensities for each wavelength that is specific to a human finger. Three or more wavelengths may be employed.
  • the compliant layer may be made of various plastic films and other materials.
  • the compliant layer may be comprised of a composite of multiple materials, each generally contributing to one or more of the following desired characteristics: i) optical FTIR performance; ii) infrared shielding to mitigate interference from external ambient light; iii) anti-glare to increase visibility of the display; iv) the tactile "feel" for the human user (including but not limited to aspects such as friction); and v) durability -- a hardcoat layer preferably replaceable in the field.
  • the various layers employed may be affixed to one another using well-known index-of-refraction matched optical adhesives.
  • a stack includes (1) a thin layer of rubber, (2) a thin-film PET (polyethylene terephthalate) film with metal coating, and (3) a thin PET film treated to have a matte surface.
  • the thin layer of rubber provides for FTIR, along with tactile response and feel.
  • the thin-film PET film with metal coating reflects/absorbs ambient infrared light.
  • the thin PET film treated to have a matte surface provides for a comfortable surface on which a user's finger or fingers can easily glide across, and for wearability.
  • the contact surface is not flat, i.e., non-planar.
  • the contact surface may be concave, convex or other non-flat design.
  • the contact surface may be spherically shaped.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Position Input By Displaying (AREA)

Claims (12)

  1. Système de détection tactile multipoint (10) comprenant :
    un guide d'onde optique (12) ayant un indice de réfraction suffisant pour amener de la lumière d'au moins une longueur d'onde reçue dans une direction prédéterminée à subir une réflexion interne totale dans le guide d'onde optique (12), le guide d'onde optique (12) ayant des premier et second côtés ;
    une couche souple (16) ayant une première surface disposée adjacente et espacée du premier côté du guide d'ondes optique (12), dans lequel de minuscules zones (16a) de la couche souple (16) sont en contact avec le guide d'onde (12) pour maintenir un intervalle d'air (24) ;
    le guide d'onde optique (12) étant un composant composite constitué d'une couche de guide d'onde optique non souple (36, 44) et d'une couche de guide d'onde optique souple (34, 42) disposée sur la couche de guide d'onde optique non souple (36, 44), la couche de guide d'onde optique (34, 42) étant disposée sur le premier côté du guide d'onde optique (12) de sorte que la couche souple (16) est disposée adjacente à la couche de guide d'onde optique souple (34, 42) du guide d'onde optique (12) ;
    la couche souple (16) étant suffisamment déformable lors d'un enfoncement de la couche souple (16) par un objet pour permettre à des parties de la première surface de la couche souple de venir en contact avec le guide d'onde optique (12), et la première surface de la couche souple (16) a un indice de réfraction suffisamment similaire à l'indice de réfraction du guide d'onde optique (12) pour amener, sur la première surface de la couche souple (16) venant en contact avec le guide d'onde optique (12), une partie de la lumière subissant une réflexion interne totale à l'intérieur du guide d'onde optique (12) à se diffuser et à s'échapper du guide d'onde optique (12) ; et
    un capteur d'image (20) adapté pour détecter au moins une partie de la lumière qui s'échappe du guide d'onde optique (12), dans lequel le capteur d'image (20) est disposé optiquement pour recevoir la lumière s'échappant du second côté du guide d'onde optique ;
    caractérisé en ce que la couche souple a une couche de film filtrant (18) disposée sur celle-ci pour filtrer la lumière de ladite au moins une longueur d'onde.
  2. Système de détection tactile multipoint (10) selon la revendication 1, dans lequel la couche souple (16) est déformable en enfonçant une seconde surface de la couche souple par l'intermédiaire d'un doigt d'un utilisateur du système de détection tactile multipoint.
  3. Système de détection tactile multipoint (10) selon la revendication 1, comprenant en outre un filtre passe-bande disposé optiquement entre le second côté du guide d'onde optique (12) et le capteur d'image (20) pour permettre sensiblement uniquement à la lumière de ladite au moins longueur d'onde de passer du guide d'onde optique au capteur d'image.
  4. Système de détection tactile multipoint (10) selon la revendication 1, comprenant en outre une source de lumière disposée directement contre un bord du guide d'onde optique (12) pour émettre de la lumière de ladite au moins une longueur d'onde et reçue par le guide d'onde optique dans au moins ladite direction prédéterminée.
  5. Système de détection tactile multipoint (10) selon la revendication 1, comprenant en outre un projecteur arrière pour projeter une image vidéo vers le second côté du guide d'onde optique (12), et dans lequel la couche flexible (16) et le guide d'onde optique sont sensiblement transparents à la lumière visible de sorte que l'image vidéo affichée peut être vue à travers la couche souple et le guide d'onde optique.
  6. Système de détection tactile multipoint (10) selon la revendication 5, comprenant en outre un diffuseur disposé sur la couche de film filtrant (18) pour diffuser l'image vidéo projetée vers le second côté du guide d'onde optique.
  7. Système de détection tactile multipoint (10) selon la revendication 5, dans lequel la couche souple (16) est adaptée pour diffuser l'image vidéo projetée vers le second côté du guide d'onde optique.
  8. Système de détection tactile multipoint (10) selon la revendication 1, comprenant en outre un film opaque à la lumière de ladite au moins une longueur d'onde disposée sur la couche souple (16) et un projecteur avant pour projeter une image vidéo sur le film.
  9. Système de détection tactile multipoint (10) selon la revendication 1, comprenant en outre un affichage à cristaux liquides, LCD, disposé sur le second côté du guide d'onde optique (12) pour afficher une image vidéo dans une direction vers la couche souple (16), et dans lequel la couche souple et le guide d'onde optique sont sensiblement transparents à la lumière visible de sorte que l'image vidéo affichée peut-être vue à travers la couche souple et le guide d'onde optique.
  10. Système de détection tactile multipoint (10) selon la revendication 9, comprenant en outre une pluralité de sources émissives pour fournir un rétro-éclairage au panneau LCD et un diffuseur de rétro-éclairage disposé entre le panneau LCD et la pluralité de sources émissives.
  11. Système de détection tactile multipoint (10) selon la revendication 9, comprenant en outre une unité de rétro-éclairage de coin à profil mince pour fournir un rétro- éclairage au panneau LCD, dans lequel le capteur d'image (20) est une caméra optique de coin (32).
  12. Système de détection tactile multipoint (10) selon la revendication 1, comprenant en outre combinaison un LCD flexible en et un rétroéclairage flexible couplé à la couche souple (16) pour afficher une image vidéo dans une direction vers un utilisateur du système de détection tactile multipoint, la combinaison du LCD flexible et du rétroéclairage flexible étant suffisamment flexible pour déformer la couche souple, lors d'un enfoncement de la combinaison, de sorte que la couche souple vient en contact avec le guide d'onde optique (12).
EP08782614.5A 2007-08-03 2008-08-04 Détection tactile multipoint par le biais d'une réflexion interne totale frustrée Not-in-force EP2188701B1 (fr)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US95396607P 2007-08-03 2007-08-03
US11/833,908 US8441467B2 (en) 2006-08-03 2007-08-03 Multi-touch sensing display through frustrated total internal reflection
PCT/US2008/072164 WO2009020940A2 (fr) 2007-08-03 2008-08-04 Détection tactile multipoint par le biais d'une réflexion interne totale frustrée

Publications (3)

Publication Number Publication Date
EP2188701A2 EP2188701A2 (fr) 2010-05-26
EP2188701A4 EP2188701A4 (fr) 2011-11-09
EP2188701B1 true EP2188701B1 (fr) 2018-04-18

Family

ID=40341988

Family Applications (1)

Application Number Title Priority Date Filing Date
EP08782614.5A Not-in-force EP2188701B1 (fr) 2007-08-03 2008-08-04 Détection tactile multipoint par le biais d'une réflexion interne totale frustrée

Country Status (3)

Country Link
EP (1) EP2188701B1 (fr)
CN (1) CN101821703A (fr)
WO (1) WO2009020940A2 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11500152B2 (en) 2017-11-29 2022-11-15 Cornell University Elastomeric lightguide coupling for continuous position localization in 1,2, and 3D

Families Citing this family (36)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8441467B2 (en) 2006-08-03 2013-05-14 Perceptive Pixel Inc. Multi-touch sensing display through frustrated total internal reflection
US8144271B2 (en) 2006-08-03 2012-03-27 Perceptive Pixel Inc. Multi-touch sensing through frustrated total internal reflection
US8125468B2 (en) 2007-07-30 2012-02-28 Perceptive Pixel Inc. Liquid multi-touch sensor and display device
US8237684B2 (en) * 2008-09-26 2012-08-07 Avago Technologies Ecbu Ip (Singapore) Pte. Ltd. User input device with planar light guide illumination plate
US8487914B2 (en) * 2009-06-18 2013-07-16 Avago Technologies General Ip (Singapore) Pte. Ltd. Optical fingerprint navigation device with light guide film
WO2010141453A2 (fr) 2009-06-01 2010-12-09 Han Jefferson Y Détection d'effleurement
US8736581B2 (en) 2009-06-01 2014-05-27 Perceptive Pixel Inc. Touch sensing with frustrated total internal reflection
US8624853B2 (en) 2009-06-01 2014-01-07 Perceptive Pixel Inc. Structure-augmented touch sensing with frustated total internal reflection
CN101937289B (zh) * 2009-06-30 2013-06-05 鸿富锦精密工业（深圳）有限公司 光学触控装置
US8730212B2 (en) 2009-08-21 2014-05-20 Microsoft Corporation Illuminator for touch- and object-sensitive display
US8902195B2 (en) * 2009-09-01 2014-12-02 Smart Technologies Ulc Interactive input system with improved signal-to-noise ratio (SNR) and image capture method
GB201004810D0 (en) * 2010-03-23 2010-05-05 St Microelectronics Res & Dev Improvements in or relating to optical navigation devices
US20120069232A1 (en) * 2010-09-16 2012-03-22 Qualcomm Mems Technologies, Inc. Curvilinear camera lens as monitor cover plate
CN102419660A (zh) * 2010-09-28 2012-04-18 扬升照明股份有限公司 触控显示装置及其操作方法
US9535537B2 (en) * 2010-11-18 2017-01-03 Microsoft Technology Licensing, Llc Hover detection in an interactive display device
US8619062B2 (en) * 2011-02-03 2013-12-31 Microsoft Corporation Touch-pressure sensing in a display panel
US8970767B2 (en) 2011-06-21 2015-03-03 Qualcomm Mems Technologies, Inc. Imaging method and system with angle-discrimination layer
TWI543045B (zh) * 2012-04-10 2016-07-21 揚明光學股份有限公司 觸控裝置和觸控投影系統
CN103383610A (zh) * 2012-05-02 2013-11-06 北京通世舟数字科技有限责任公司 一种液晶屏多点触摸方法
ITMI20120813A1 (it) 2012-05-11 2013-11-12 Fond Istituto Italiano Di Tecnologia Dispositivo e processo di rilevazione tattile.
US9207807B2 (en) 2012-05-25 2015-12-08 Young Optics Inc. Vehicular optical touch apparatus
US20140118270A1 (en) * 2012-10-26 2014-05-01 Qualcomm Incorporated System and method for providing infrared gesture interaction on a display
TWI484389B (zh) * 2012-10-31 2015-05-11 Au Optronics Corp 觸控顯示裝置
US20140152914A1 (en) * 2012-11-30 2014-06-05 Corning Incorporated Low-Fe Glass for IR Touch Screen Applications
CN103677448B (zh) * 2013-12-31 2017-03-22 上海交通大学 基于光栅结构的光波导式触摸屏
CN104020896B (zh) * 2014-05-14 2018-10-02 上海交通大学 基于光栅的光学触控屏
CN105526999B (zh) * 2015-12-31 2017-12-19 北京工业职业技术学院 电子皮带秤称重仪表触摸按钮
KR102040651B1 (ko) 2016-04-29 2019-11-06 엘지디스플레이 주식회사 광학식 이미지 인식 센서 내장형 평판 표시장치
CN108664855A (zh) * 2017-03-30 2018-10-16 上海箩箕技术有限公司 指纹成像模组和电子设备
KR102418802B1 (ko) 2017-05-02 2022-07-11 엘지디스플레이 주식회사 표시장치
CN109426790A (zh) * 2017-08-31 2019-03-05 上海箩箕技术有限公司 显示模组
CN108287632A (zh) * 2018-01-19 2018-07-17 昆山国显光电有限公司 压力触控感测模组和显示装置
WO2020047770A1 (fr) * 2018-09-05 2020-03-12 Fingerprint Cards Ab Module de capteur d'empreinte digitale optique et procédé de fonctionnement d'un module de d'empreinte digitale optique
CN113157127B (zh) * 2021-02-04 2024-11-05 识瓴电子科技(南通)有限责任公司 智能表面系统集成的方法
WO2025040386A1 (fr) * 2023-08-23 2025-02-27 Ams-Osram Ag Dispositif capteur, dispositif électronique et procédé de fonctionnement du dispositif capteur
CN119085898A (zh) * 2024-08-06 2024-12-06 业泓科技(成都)有限公司 光学式触觉传感器

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4484179A (en) * 1980-04-16 1984-11-20 At&T Bell Laboratories Touch position sensitive surface
US4668861A (en) 1984-12-12 1987-05-26 The Regents Of The University Of California Tactile sensor employing a light conducting element and a resiliently deformable sheet
AU759440B2 (en) * 1998-01-26 2003-04-17 Apple Inc. Method and apparatus for integrating manual input
US6972753B1 (en) * 1998-10-02 2005-12-06 Semiconductor Energy Laboratory Co., Ltd. Touch panel, display device provided with touch panel and electronic equipment provided with display device
JP3670896B2 (ja) 1999-08-06 2005-07-13 日立ソフトウエアエンジニアリング株式会社 電子ボードシステム
DE10047920A1 (de) * 2000-09-27 2002-04-18 Siemens Ag Vorrichtung zum Erkennen einer Berührungsposition auf einem Bedienfeld
DE50308334D1 (de) * 2002-05-07 2007-11-22 Schott Ag Beleuchtungseinrichtung für Schaltflächen
CN1922470A (zh) * 2004-02-24 2007-02-28 彩光公司 用于平板显示器的光笔和触摸屏数据输入系统和方法
US7598949B2 (en) * 2004-10-22 2009-10-06 New York University Multi-touch sensing light emitting diode display and method for using the same
US20060158437A1 (en) * 2005-01-20 2006-07-20 Blythe Michael M Display device
WO2006081633A1 (fr) * 2005-02-07 2006-08-10 Rpo Pty Limited Modele de guide d'ondes incorporant l'optique reflexive
US7705835B2 (en) * 2005-03-28 2010-04-27 Adam Eikman Photonic touch screen apparatus and method of use
US8013845B2 (en) 2005-12-30 2011-09-06 Flatfrog Laboratories Ab Optical touch pad with multilayer waveguide

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
JAIN S C ET AL: "ELECTRO-OPTIC STUDIES ON POLYMER DISPERSED LIQUID CRYSTAL FILMS PREPARED BY SOLVENT-INDUCED PHASE SEPARATION TECHNIQUE", MOLECULAR CRYSTALS AND LIQUID CRYSTALS(INC. NONLINEAR OPTICS ), GORDON AND BREACH SCIENCE PUBLISHERS, READING, GB, vol. 188, 1 September 1990 (1990-09-01), pages 251 - 259, XP000165253 *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11500152B2 (en) 2017-11-29 2022-11-15 Cornell University Elastomeric lightguide coupling for continuous position localization in 1,2, and 3D
US11614583B2 (en) 2017-11-29 2023-03-28 Cornell University Waveguide and sensor based on same
US11921321B2 (en) 2017-11-29 2024-03-05 Cornell University Elastomeric lightguide coupling for continuous position localization in 1,2, and 3D

Also Published As

Publication number Publication date
EP2188701A4 (fr) 2011-11-09
WO2009020940A3 (fr) 2009-04-02
WO2009020940A2 (fr) 2009-02-12
EP2188701A2 (fr) 2010-05-26
CN101821703A (zh) 2010-09-01

Similar Documents

Publication Publication Date Title
EP2188701B1 (fr) Détection tactile multipoint par le biais d'une réflexion interne totale frustrée
US8144271B2 (en) Multi-touch sensing through frustrated total internal reflection
US8441467B2 (en) Multi-touch sensing display through frustrated total internal reflection
Han Low-cost multi-touch sensing through frustrated total internal reflection
EP2047308A2 (fr) Dispositif d'affichage à détection à touches multiples par l'intermédiaire de réflexion interne totale frustrée
CN101743527B (zh) 基于受抑全内反射的触摸屏
US8581852B2 (en) Fingertip detection for camera based multi-touch systems
Hodges et al. ThinSight: versatile multi-touch sensing for thin form-factor displays
US9348463B2 (en) Retroreflection based multitouch sensor, method and program
JP5346081B2 (ja) ペントラッキングを組み込んだマルチタッチ式タッチスクリーン
US20090267919A1 (en) Multi-touch position tracking apparatus and interactive system and image processing method using the same
US20060290684A1 (en) Coordinate detection system for a display monitor
TWI492128B (zh) 光學觸控顯示裝置
TW200525436A (en) Touch input screen using a light guide
Izadi et al. Thinsight: a thin form-factor interactive surface technology
CN102419660A (zh) 触控显示装置及其操作方法
TWI502417B (zh) 觸控顯示裝置及其操作方法
US20100271337A1 (en) Touch panel and touch display apparatus having the same
CN103365487B (zh) 触控装置和触控投影系统
CN102129330A (zh) 触控屏幕、触控模块及控制方法
TWI897270B (zh) 具感應功能之空中成像裝置

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20100302

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MT NL NO PL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL BA MK RS

A4 Supplementary search report drawn up and despatched

Effective date: 20111007

RIC1 Information provided on ipc code assigned before grant

Ipc: G06F 3/041 20060101AFI20110930BHEP

Ipc: G06F 3/042 20060101ALI20110930BHEP

17Q First examination report despatched

Effective date: 20111027

DAX Request for extension of the european patent (deleted)
GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

INTG Intention to grant announced

Effective date: 20171123

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MT NL NO PL PT RO SE SI SK TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602008054905

Country of ref document: DE

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 991152

Country of ref document: AT

Kind code of ref document: T

Effective date: 20180515

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: FP

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 11

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180718

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180718

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20180724

Year of fee payment: 11

Ref country code: NL

Payment date: 20180712

Year of fee payment: 11

Ref country code: FR

Payment date: 20180712

Year of fee payment: 11

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180719

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20180801

Year of fee payment: 11

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 991152

Country of ref document: AT

Kind code of ref document: T

Effective date: 20180418

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180820

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602008054905

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

26N No opposition filed

Effective date: 20190121

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180804

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180831

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180831

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20180831

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180804

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180831

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180804

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 602008054905

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

REG Reference to a national code

Ref country code: NL

Ref legal event code: MM

Effective date: 20190901

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20190804

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20080804

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180418

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190901

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200303

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190831

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180818

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190804