WO2008141459A1 - Mouth-operated input device - Google Patents
Mouth-operated input device Download PDFInfo
- Publication number
- WO2008141459A1 WO2008141459A1 PCT/CA2008/000986 CA2008000986W WO2008141459A1 WO 2008141459 A1 WO2008141459 A1 WO 2008141459A1 CA 2008000986 W CA2008000986 W CA 2008000986W WO 2008141459 A1 WO2008141459 A1 WO 2008141459A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- lip
- oral cavity
- force
- radiation
- signals
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61F—FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS
- A61F4/00—Methods or devices enabling patients or disabled persons to operate an apparatus or a device not forming part of the body
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H1/00—Details of electrophonic musical instruments
- G10H1/32—Constructional details
- G10H1/34—Switch arrangements, e.g. keyboards or mechanical switches specially adapted for electrophonic musical instruments
- G10H1/344—Structural association with individual keys
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H3/00—Instruments in which the tones are generated by electromechanical means
- G10H3/12—Instruments in which the tones are generated by electromechanical means using mechanical resonant generators, e.g. strings or percussive instruments, the tones of which are picked up by electromechanical transducers, the electrical signals being further manipulated or amplified and subsequently converted to sound by a loudspeaker or equivalent instrument
- G10H3/14—Instruments in which the tones are generated by electromechanical means using mechanical resonant generators, e.g. strings or percussive instruments, the tones of which are picked up by electromechanical transducers, the electrical signals being further manipulated or amplified and subsequently converted to sound by a loudspeaker or equivalent instrument using mechanically actuated vibrators with pick-up means
- G10H3/16—Instruments in which the tones are generated by electromechanical means using mechanical resonant generators, e.g. strings or percussive instruments, the tones of which are picked up by electromechanical transducers, the electrical signals being further manipulated or amplified and subsequently converted to sound by a loudspeaker or equivalent instrument using mechanically actuated vibrators with pick-up means using a reed
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
- G10H2220/155—User input interfaces for electrophonic musical instruments
- G10H2220/361—Mouth control in general, i.e. breath, mouth, teeth, tongue or lip-controlled input devices or sensors detecting, e.g. lip position, lip vibration, air pressure, air velocity, air flow or air jet angle
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
- G10H2220/461—Transducers, i.e. details, positioning or use of assemblies to detect and convert mechanical vibrations or mechanical strains into an electrical signal, e.g. audio, trigger or control signal
- G10H2220/561—Piezoresistive transducers, i.e. exhibiting vibration, pressure, force or movement -dependent resistance, e.g. strain gauges, carbon-doped elastomers or polymers for piezoresistive drumpads, carbon microphones
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H2230/00—General physical, ergonomic or hardware implementation of electrophonic musical tools or instruments, e.g. shape or architecture
- G10H2230/045—Special instrument [spint], i.e. mimicking the ergonomy, shape, sound or other characteristic of a specific acoustic musical instrument category
- G10H2230/155—Spint wind instrument, i.e. mimicking musical wind instrument features; Electrophonic aspects of acoustic wind instruments; MIDI-like control therefor
- G10H2230/195—Spint flute, i.e. mimicking or emulating a transverse flute or air jet sensor arrangement therefor, e.g. sensing angle or lip position to trigger octave change
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10H2250/00—Aspects of algorithms or signal processing methods without intrinsic musical character, yet specifically adapted for or used in electrophonic musical processing
- G10H2250/315—Sound category-dependent sound synthesis processes [Gensound] for musical use; Sound category-specific synthesis-controlling parameters or control means therefor
- G10H2250/461—Gensound wind instruments, i.e. generating or synthesising the sound of a wind instrument, controlling specific features of said sound
Definitions
- This application relates to mouth-operated input devices which may be used as input and/or control devices for a wide variety of systems.
- Particular embodiments of the invention provide mouth-operated input devices for electronic instrument simulators and mouth-operated input devices for operation and/or control of various components and/or systems.
- Non-limiting examples of such systems include: electronic wind instrument simulators; systems for the assistance of disabled individuals who may be without the use of their hands or who may not have hands; systems which allow people to interact with various devices when their hands are otherwise encumbered or occupied (i.e. "hands free” operation); and systems which allow mouth-operated input in addition to more traditional input means to interact with various devices.
- Electronic instrument simulators such as keyboard-activated synthesizers which use hand-operated inputs to emulate a conventional piano and other instruments, are relatively popular.
- Electronic instrument simulators and apparatus for electronically controlling conventional instruments have been disclosed in US patent Nos. 4,085,646; 5,543,580; 5,459,280; 5,149,904; and 7,049,503, for example.
- electronic wind instrument simulators have not achieved the same level of popularity as their conventional hand-operated counterparts.
- One reason for the lack of popularity of electronic wind instrument simulators is the limited ability of prior art simulators to emulate the response of the instrument to the musician's "embouchure".
- Embouchure may generally be described as the way that the musician's mouth (including, without limitation, lips, tongue, cheeks, throat, jaw and teeth) interact with the instrument mouthpiece.
- the sound of the instrument is heavily influenced by embouchure.
- Embouchure may have many different aspects which may related to one or more physical gestures.
- Non-limiting examples of various aspects of embouchure include the force with which the lips are pressed on the mouthpiece, the position of the lips and tongue relative to the mouthpiece, the tilting and/or curvature of the lips and tongue and the spatial characteristics of the oral cavity.
- musicians often learn to control various aspects of their embouchure subconsciously.
- embouchure typically influences the characteristics of the resonant system formed by the musician's mouth, the instrument mouthpiece and the instrument body. It has proven to be difficult to sense and/or simulate embouchure.
- Prior art wind instrument simulators have not been able to provide the sensitivity, dynamic range or multiple degrees of freedom necessary to simulate embouchure in a manner that is satisfactory to musicians or in a manner that allows the wind instrument simulators to faithfully reproduce the variety sounds generated by their conventional counterparts.
- Mouth-operated input devices may have a wide variety of applications other than for the control of instrument simulators. Mouth-operated input devices may enable hands free operation of various components and/ or systems. Mouth-operated input devices may also be used as additional input and/or control devices for systems incorporating more conventional (e.g. hand-operated) input devices. By way of non-limiting example, mouth-operated input devices may be used by disabled individuals with limited capability of using conventional hand-operated input devices and by underwater divers or astronauts whose protective suits limit their ability to use their hands to operate input devices. Such mouth-operated input devices can be used as input and/or control devices for a wide variety of systems.
- Such user-interaction characteristics may include, for example, the force with which the lips and/or tongue are pressed on the input device, the position of the lips and tongue relative to the input device, the tilting and/or curvature of the lips and tongue and the spatial characteristics of the oral cavity.
- Examples of prior art mouth-operated input devices include: • the tongue-controlled microjoystick disclosed at http : //www . asel . udel . edu/robotics/chameleon/chameleon . html ; • the dual-joystick tetramouseTM device disclosed at http://tetramouse.com; • the electricjoyTM joystick device disclosed at http : //www . genesisone . net/electricjoy . htm;
- Figure 1 is schematic diagram of a wind instrument simulator incorporating a mouth-operated input device according to a particular embodiment of the invention
- Figures 2A, 2B, 2C, 2D, 2E and 2F schematically illustrate an oral cavity sensing system suitable for use with the Figure 1 mouth-operated input device
- Figures 3 A and 3B are different isometric views of a mouth-operated input device according to another embodiment of the invention.
- Figure 4 is a schematic block diagram of the oral cavity sensing system of the Figure 3 mouth-operated input device;
- Figures 5A, 5B and 5C (collectively, Figure 5) schematically illustrate a lip sensing system according to a particular embodiment of the invention, which is suitable for use with the Figure 1 mouth-operated input device;
- Figure 6 is a schematic block diagram of a lip sensing system of the Figure 3 mouth-operated input device
- Figure 7A, 7B, 7C (collectively, Figure 7) schematically depict the detection of lip curvature by the lip sensing system of Figure 6;
- Figure 8A, 8B, 8C (collectively, Figure 8) schematically depict the detection of lip angle by the lip sensing system of Figure 6;
- Figure 9 is a schematic diagram depicting the breath-pressure-sensor of the Figure 1 mouth-operated input device; - A -
- Figure 10 is a schematic diagram depicting the breath-pressure sensing system of the Figure 3 mouth-operated input device
- Figure 11 is a schematic block diagram of an instrument according to a stand alone embodiment of the invention
- Figure 12 is a schematic block diagram of an instrument according to a computer-hosted embodiment of the invention
- Figures 13A, 13B and 13C (collectively, Figure 13) schematically illustrate examples of lip sensing systems for mouth-operated input devices according to other embodiments of the invention
- Figure 14 is a schematic depiction of a normalization circuit that may be used to normalize oral cavity sensor output signal(s) in accordance with a particular embodiment of the invention
- Figure 15 is a partial isometric view of a mouth-operated input device according to another embodiment of the invention.
- Figure 16 is a schematic block diagram of the lip sensing system of the Figure 15 mouth-operated input device;
- Figure 17A is an exploded isometric view of a mouth-operated joystick device according to another embodiment of the invention.
- Figure 17B is an exploded isometric view of the force/joystick sensor system of the Figure 17A device
- Figure 18 is a schematic depiction of the Figure 17 A device
- Figure 19 is schematic isometric view of a mouth-operated input device incorporating a system for enabling multi-dimensional control with a user lip according to another embodiment of the invention
- Figure 20 is a schematic isometric view of the Figure 19 mouth- operated input device with the external mouthpiece component removed to show additional detail
- Figure 21 is a schematic isometric view of a force-sensitive-resistor (FSR) used in the Figure 19 mouth-operated input device
- Figures 22A and 22B are schematic isometric views of the external mouthpiece component of the Figure 19 mouth-operated input device.
- Figures 23 A and 23B are schematic isometric views of the internal mouthpiece component of the Figure 19 mouth-operated input device. Detailed Description
- Mouth-operated input devices are provided for interaction with various systems.
- mouth-operated input devices are used to control wind instrument simulators.
- mouth- operated input devices according to the invention sense various characteristics of the user's mouth and output corresponding sensor output signals which may be used as inputs for a variety of different systems.
- a mouth-operated input device comprises a convexity around which a user's lip may deform and an optical sensor for emitting radiation into a region of space on a first side of the convexity. This region of space adjacent the first side of the convexity may comprise a concavity.
- the optical sensor may be configured to detect radiation reflected by the user's Hp as it deforms around the convexity and into the region of space on the first side of the convexity.
- the optical sensor may additionally or alternatively be configured to detect radiation transmitted through the region as the user's lip deforms around the convexity and into the region of space (i.e. blocking the transmission of radiation).
- a first portion of the user's lip may be located on the first side of the convexity and a second portion of the user's lip may be received on an opposing side of the convexity.
- the application of force to the lip may cause the first portion of the user's lip to deform into the region of space adjacent the first side of the convexity.
- An amount of reflected or transmitted radiation detected by the sensor may be related to an amount of deformation of the user's lip into this region of space.
- the application of different levels of force to the lip causes different amounts of lip deformation and different amounts of extension of the first portion of the lip into the region, which in turn causes different amounts of reflected or transmitted radiation detected by the sensor.
- the convexity may be formed by a step profile between two spaced-apart surfaces.
- Another aspect of this invention provides a lip-sensing system for the mouth- operated input device which facilitates multi-dimensional control with a single one of the user's lips.
- Particular embodiments incorporate strategically positioned pluralities of lip force sensors.
- Lip force sensors may comprise force sensitive resistors (FSRs). Such FSRs may be pre-loaded by protrusions to assist the user with reaching the FSR threshold activation pressure. Such protrusions may additionally or alternatively be shaped to concentrate lip force onto relatively small areas of the active regions of the FSRs.
- Mouth-operated input devices may also comprise an optical oral cavity sensor for emitting radiation into the user's oral cavity and detecting reflected radiation from one or more surfaces of the user's oral cavity.
- the oral cavity sensor may be insertable into the user's oral cavity.
- the radiation emitted by the sensor may be oriented such that the reflected radiation detected by the sensor is correlated with a distance between the sensor and a forward facing surface of the user's tongue.
- the radiation emitted by the sensor may be oriented such that the reflected radiation detected by the sensor is correlated with a distance between an apex of the user's tongue and a roof of the user's mouth.
- Mouth-operated input devices may also comprise a force sensor which senses the force that the user's mouth is applying (in one or more directions) to the device itself.
- the mouth- operated input device maps this detected force to a position.
- Figure 1 schematically depicts a wind instrument simulator 10 incorporating a mouth-operated input device 12 according to a particular embodiment of the invention.
- instrument simulator 10 may comprise a plurality of additional inputs 14 which may perform a variety of additional functions.
- inputs 14 may comprise finger operated keys for note selection, timbre manipulation or other control functions.
- Inputs 14 may additionally or alternatively be provided on mouth- operated input device 12.
- Instrument simulator 10 and/or mouth-operated input device 12 may also comprise instrument electronics 16 for generating an output signal 18 based on inputs 14 and/or data received from mouth-operated input device 12.
- instrument electronics 16 comprise a digital processor for generating output signal 18 and output signal 18 may control the audio output of synthesizer 13.
- Output signal 18 may comprise one or more control signal(s) which conform to a known audio control protocol or to some other communication and/or control protocol.
- audio protocols may include: MIDI (Musical Instrument Digital Interface), MIDI over USB (Universal Serial Bus), OSC (Open Sound Control), and SKINI (Synthesis tool Kit Instrument Network Interface).
- output signal 18 may alternatively be used as an input signal or as input parameter(s) to an analog audio synthesizer or a digital audio synthesis algorithm.
- mouth-operated input device 12 comprises a housing 20 having a mouth-engaging end 22 with a proximal surface 24, an upper surface 30 and a lower surface 36.
- a user may interact with mouth-engaging end 22 of housing 20 by placing their upper lip 26 (and possibly one or more of their upper teeth 28) on upper surface 30 and their lower lip 32 (and possibly one or more of their lower teeth 34) on lower surface 36.
- Lower surface 36 and/or upper surface 30 may comprise one or more convexities 37.
- lower surface 36 has a stepped profile comprising a first portion (upper portion 36B) and a second portion (lower portion 36A) located on either side of convexity 37.
- lower surface 36 also comprises a concavity 39 between lower portion 36A and upper portion 36B.
- lower surface 36 may comprise more than one convexity 37 and/or more than one concavity 39.
- Convexity 37 may provide a relatively sharp corner.
- a radius of curvature of convexity 37 is in a range of 0.1 mm - 3 mm.
- the radius of curvature of convexity 37 is in a range of 0.25 mm - 1 mm. In some embodiments, convexity 37 subtends an angle in a range of 30° -150°. In currently preferred embodiments, convexity 37 subtends an angle in a range of 75°-120° .
- housing 20 may contain a number of sensors for sensing various aspects of the interaction between the user's mouth and input device 12.
- input device 12 comprises a breath-pressure-sensor 42, which is connected to the mouth-engaging end 22 by a suitable conduit 44.
- input device 12 also comprises an oral cavity sensor 46 and a lower lip sensor 48.
- Oral cavity sensor 46 may comprise one or more optical reflectivity sensors and lip sensor 48 comprises one or more optical reflectivity sensors and/or one or more optical transmission sensors.
- Housing 20 and/or instrument 10 may also contain signal conditioning electronics 50, which condition the output signals from sensors 42, 46, 48 before providing them to instrument electronics 16.
- Instrument electronics 16 may also be located in housing 20 and/or in instrument 10.
- sensors 42, 46, 48 share signal conditioning electronics 50.
- device 12 and/or instrument 10 comprise independent signal conditioning electronics 50 for each sensor 42, 46, 48.
- signal conditioning electronics 50 may work cooperatively with instrument electronics 16 to receive signals (not shown in Figure 1) from various sensors 42, 46, 48 and to use these sensor signals to generate output signal 18, which may in turn control the audio output of synthesizer 13.
- Mouth-operated input device 12 is capable of sensing a number of user- interaction characteristics related to the way in which a user interacts with device 12. Such user-interaction characteristics may be indicative of various aspects of embouchure.
- Figure 2 schematically depicts how device 12 is capable of sensing spatial characteristics of the user's oral cavity 52 including the user's tongue 54. For clarity, some details of device 12 not related to sensing tongue and/or oral cavity characteristics have been removed from Figure 2.
- Oral cavity sensor 46 my be an optical reflection-type sensor which emits radiation and senses the intensity of reflected radiation.
- Oral cavity sensor 46 comprises a radiation emitter.
- the radiation emitter is a LED emitter 6OA.
- LED emitter 6OA may emit radiation in the visible, infrared or near infrared spectrum. The approximately 400-660 nanometer wavelength common to a wide variety of LEDs represents one particular choice, as saliva is relatively transparent at such wavelengths.
- Oral cavity sensor 46 also comprises a radiation detector for sensing reflected radiation. In the Figure 2 embodiment, the radiation detector is a phototransistor 6OB.
- Optical reflection-type sensors of this type are known in the art. Examples of such sensors include the OPB745 manufactured and sold by Optek Technology, Inc. of Carrollton, Texas.
- sensor 46 comprises discrete components.
- LED emitter 6OA may be implemented by the FA1105W-TR LED from Stanley Electric Sales of America, Inc. of Irvine, California and phototransistor 60B may be implemented by the PT-IOOMC-OMP from Lumex, Inc. of Palatine, Illinois.
- oral cavity sensor 46 comprises, or is optically coupled to, a light pipe 58, such that some component(s) of oral cavity sensor 46 (e.g. LED 6OA and/or phototransistor 60B) may be located remotely (i.e. away from mouth-engaging end 22 of device 12).
- light pipe 58 is not required and LED 6OA and phototransistor 6OB (or other radiation emitting and radiation detecting components of sensor 46) may be located adjacent to proximal surface 24 of input device 12.
- radiation emitted from LED 6OA (or other radiation emitting device) may travel through a different light pipe than radiation that is incident on phototransistor 60B (or other radiation detection device).
- additional optics 62A may be provided for shaping and/or directing the sensor output radiation and additional optics 62B may be provided for coupling remotely located LED 6OA and phototransistor 6OB (or other radiation emitting and radiation detecting components of sensor 46) to light pipe 58.
- additional beam shaping and/or directing optics 62 may comprise, without limitation, lenses, mirrors, prisms, gratings, beam splitters, polarizers and other beam-shaping and/or coupling optical elements.
- Such additional optics 62 may be static or moveable.
- Optics 62 may comprise, or have their outer surfaces coated with, hydrophilic material, such that saliva does not bead on their outer surfaces.
- Optics 62 may comprises polarizer(s) and/or polarizing filter(s). Polarization of the sensor output beam using a polarizing filter may reduce the effect of specular reflection caused by saliva. If reflected radiation is passed through a polarizing filter orthogonal to that of the sensor output beam, then specularly reflected radiation will tend to be blocked by the orthogonal filter because specularly reflected radiation retains its polarization whereas diffusely reflected radiation does not.
- sensor 46 may optionally be controlled by signal 23 from instrument electronics 16 and/or by signal 25 from signal conditioning electronics 50. Instrument electronics 16 and signal conditioning electronics 50 may comprise one or more controllers suitably configured for this purpose. Signals 23, 25 may comprise one way or two way control signals.
- Radiation emitted from sensor 46 may be reflected from one or more surfaces of the user's mouth, including, for example, lips 26, 32, teeth 28, 34, tongue 54, roof 53, cheeks (not shown) and throat 56.
- One characteristic of the user's oral cavity 52 is the position of the user's tongue 54 relative to the front of oral cavity 52 and/or relative to device 12 (i.e. the forward-backward position of tongue 54).
- Oral cavity sensor 46 is capable of detecting radiation reflected from a "forward-facing surface" 51 of tongue 54.
- "Forward-facing surface” 51 may be defined as any surface of tongue 54 capable of directly receiving radiation emitted from sensor 46. Radiation reflected from forward-facing surface 51 of tongue 54 may be sensed by oral cavity sensor 46 and may be used to obtain an oral cavity sensor output signal 64 which is related to the position of the user's tongue 54 relative to the front of oral cavity 52.
- oral cavity sensor 46 is designed such that oral cavity sensor output signal 64 changes monotonically as forward-facing surface 51 of tongue 54 moves forward or backward within oral cavity 52.
- Monotonic variation of oral cavity sensor output signal 64 with the position of forward-facing surface 51 of tongue 54 may be achieved by designing and/or selecting various parameters of oral cavity sensor 46 including, by way of non-limiting example: angular beam shape or profile, beam width, beam orientation, uniformity of illumination across the beam and polarization of the beam. These parameters may also be designed and/or selected to provide oral cavity sensor output signal 64 with a suitably high dynamic range.
- Figures 2A-2D illustrate various possible positions of the user's tongue 54.
- the user has configured their tongue 54, such that forward-facing surface 51 is positioned relatively close to proximal surface 24 of device 12.
- the user's tongue 54 is configured such that its forward-facing surface 51 is further from proximal surface 24 of device 12.
- the user has curved their tongue 54 such that tongue 54 is pointed upwardly and forward- facing surface 51 is actually the surface typically referred to as the undersurface of tongue 54.
- forward-facing surface 51 of tongue 54 is positioned relatively close to proximal surface 24 of device 12.
- forward-facing surface of tongue 54 is positioned relatively far from proximal surface 24 of device
- Another characteristic of the user's oral cavity 52 is the height of the apex 55 of the user's tongue 54 relative to roof 53 of oral cavity 52 and to device 12. This oral cavity characteristic may be correlated to a cross-sectional area of oral cavity 52 in the region of apex 55.
- the user has configured their tongue 54 and/or their jaw (not shown), such that the apex 55 of tongue 54 is relatively far from the roof 53 of their mouth and the cross-sectional area of the user's oral cavity 52 in the region of apex 55 is relatively large.
- the user has configured their tongue 54 and/or their jaw, such that the apex 55 of tongue 54 is relatively close to the roof 53 of their mouth and the cross-sectional area of the user's oral cavity 52 in the region of apex 55 is relatively small.
- apex 55 of tongue 54 may also be adjusted relative to device 12 when tongue 54 is pointed upwardly as shown in Figures 2C and 2D. In such a configuration, apex 55 is at or near the tip of tongue 54.
- Figure 2C the user has adjusted their tongue 54 and/or their jaw such that apex 55 is relatively far from the roof 53 of oral cavity 52 and in Figure ID, the user has adjusted their tongue 54 and/or their jaw such that apex 55 is relatively close to the roof 53 of oral cavity 52.
- a musician may change the shape and/or size of the resonant cavity in their mouth and may thereby change the characteristics of the resonant cavity formed by the combination of the instrument and the musician's mouth. These changes in the resonant cavity may cause corresponding changes in the frequency, pitch, timbre or other characteristics of the sound created by the instrument.
- the spatial characteristics of the musician's oral cavity represent an aspect of the musician's embouchure.
- Oral cavity sensor 46 senses the spatial characteristics of the user's oral cavity 52 (including the user's tongue 54) and creates an oral cavity sensor output signal 64 representative of these spatial characteristics.
- Signal processing electronics 50 and/or instrument electronics 16 may use oral cavity sensor output signal 64 to generate one or more processed oral cavity signals 17.
- Instrument electronics 16 may further process processed oral cavity signals 17 to generate output signal 18.
- Output signal 18 may in turn control the audio output of synthesizer 13 ( Figure 1).
- Oral cavity sensor output signal 64 may be processed by signal conditioning electronics 50 and/or instrument electronics 16.
- the processing performed on oral cavity sensor output signal 64 by signal conditioning electronics 50 and/or instrument electronics 16 may take place in the analog and/or digital domain and may generally involve any of a variety of processing techniques, including, without limitation: digitizing, amplifying, inverting, filtering, scaling, offsetting, linearizing, differentiating, integrating, averaging, normalizing, performing linear and/or nonlinear mathematical transformations (e.g. Fourier transforms and logarithmic functions) and the like.
- the result of the processing by signal conditioning electronics 50 and/or instrument electronics 16 is one or more processed oral cavity signal(s) 17, which may generally comprise any function & ⁇ of oral cavity sensor output signal 64.
- the function & used to generate processed oral cavity signal(s) 17 may include any of the analog or digital processing techniques discussed above.
- FIG. 14 schematically illustrates a normalization circuit 61 according to a particular embodiment of the invention, which represents a technique for processing oral cavity sensor output signal 64 to generate a normalized oral cavity signal 64' according to a particular embodiment of the invention.
- normalization circuit 61 may be implemented as a part of signal conditioning electronics 50 and/or instrument electronics 16. Normalization circuit 61 normalizes oral cavity sensor output signal 64 to account for variations in the amount of ambient light in the environment in which device 12 is being used. Such ambient light may penetrate the cheeks of a user and be detected by optical sensor 46, for example. Normalization circuit 61 or components of normalization circuit 61 may be controlled by a controller (not shown).
- Normalization circuit 61 takes two temporally spaced apart samples of oral cavity sensor output signal 64. A first sample 64 A is taken when LED 6OA (see
- Figure 2 is ON and a second sample 64B is taken a short time ( ⁇ t) later when LED 6OA is OFF.
- Sample 64B represents the ambient light level detected by oral cavity sensor 46.
- the time ( ⁇ t) between the first and second samples may be less than about 1 ms. In particular embodiments, the time ( ⁇ t) is less than about 100 ⁇ s.
- Oral cavity sensor output signal samples 64 A, 64B are amplified by corresponding linear amplifiers to provide amplified oral cavity sensor output signal samples 64A', 64B'.
- Circuit 61 comprises a sample and hold circuit 71 which is used to temporarily retain one of the one of the amplified oral cavity sensor output signal samples (sample 64 A' in the illustrated embodiment).
- Amplified oral cavity sensor samples 64A', 64B' are then provided to difference amplifier 59, which amplifies a difference between amplified oral cavity sensor samples 64A', 64B' to provide difference signal 19.
- difference signal 19 is representative of the normalized oral cavity output signal (i.e. an oral cavity output signal that is attributable directly to the signal from LED 6OA and not to ambient radiation).
- circuit 61 also comprises optional signal conditioning circuitry 73A and a D/A converter 73B which digitize difference signal 19 to output normalized oral cavity signal 64' .
- Optional signal conditioning circuitry 73A may comprise a low pass filter and additional amplification for example.
- Normalization circuit 61 represents one particular circuit for processing oral cavity sensor output signal 64 to generate a normalized oral cavity output signal 64'. Normalized oral cavity sensor output signal 64' may be further processed using other signal processing circuits and/or processing algorithms to produce one or more different processed oral cavity output signal(s) 17 ( Figure 2). In some embodiments, normalization circuit 61 is not required. Those skilled in the art will appreciate that other normalization circuits may be envisaged to reduce the impact of ambient light on processed oral cavity output signal(s) 17. In particular, the difference between the ambient oral cavity signal (i.e. when the emitter is inactive) and the oral cavity signal when the emitter is active may be obtained in the digital (rather than analog) domain.
- Processed oral cavity signal(s) 17 may be used by instrument electronics 16 to generate output signal 18 which may in turn control the audio output of synthesizer 13.
- Processed oral cavity signal(s) 17 are related to the sensed spatial characteristics of the user's oral cavity 52. Accordingly, using mouth-operated input device 12, the audio output of synthesizer 13 may be controlled by the spatial characteristics of the user's oral cavity 52. Such spatial characteristics may include: the volume of oral cavity 52, the distance of the apex 55 of tongue 54 from the proximal surface 24 of device 12, the distance between the apex 55 of tongue 54 and the roof 53 of the user's mouth, and/or the cross-sectional area of oral cavity 52 in the region of apex 55, for example.
- Processed oral cavity signal(s) 17 may additionally or alternatively be correlated to the rate of change of any of these sensed spatial characteristics of the user's oral cavity 52.
- signal conditioning electronics 50 and instrument electronics 16 are separate components. Signal conditioning electronics 50 may additionally or alternatively be a part of oral cavity sensor 46, instrument electronics 16 and/or synthesizer 13. Instrument electronics 16 may also be a part of signal conditioning electronics 50 and/or synthesizer 13. Depending on the nature of signal conditioning electronics 50 and instrument electronics 16, oral cavity output signal 64 may have many forms. In the illustrated embodiment, signal conditioning electronics 50 cooperate with instrument electronics 16 to generate audio output signal 18 based at least in part on oral cavity sensor output signal 64.
- the radiation emitted by oral cavity sensor 46 may have a conical shape with a divergence profile in a range of 20° -100° .
- a divergence profile provides oral cavity sensor 46 (and oral cavity sensor signal 64) with a suitably high dynamic range which can accurately detect small changes in the spatial characteristics of oral cavity 52.
- the radiation emitted by oral cavity sensor 46 may be substantially uniform throughout the conical profile.
- the radiation emitted from oral cavity sensor 46 may be oriented slightly downwardly (i.e. toward throat 56), so that sensor 46 may accurately sense reflection from tongue 54.
- the radiation emitted from sensor 46 may be oriented at a downward angle in a range of 0°-80° with respect to a longitudinal axis 21 (see Figure 1) of input device 12 (i.e. an axis of input device 12 which extends into the user's mouth and which may be generally parallel with a roof 53 of the user's mouth).
- the radiation emitted from sensor 46 is oriented at a downward angle in a range of 0°-45° with respect to the longitudinal axis 21 of input device 12. In currently preferred embodiments, this angular range is 5°-30° .
- the radiation emitted from oral cavity sensor 46 may alternatively be oriented slightly upwardly.
- the radiation emitted from sensor 46 is oriented at an upward angle in a range of 0°-45° with respect to the longitudinal axis 21 of input device 12. In currently preferred embodiments, this angular range is 5°-30°.
- the orientation with which the user holds instrument 10 (Figure 1) and input device 12 relative to their mouth will impact the angle of radiation emitted from oral cavity sensor 46 relative to throat 56. That is, the user may be capable of altering the orientation of the longitudinal axis 21 of input device 12 and, in so doing, may also be capable of altering the angle of radiation emitted from oral cavity sensor 46.
- the radiation emission angle of oral cavity sensor 46 is made adjustable, for example by one or more suitable mechanical micro-manipulators.
- a suitable micro-manipulator may be an adjustment screw similar to the type used in corrective eyewear, for example.
- a user may also purposefully adjust the orientation of instrument 10 and input device 12 to influence the oral cavity reflectivity response characteristics.
- FIG. 3 depicts a mouth-operated input device 212 according to another embodiment of the invention.
- mouth-operated input device 212 of Figure 3 is similar to mouth-operated input device 12 of Figure 1.
- Features of device 212 which are similar to features of device 12 are referred to using similar reference numerals that are preceded by the digit "2".
- One aspect of device 212 that differs from device 12 is that device 212 has an oral cavity sensing system 209 which comprises two oral cavity sensors 246', 246".
- oral cavity sensors 246', 246" are transversely spaced apart from one another, although they may be spaced apart in some other direction.
- FIG 4 is a schematic block diagram of oral cavity sensing system 209 of the Figure 3 input device 212 according to a particular embodiment of the invention.
- Each of oral cavity sensors 246', 246" may be implemented in a manner similar to, and have characteristics similar to, oral cavity sensor 46 of device 12 described above.
- the radiation emitted from oral cavity sensors 246', 246" may be oriented in different directions. In the illustrated embodiment, radiation emitted from sensor 246' is oriented slightly downwardly (i.e. toward tongue 54 and throat 56) and radiation emitted from sensor 246" is oriented slightly upwardly (i.e. toward roof 53 of the user's mouth).
- the radiation emitted from sensor 246' may be oriented at a downward angle in a range of 0°-80° with respect to the longitudinal axis 221 of input device 212 and the radiation emitted from sensor 246" may be oriented at an upward angle in a range of 0°-80° with respect to the longitudinal axis 221 of input device 212.
- sensor 246' is oriented at a downward angle in a range of 0°-45° and the radiation emitted from sensor 246" is oriented at an upward angle in a range of 0°-45°.
- these angular range are 5°-30°.
- mouth-operated input device 212 and its oral cavity sensors 246', 246" sense different spatial characteristics of the user's oral cavity 52.
- oral cavity sensing system 209 may comprise an optional, additional or alternative pair of oral cavity sensors 246 A' 246A".
- Additional or alternative oral cavity sensors 246 A' 246 A" may be oriented such that sensor 246 A' is oriented at a first transverse (i.e.
- sensor 246 A" is oriented at an opposing transverse angle with respect to the longitudinal axis 221 of device 212.
- the opposing transverse angles of sensors 246A', 246 A" may be in a range of 0°-90° with respect to the longitudinal axis 221 of device 212. In currently preferred embodiments, thee angular range are 10° -45° .
- oral cavity sensors 246', 246" sense spatial characteristics of the user's oral cavity 52 and generate corresponding oral cavity sensor output signals 264', 264" representative of these spatial characteristics.
- Oral cavity sensor output signals 264', 264" may be related to: the volume of oral cavity 52, the distance of the forward-facing surface 51 of tongue 54 from the proximal surface 24 of device 12, the distance between the apex 55 of tongue 54 and the roof 53 of the user's mouth (which may be dependent on tongue shape and jaw angle) and/or the cross-sectional area of oral cavity 52 in the region of apex 55 (which may also be dependent of tongue shape and jaw angle).
- Oral cavity sensor output signals 264', 264" may be normalized using a circuit similar to circuit 61 ( Figure 14) or some other normalization process to substract out the effect of ambient light.
- Oral cavity sensor output signals 264', 264" may be used to generate one or more processed oral cavity signals 217.
- Instrument electronics 216 use processed oral cavity signals 217 to generate output signal 218.
- oral cavity sensor output signals 264', 264" are used to generate two processed oral cavity signals 217A, 217B.
- Instrument electronics 216 may make use of one or both of processed oral cavity signals 217A, 217B to generate output signal 218.
- Output signal 218 may in turn control the audio output of synthesizer 13 ( Figure 1).
- oral cavity sensor output signals 264', 264" are used to generate a different number of processed oral cavity signal(s) 217.
- oral cavity sensor output signals 264', 264" may be processed in the analog and/or digital domain by their corresponding signal conditioning electronics 250', 250" and/or by instrument electronics 216.
- Signal conditioning electronics 250', 250" and instrument electronics 216 may have features similar to signal conditioning electronics 50 and instrument electronics 16 described above.
- processed oral cavity signals 217A, 217B may be any function & of one or both of oral cavity sensor output signals 264', 264".
- the function £F used to generate processed oral cavity signals 217A, 217B may include any of the analog or digital processing techniques discussed above.
- oral cavity sensor output signals 264', 264" may be normalized using a circuit similar to circuit 61
- ⁇ " dA+e
- Processed oral cavity signals 217A, 217B may generally comprise any function ⁇ of one or both of normalized oral cavity sensor output signals 264', 264". [0064] Without being bound by any particular theory, it is thought that one of the resonant frequencies of a user's oral cavity 52 is related to the volume of the cavity between the forward-facing surface 51 of the tongue 54 and user's lips 26, 32. Those skilled in the art will appreciate that a processed oral cavity signal 217A, 217B representative of the location of forward-facing surface 51 relative to the proximal surface 24 of device 12 may be used as a basis for approximating this resonance.
- a processed oral cavity signal 217A, 217B representative of the height of apex 55 relative to roof 53 may be used as a basis for approximating this resonance.
- Figure 4 depicts optional, additional or alternative oral cavity sensors 246A', 246A".
- These oral cavity sensors 246A', 246A” may generate oral cavity sensor output signals 264A', 264A", which may be processed by signal conditioning electronics 250A', 250A” and by instrument electronics 216.
- the processing of oral cavity sensor output signals 264A', 264A” may be similar in many respects to that discussed above for oral cavity sensor output signals 264', 264” .
- oral cavity sensor output signals 264A', 264 A” may be normalized using a circuit similar to circuit 61 ( Figure 14) as a part of any processing performed by signal conditioning electronics 250A', 250A” and/or instrument electronics 216.
- circuit 61 Figure 14
- Oral cavity sensor output signals 264A', 264A" may be used by instrument electronics 216 to help generate processed oral cavity signals 217A, 217B.
- processed oral cavity signals 217A, 217B may generally comprise any function & ⁇ of oral cavity sensor output signals 264', 264" and/or oral cavity sensor output signals 264A', 264A" .
- Oral cavity sensor output signals 264A', 264A" may also be used by instrument electronics 216 to generate additional or alternative processed oral cavity signals 217 (not shown in Figure 4). Such additional or alternative processed oral cavity signals 217 may generally be any function ⁇ ⁇ of oral cavity sensor output signals 264', 264" and/or oral cavity sensor output signals 264A', 264A". [0068] In one particular example, where sensors 246A', 246 A" are oriented at slightly transverse (i.e. sideways) angles (as described above), oral cavity sensor output signals 264A', 264A" may be related to the side to side position of tongue 54 in oral cavity 52.
- a processed oral cavity signal 217 which has been high-pass filtered may be related to the gesture of "flutter tongue" .
- Flutter tongue is a gesture used by conventional wind instrument musicians, where the musician vibrates their tongue at a relatively high frequency in a manner similar to that involved in rolling an 'R' sound in speech.
- Another example of generating a processed oral cavity signal 217 involves applying a threshold filter to one of oral cavity sensor output signal 264', 264", 264A', 264 A" and/or some combination of oral cavity sensor output signals 264', 264", 264A', 264A".
- a processed oral cavity signal 217 which is above a certain threshold may indicate that the user's tongue is touching proximal surface 24 which may be related to the gesture of "tonguing".
- Tonguing is a gesture which may used by conventional wind instrument musicians to articulate spaces between notes by influencing the flow of air through the instrument.
- Another form of tonguing is a gesture which may be used by players of conventional reed instruments to stop the vibration of the reed with the surface of their tongue.
- oral cavity sensing system 209 incorporates an additional sensor (not shown) for detecting a signal representative of the short range proximity of the end of the user's tongue 54.
- a sensor may also be an optical reflection-type sensor and the sensor and/or its signal conditioning electronic may be specifically designed and/or selected for short range proximity detection or threshold proximity detection.
- Such a sensor may be oriented sharply downwardly from proximal surface 224 of device 212 and can be used to detect when tongue 54 touches mouthpiece 212 at or near proximal surface 224. This sensor may also be capable of providing a sensor output signal related to the concept of "tonguing" .
- Instrument electronics 216 may make use of processed oral cavity signals 217A, 217B (and any additional processed oral cavity signals 217 which are not shown in Figure 4) to generate output signal 218 which may in turn control the audio output of synthesizer 13 ( Figure 1).
- processed oral cavity signals 217A, 217B are related to a pair of sensed spatial characteristics of the user's oral cavity 52 (e.g. the distance between the apex 55 of tongue 54 and the roof 53 of the user's mouth and the distance of the forward-facing surface 51 of tongue 54 from the proximal surface 24 of device 12).
- mouth-operated input device 212 may be used to independently control the audio output of synthesizer 13 by way of two different spatial characteristics of the user's oral cavity
- 264A" may be processed to generate any number of processed oral cavity signal(s) 217, each of which may (alone or in combination with other oral cavity sensor output signals 264', 264", 264A', 264A") be related to a different spatial characteristic of the user's oral cavity.
- the provision of multiple oral cavity sensors 246', 246", 246A', 246 A” allows for a model of the user's oral cavity that has multiple degrees of freedom (i.e. multiple processed oral cavity signals 217 that may relate to a corresponding plurality of physical gestures/characteristics).
- Processed oral cavity signal(s) 217 may additionally or alternatively be correlated to the rate of change of any of these sensed spatial characteristics of the user's oral cavity 52.
- Figure 5 schematically depicts how mouth-operated input device 12 (Figure 1) is capable of sensing the force applied to the mouth-engaging end 22 of device 12 by lower lip 32 according to a particular embodiment of the invention. For clarity, details of device 12 which are not related to sensing the force applied by lower lip 32 are not shown in Figures 5.
- the user's lower lip 32 contacts mouth-engaging end 22 of device 12 at or near convexity 37.
- Lower lip 32 comprises an inward portion 32A (located on an inward side 37A of convexity 37) and an outward portion 32B (located on an outward side 37B of convexity 37). Outward portion 32B of lip 32 may be received on lower portion 36A of lower surface 36.
- inward portion 32A of lower lip 32 is not substantially deformed.
- inward portion 32A and outward portion 32B of lip 32 both extend upwardly approximately to the level of lower portion 36A of lower surface 36.
- the user is applying an intermediate force with inward portion 32A of lower lip 32, such that inward portion 32A of lip 32 deforms around convexity 37 and extends upwardly above lower portion 36A and into a region of space 33 on inward side 37A of convexity 37 and adjacent intermediate surface 40.
- Region 33 may be located between convexity 37 and concavity 39.
- Region 33 may be located between a plane substantially parallel with a portion of lower portion 36A immediately outside 37B of convexity 37 and a plane substantially parallel with a portion of upper portion 36B on the inside 37A of convexity 37.
- outward portion 32B of lower lip 32 need not change significantly as the user applies more lip-force to inner portion 32A of lower lip 32.
- Outward portion 32B of lower lip 32B may continue to rest on lower portion 36A of lower surface 36 and may be independently controlled by the user for a different purpose as discussed in more detail below.
- the aperture between the outward portions of upper lip 26 and lower lip 32 is maintained at a relatively constant size, which is approximately equal to the circumference of device 12.
- inward portion 32A of lip 32 deforms around convexity 37 and extends a significant distance above lower portion 36A and into region 33.
- Inward portion 32A of lip 32 may also deform such that it is closer to intermediate surface 40.
- inward portion 32A of lip 32 may reach upper portion 36B of lower surface 36 and may become flattened against upper portion 36B.
- inward portion 32A of lip 32 may reach intermediate surface 40 and may become flattened against intermediate surface 40.
- mouth-operated input device 12 comprises a lip sensor 48.
- Lip sensor 48 may be a optical reflection-type sensor which emits radiation and senses the intensity of reflected radiation of the same general type as oral cavity sensor 46 discussed above. Lip sensor 48 may be oriented to direct radiation into region 33 on inward side 37A of convexity 37. In the Figure 5 embodiment, lip sensor 48 is positioned to direct radiation out of intermediate surface 40 and into region 33. In other embodiments, lip sensor 48 is positioned to direct radiation out of upper surface 36B and into region 33.
- lip sensor 48 may comprise, or may be optically coupled to, a light pipe 66, such that some components of lip sensor 48 (e.g. radiation emitter 7OA and radiation detector 70B) may be located remotely. In other embodiments, light pipe 66 is not required and radiation emitter 7OA and radiation detector 7OB may be located and/or oriented to direct radiation directly into region 33.
- Lip sensor 48 and/or light pipe 66 may comprise additional optics 68A for shaping and/or directing the sensor output radiation and/or additional optics 68B for coupling remotely located radiation emitter 7OA and detector 7OB to light pipe 66. Optics 68 may be similar to optics 62 discussed above.
- region 33 Radiation emitted from lip sensor 48 into region 33 is reflected from inward portion 32A of the user's lower lip 32.
- the shape of the mouth-engaging end 22 of device 12 i.e. providing region 33 on inward side 37A of convexity 37
- region 33 is created by the stepped profile of lower surface 36 (i.e. spaced-apart lower and upper portions 36A, 36B), which permit lower lip 32 to deform around convexity such that inward portion 32A of lip 32 extends into region 33.
- Figure 5 A shows that when the force applied by the user to the inward portion 32A of lower lip 32 is relatively small, lower portion 36A and convexity 37 maintain the inward portion 32 A of lower lip 32 at a location that is, for the most part, at or below the level of lower portion 36A (i.e. not extending substantially into region 33).
- lip sensor 48 detects relatively little reflected radiation, because a significant portion of radiation 72 emitted by sensor 48 either does not impinge on lower lip 32 or reflects from lower lip 32, but is not received by sensor 48.
- Lip sensor 48 detects a larger amount of reflected radiation when the force applied to inward portion 32A of lower lip 32 is at the intermediate level of Figure 5B, because inward portion 32A of lower lip 32 deforms around convexity 37, extends above lower portion 36A and into region 33 on inward side 37A of convexity 37. When inward portion 32A of lip 32 extends into region 33, it reflects more radiation back toward sensor 48.
- Figure 5C shows that when the force applied to inward portion 32A of lower lip 32 is at a relatively high level, inward portion 32A of lower lip 32 deforms around convexity 37, extends significantly above lower portion 36A and into region 33.
- inward portion 32A of lip 32 is forced harder, it may deform further by flattening against upper portion 36B and/or against intermediate surface 40.
- lip sensor 48 detects even more reflected radiation, because a larger percentage of the radiation emitted by lip sensor 48 is reflected by inward portion 32A of lip 32 and detected by sensor 48.
- a musician When playing a conventional wind instrument, a musician frequently (and often subconsciously) changes the force applied by their lip(s) on the mouthpiece of the instrument to modulate the sound emanating from the instrument. For example, by applying different levels of force, a musician may alter the frequency, pitch, timbre or other characteristics of the instrument and thereby change the sound created by the instrument.
- the force applied by the musician's lip(s) represents an aspect of the musician's embouchure.
- Device 12 models this aspect of a conventional instrument.
- Lip sensor 48 senses the force applied by inward portion 32A of the user's lip 32 and creates a lip sensor output signal 76 representative of this force.
- Lip sensor output signal 76 may be used by signal conditioning electronics 50 and/or instrument electronics 16 to generate one or more processed lip signals 77.
- Instrument electronics 16 may then use processed lip signals 77 to generate output signal 18.
- Output signal 18 may in turn be used to control the audio output of synthesizer 13 ( Figure 1).
- Those skilled in the art will appreciate that the size of region 33 (i.e.
- the distance between upper portion 36B and lower portion 36A) and/or other parameters of lip sensor 48 may be designed so that lip sensor output signal 76 is a monotonically increasing function of the force applied by inward portion 32 A of lip 32 to mouth-engagement end 22 of device 12.
- the spacing between upper and lower portions 36 A, 36B is in a range of 2-15 mm.
- Other aspects of device 12 which may be used to provide a monotonic relationship between lip sensor output signal 76 and the force applied by inward portion 32A of lip 32 include, without limitation: angular beam shape or profile, beam width, beam orientation, uniformity of illumination across the beam, polarization of the beam, the radius of curvature of convexity 37 and reflectivity of mouthpiece 12. These parameters may also be designed to provide Hp sensor output signal 76 with a suitably high dynamic range which varies between about zero applied force to the maximum lip-force that a normal user may apply.
- lip sensor output signal 76 is processed by signal conditioning electronics 50 and/or instrument electronics 16 to output a single processed lip signal 77.
- lip sensor output signal 76 may be processed by signal conditioning electronics 50 and/or instrument electronics 16 to output a plurality of processed lip signal(s) 77.
- Signal conditioning electronics 50 and instrument electronics 16 may function in a manner similar to, and have characteristics similar to, those discussed above with respect to oral cavity sensor output signal 64.
- signal conditioning electronics and/or instrument electronics 16 may comprise a normalization circuit that operates to provide normalized lip sensor output signals 76. Such a normalization circuit may operate in a manner substantially similar to normalization circuit 61 ( Figure 14) discussed above for oral cavity sensor output signal 64. For the discussion that follows, it is assumed that lip sensor output signal 76 is normalized.
- Processed lip signal 77 may generally be any function W of lip sensor signal 76. Instrument electronics 16 may use processed lip signal 77 to generate output signal 18 which in turn controls the audio output of synthesizer 13. Processed lip signal 77 is related to the sensed lip-force applied by the user to inward portion 32A of lower lip 32. Accordingly, mouth-operated input device 12 may be used to modulate the audio output of synthesizer 13 based on the lip-force applied by a user. Processed lip signal(s) 77 may additionally or alternatively be correlated to the rate of change of the sensed lip-force applied by the user.
- Convexity 37 and/or the spaced apart lower and upper portions 36A, 36B of lower surface 36 may also function to provide a user with tactile feedback. The user will be able to feel their lip 32 deforming around convexity 37 and into region 33 as they apply greater lip-force. Convexity 37 and/or the spaced apart lower and upper portions 36A, 36B of lower surface 36 also provide the user with an identifiable "home position" around which the user can control their lip-force and lip-position with high precision.
- the user may change the position of their lip 32 with respect to region 33 by moving their lip 32 or by moving instrument 10 to reflect differing amounts of radiation into lip sensor 48 and to thereby influence lip sensor output signal 76, processed lip signal 77, output signal 18 and the sound created by instrument 10 ( Figure 1).
- a user may also move their lower lip 32 and/or instrument 10 to change the relative size of lower lip portions 32A, 32B (i.e. the amount of lip 32 that is on either side of convexity 37).
- Mouth-operated input device 212 of the Figure 3 embodiment incorporates a lip sensing system 211 which comprises three lip sensors 248L, 248C and 248R.
- Lip sensors 248L, 248C, 248R are oriented to direct radiation into region 233 on inward side 237A of convexity 237.
- lip sensors 248L, 248C, 248R are provided at transversely spaced apart locations and are configured to direct radiation out of intermediate surface 240.
- Lip sensors 248L, 248C, 248R may be used to sense the force applied to inward portion 32A of lip 32 in a manner similar to lip sensor 48 of device 12.
- Lip sensors 248L, 248C, 248R may be also able to sense other characteristics of a user's lower lip 32 as discussed in more detail below.
- Figure 6 is a schematic block diagram of lip sensing system 211 of the Figure 3 input device 212 according to a particular embodiment of the invention. Each of lip sensors 248L, 248C, 248R may be implemented in a manner similar to, and have characteristics similar to, lip sensor 48 of device 12 described above. [0094] Lip sensors 248L, 248C, 248R may optionally be controlled by signals 207L, 207C, 207R from instrument electronics 216 and/or by signals 215L, 215C, 215R from signal conditioning electronics 250L, 250C, 250R.
- Instrument electronics 216 and signal conditioning electronics 250L, 250C, 250R may comprise one or more controllers suitably configured for this purpose.
- Signals 207L, 207C, 207R, 215L, 215C, 215R may comprise one way or two way control signals.
- lip sensors 248L, 248C, 248R sense characteristics of the user's lower lip 32 and create corresponding lip sensor output signals 276L, 276C, 276R representative of these characteristics.
- Lip sensor output signals 276L, 276C, 276R may be correlated to the force that the user applies to their lower lip 32, the position of lower lip 32 with respect to device 212, the shape of lower lip 32 (e.g.
- Lip sensor output signals 276L, 276C, 276R may be processed by their respective signal conditioning circuits 250L, 250C, 250R and/or by instrument electronics 216 to generate one or more processed lip signals 277.
- Instrument electronics 216 may use one of more processed lip signals 277 to generate output signal 218 on the basis of these characteristics.
- lip sensor output signals 276L, 276C, 276R are used to generate three processed lip signals 277A, 277B, 277C, each of which may be used independently by instrument electronics 216 in the generation of output audio signal 218.
- lip sensor output signals 276L, 276C, 276R are used to generate a different number of processed lip signal(s) 277.
- Output signal 218 in turn controls the audio output of synthesizer 13 ( Figure 1).
- lip sensor output signals 276L, 276C, 276R may be processed in the analog and/or digital domain by their corresponding signal conditioning electronics 250L, 250C, 250R and/or instrument electronics 216.
- Signal conditioning electronics 250L, 250C, 250R and instrument electronics 216 may have features similar to signal conditioning electronics 50 and instrument electronics 16 described above.
- the processing performed on lip sensor output signals 276L, 276C, 276R by signal conditioning electronics 250L, 250C, 250R and/or instrument electronics 216 may involve any of a variety of analog and digital signal processing techniques similar to those described above for oral cavity sensor output signals 264', 264".
- lip sensor output signals 276L, 276C, 276R may be normalized using a circuit similar to circuit 61 ( Figure 14) as a part of any processing performed by signal conditioning electronics 250L, 250C, 250R and/or instrument electronics 216. For the discussion that follows, it is assumed that lip sensor output signals 276L, 276C, 276R are normalized.
- processed lip signals 277 A, 277B, 277C may be any function & of one or more of lip sensor output signals 276L, 276C, 276R.
- An example of a processed lip signal 277 A, 277B, 277C is a signal representative of overall force applied to inward portion 32A of lip 32. The sensing of lip-force for a mouth- operated input device 12 incorporating a single lip sensor 48 was discussed above ( Figure 5). Increasing the force applied to inward portion 32A of lower lip 32 causes increasing reflection detected by lip sensor 48 and corresponding increases in lip sensor output signal 76.
- increasing the force applied to inward portion 32A of lower lip 32 causes increasing reflection detected by each of lip sensors 248L, 248C, 248R and corresponding increases in lip sensor output signals 276L, 276C, 276R.
- a processed lip signal 277 representative of overall force applied to inward portion 32A of lip 32 may be an additive combination of lip sensor output signals 276L, 276C, 276R.
- the coefficients a, b, c, d may be selected to provide function & with a suitable dynamic range. The coefficients may also be selected to minimize zero offset.
- the signals L, C, R may be filtered, linearized (e.g.
- a processed lip signal 277 is a signal representative of the curvature of inward portion 32A of lip 32.
- the force applied to, and position of, inward portion 32A of lip 32 may be measured by a plurality of sensors 248 which are configured to measure certain parts of the lip and to generate corresponding lip sensor output signals 276 and the lip sensor output signals 276 may be combined to provide a processed lip signal 277 which varies with the curvature of inward portion 32A of lip 32.
- Figures 7A, 7B and 7C are schematic drawings which respectively represent flat lip curvature, negative lip curvature and positive lip curvature.
- the coefficients a, b, c, d may be selected to provide function & " with a suitable dynamic range. The coefficients may also be selected to minimize zero offset.
- the condition (L+R)/2 ⁇ C can be defined to represent the negative curvature of Figure 7B
- the condition (L+R)/2 > C can be defined to represent the positive curvature of Figure 7C.
- the signals L, R, C may be filtered, linearized (e.g. using a look up table), scaled and/or offset prior to using them in a curvature function of this nature.
- the output of the function ⁇ " may be filtered, linearized, scaled and/or offset.
- a processed lip signal 277 is a signal representative of the angle of inward portion 32A of lip 32.
- Figures 8A, 8B and 8C are schematic drawings which respectively represent flat lip angle, negative lip angle and positive lip angle.
- the coefficients a, b, c, d may be selected to provide function & with a suitable dynamic range.
- the coefficients may also be selected to minimize zero offset.
- the condition L > R can be defined to represent the negative angle of inward portion 32A of lip 32 ( Figure 8B)
- the condition L ⁇ R can be defined to represent the positive angle of inward portion 32A of lip 32 ( Figure 8C).
- the signals L and R may be filtered, linearized (e.g. using a look up table), scaled and/or offset prior to using them in a lip angle function of this nature.
- the output of the function & may be filtered, linearized, scaled and/or offset.
- subsets of lip sensor output signals 276L, 276C, 276R could also be used in a similar manner to generate one or more processed lip-position signals 277 A, 277B, 277C representative of localized lip angle.
- Processed lip signals 277 may generally comprise any function & ⁇ of one or more of lip sensor output signals 276L, 276C, 276R.
- Individual processed lip signals 277 A, 277B, 277C may be defined such that they can be substantially independently controlled by a skilled user. For example, where one of processed lip signals 277 A, 277B, 277C relates to the force applied by inward portion 32A of lower lip 32 and another one of processed lip signals 277 A, 277B, 277C relates to the angle of inward portion 32 A of lower lip 32, then a user who becomes adept at using input device 212 may be able to substantially independently control the force that they apply to inward portion 32 A of their lower lip 32 and the angle of inward portion 32A of lower lip 32, so as to use these characteristics to independently control processed lip signals 211 A, 277B, 277C and to thereby independently influence the generation of output signal 218.
- Instrument electronics 216 may use processed lip signals 211 A, 277B, 277C to generate output signal 218 which in turn controls the audio output of synthesizer 13.
- lip sensor output signals 276L, 276C, 276R may be processed to generate a different number of processed lip signals 277, each of which may (alone or in combination with other processed lip signals 277) be related to sensed spatial characteristics of the user's lower lip 32.
- the provision of multiple lip sensors 248L, 248C, 248R allows for a model of the user's lip that has multiple degrees of freedom (i.e. multiple processed lip signals 277 that may relate to a corresponding plurality of physical gestures/characteristics).
- Processed lips signals 277 may also be related to the rate of change of such sensed spatial characteristics of the user's lower lip 32.
- Figure 15 is a partial isometric view of a mouth-operated input device 312 according to another embodiment of the invention.
- mouth-operated input device 312 of Figure 15 is similar to mouth-operated input device 12 ( Figure 1) and mouth-operated input device 212 ( Figure 3).
- Features of device 312 which are similar to features of device 12 and device 212 are referred to using similar reference numerals that are preceded by the digit "3".
- oral cavity sensing system 309 of mouth-operated input device 312 is substantially similar to oral cavity sensing system 209 of mouth-operated input device 212.
- device 312 incorporates a transmissive optical sensor 348 for detecting characteristic(s) of inward portion 32A of a user's lower lip 32.
- Transmissive optical lip sensor 348 is depicted schematically in Figure 16. For clarity, some details of device 312 not related to sensing characteristic(s) of inward portion 32A of a user's lower lip 32 have been removed from Figure 16.
- Lip sensor 348 is an optical transmission-type sensor which emits radiation and senses the intensity of transmitted radiation detected at a remote location.
- Lip sensor 348 comprises a radiation emitter 370A which may be substantially similar to the radiation emitters described above.
- Lip sensor 348 also comprises a radiation detector 370B for sensing transmitted radiation which may be substantially similar to the radiation detectors described above.
- Detector 370B is spaced apart from emitter 370A. As shown in Figure 15, emitter 370A may be located on a first transverse side of region 333 (i.e. between convexity 337 and concavity 339 and between lower portion 336A and upper portion 336B of lower surface 336).
- detector 370B may be located on a transversely opposing second side of region 333 (i.e. so as to detect radiation that is transmitted through region 333 by emitter 370A).
- emitter 370A is located on lobe 34 IA on one transverse side of region 333 and detector 370B is located on lobe 34 IB on the opposing transverse side of region 333, although other configurations are possible.
- emitter 370A comprises, or is optically coupled to, a light pipe 366 A and detector 370B comprises, or is optically coupled to, a light pipe 366B, such that emitter 370A and/or detector 370B may be located remotely (i.e. away from mouth-engaging end 322 of device 312).
- additional optics 368A may be provided for shaping and/or directing the radiation emitted from emitter 370A and/or light pipe 366A and additional optics 368B may be provided for receiving radiation at remotely located detector 370B and/or light pipe 366B.
- Such optics may be similar to the coupling optics described above.
- Lip sensor 348 may optionally be controlled by signal 307 from instrument electronics 316 and/or by signals 315 from signal conditioning electronics 350.
- Instrument electronics 316 and signal conditioning electronics 350 may comprise one or more controllers suitably configured for this purpose.
- Signals 307, 315 may comprise one way or two way control signals.
- Lip sensor 348 detects the force that a user is applying to the inward portion 32 A of lip 32.
- the deformation of inward portion 32A around convexity 337 and into region 333 will be relatively low (see Figure 5A which schematically depicts this situation). Consequently, a relatively large amount of the radiation emitted by emitter 370A will be detected by detector 370B on the transversely opposing side of region 333 and lip sensor output signal 376 will be relatively high.
- inward portion 32A of their lower lip 32 As the user applies an intermediate level of force to the inward portion 32A of their lower lip 32, the deformation of inward portion 32A around convexity 337 and into region 333 will be moderate (see Figure 5B which schematically depicts this situation). Under these conditions, inward portion 32A of lip 32 will intercept some of the radiation from emitter 370A before such radiation crosses region 333 to reach detector 370B. As a result, the amount of radiation detected by detector 370B will be at an intermediate level and lip sensor output signal 376 will be at an intermediate level (i.e. less than the low force situation described above).
- inward portion 32A of lip 32 will intercept a significant amount of the radiation from emitter 370A before such radiation crosses region 333 to reach detector 370B.
- the amount of radiation detected by detector 370B will be at a relatively low level and lip sensor output signal 376 will be at a relatively low level (i.e. less than the low force and intermediate force situations described above).
- lip sensor 348 creates a lip sensor output signal 376 that is inversely representative of the force applied to inward portion 32A of lip 32.
- lobes 341A, 341B, upper portion 336B of lower surface 336 and intermediate surface 340 enclose region 333 and thereby improve the dynamic range of transmissive sensor 348.
- lip sensor output signal 376 may be used by signal conditioning electronics 350 and/or instrument electronics 316 to generate one or more processed lip signals 377. Instrument electronics 316 may then use processed lip signals 377 to generate output signal 318. Output signal 318 may in turn be used to control the audio output of synthesizer 13 ( Figure 1).
- lip sensing system 311 of device 312 may be similar to, or may be varied in manners similar to, the lip sensing systems 11, 211 discussed above.
- the size of region 333 i.e. the distance between upper portion 336B and lower portion 336A
- other parameters of lip sensor 348 may be designed so that lip sensor output signal 376 is a monotonically changing function of the force applied by inward portion 32A of lip 32 to mouth-engagement end 322 of device 312.
- lip sensor output signal 376 may be processed by signal conditioning electronics 350 and/or instrument electronics 316 to output a plurality of processed lip signals 377 and such processed lip signal(s) may generally be any function & of lip sensor signal 376.
- mouth operated input device 312 may make use of a plurality of transmissive sensors for detecting various characteristics related to the inward portion 32 A of a user's lip 32. Such a plurality of transmissive sensors may be used in a manner similar to the three lip sensors 248L, 248C, 248R of device 212.
- Figures 19-23 schematically depict a mouth-operated input device 512 according to another embodiment of the invention. In many respects, mouth-operated input device 512 is similar to mouth-operated input device 12 ( Figure 1) and mouth- operated input device 212 ( Figure 3).
- Mouth-operated input device 512 differs from mouth-operated input devices 12 and 212 in that mouth-operated input device 512 incorporates a non-optical lip-sensing system 513.
- lip- sensing system 513 of mouth-operated input device 512 is based on sensing lip force using one or more force-sensitive resistors (FSRs).
- FSRs force-sensitive resistors
- Mouth-operated input device 512 incorporates a lip-sensing system 513.
- lip-sensing system 513 is configured to sense various characteristics of a user's lower lip. This is not necessary. Mouth-operated input device 512 could be modified to incorporate features similar to those of lip-sensing system 513 for sensing the characteristics of the user's upper lip in addition to or as an alternative to sensing the characteristics of the user's lower lip.
- Lip-sensing system 513 includes a mouthpiece 514 which can be placed at least partially in the user's mouth.
- the user's lower lip may come into contact with lower surface 517 of mouthpiece 514 when the mouthpiece 514 is partially inserted into the user's mouth.
- the user's upper lip and/or upper teeth may rest on upper surface 518 of mouthpiece 514.
- the angle between lower surface 517 and upper surface 518 of mouthpiece 514 may be in a range of 0°-50° . In particular embodiments, this angular range may be 15° -45°.
- Lip-sensing system 513 incorporates a plurality of lip force sensors 548 A, 548B, 548C (collectively, lip force sensors 548) which are configured (as described in more detail below) to detect the localized force applied to various regions of lower surface 517 by various portions of the user's lower lip.
- Each of lip force sensors 548 is capable of generating a corresponding lip force sensor output signal 576A, 576B, 576C (collectively, lip force sensor signals 576) which is representative of the force detected by that particular lip force sensor 548.
- lip force sensors 548 are implemented by force-sensing resistors (FSRs) 549 A, 549B, 549C (collectively, FSRs 549).
- FSRs force-sensing resistors
- lip force sensors 548 may be implemented using the FSRs sold by Interlink Electronics, Inc. of Camarillo, California under part number 400 or similar FSRs sold by Tekscan, Inc. of South Boston, Massachusetts.
- a single FSR 549 is shown in Figure 21.
- FSRs 549 have a resistance which varies with the force/pressure applied to their active regions 526.
- active regions 526 of FSRs 549 are generally circular in cross-section.
- active region 526 is dimensioned to be less than 10 mm across (e.g. in diameter).
- FSRs 549 have tail portions 527 which are mounted on circuit board 532 and which are suitably connected to provide corresponding lip force sensor output signals 576.
- FSRs 549 may be connected as part of a voltage divider circuit.
- the active regions 526A, 526B, 526C of FSRs 549A, 549B, 549C are located at spaced apart locations within mouthpiece 514.
- lip-sensing system comprises three FSRs 549 wherein FSRs 549A, 549C have active regions 526 A, 526C which are spaced apart from one another along a transverse dimension (see arrow 528 of Figure 20) and FSR 549B has an active region 526B which is spaced apart from active regions 526 A, 526C along a longitudinal dimension (see arrow 530 of Figure 20). Longitudinal dimension 530 and transverse dimension 528 may be generally orthogonal to one another. As explained in more detail below, this pattern of active regions 526 of FSRs 549 allows measurement of various characteristics of the user's lower lip.
- FSRs 549 have a resistance that may vary with applied force.
- FSRs 549 typically have a threshold activation pressure (i.e. a threshold activation force per unit area). When FSRs 549 experience forces per unit area that are below their activation pressure, then their resistance does not vary with the applied force (e.g. FSR 549 may represent an open circuit).
- This threshold activation pressure may be in a range of .005-.125 N/mm 2 , for example.
- the forces that can be applied by most users' lips are relatively small and may not be able to overcome the threshold activation pressure of typical FSRs. Additionally, in some applications, such as instrument simulation, it may be desirable to have lip-sensing system 513 be responsive to a range of lip forces which may be below the threshold activation pressure of typical FSRs.
- Mouthpiece 514 may be designed to overcome this characteristic of FSRs 549 by configuring mouthpiece 514 to "pre-load" active regions 526 of FSRs 549 and/or to concentrate applied lip force onto a relatively small area of active regions 526 of FSRs 549.
- Mouthpiece 514 is shown in more detail in Figures 22A, 22B, 23A and 23B.
- mouthpiece 514 comprises an external mouthpiece component 534 ( Figures 22A, 22B) and an internal mouthpiece component 536 ( Figures 23 A, 23B).
- Internal mouthpiece component 536 may receive a portion of circuit board 532 in its cavity 538.
- Internal mouthpiece component 536 may be fabricated from a material that is opaque.
- External mouthpiece component 534 includes lower surface 517 and upper surface 518 described above and receives internal mouthpiece component 536 in its cavity 540 ( Figure 22B).
- external mouthpiece component 534 may be fabricated in whole, or in suitable part(s), from an elastomeric material. This elastomeric material may be used to seal mouthpiece 514 and to transfer lip force to FSRs 549.
- the elastomeric material has a hardness rating in a range of Shore A 5 to Shore A 120.
- external mouthpiece component 534 is relatively more deformable than internal mouthpiece component 536.
- Internal mouthpiece component 536 may support portions of external mouthpiece component 534.
- external mouthpiece component 534 is transparent or translucent (i.e. facilitates transmission of light therethrough). This characteristic of external mouthpiece component 534 is useful for the operation of optical sensors which may be part of mouth-operated input device 512 and which may be used to sense other characteristics of the user's mouth (e.g. the optical sensors used to sense characteristics of the user's oral cavity).
- external mouthpiece component 34 comprises a plurality of protrusions 547 A, 547B, 547C (collectively, protrusions 547) which extend from an interior surface 542 of external mouthpiece component 534 and into cavity 540.
- Protrusions 547 may be located at locations corresponding to active regions 526 of FSRs 549 so as to transfer force from various portions of the users lower lip, through lower surface 517 and to FSRs 549.
- Protrusions 547 may also be shaped and/or sized to pre-load FSRs 549 and/or to concentrate force onto relatively small areas of active regions 526 of FSRs 549.
- protrusions 547 may be sized such that when circuit board 532 is inserted into mouthpiece 514, protrusions 547 interact with (i.e. contact) corresponding active regions 526 of FSRs 549 to create force therebetween. Such forces may cause deformation of protrusions 547 (which may be elastomeric) and/or FSRs 549.
- the forces between protrusions 547 and active regions 526 of FSRs 549 pre-load FSRs 549 with forces that may be at or near the activation threshold pressure of FSRs 549.
- the pre-load force per unit area between protrusions 547 and active regions 526 is in a range of 50% -150% of the threshold activation pressure for FSRs 524. In some embodiments, this range is 80% -120% .
- Protrusions 547 may additionally or alternatively be shaped such that the areas of protrusions 547 that contact corresponding active regions 526 of FSRs 549 represent a relatively small portion of the area of the active regions 526. In this manner, protrusions 547 can concentrate lower lip force from a relatively large area onto a relatively small area of active regions 526, thereby increasing the pressure.
- protrusions 547 are frustro-conical in shape to achieve this effect.
- protrusions 547 may have other shapes where the distal regions of protrusions 547 have smaller cross-sectional areas than the regions of protrusions 547 that are relatively close to interior surface 542.
- the areas of protrusions 547 that contact corresponding active regions 526 of FSRs 549 are in a range of 10% -75% of the area of active regions 526. In some embodiments, this range is 20% -50% . It should be noted that when protrusions 547 are formed from elastomeric material, the area of protrusions 547 in contact with active regions 526 may increase under application of larger amounts of external force. [0128] As discussed above, lip force sensor signals 576 may be processed by suitable signal conditioning electronics 550 and/or instrument electronics 516.
- Signal conditioning electronics 550 and instrument electronics 516 may be similar to (and include components similar to) signal conditioning electronics 50, 250L, 250R, 250C, 350 and instrument electronics 16, 216, 316 of the other mouth-operated input devices 12, 212, 312 described herein.
- signal conditioning electronics 550 process lip force sensor signals 576 to provide processed lip signals 577A, 577B, 577C (collectively processed lip signals 577) similar to processed lip signals 77, 277A, 277B, 277C, 377 of devices 12, 212, 312 and instrument electronics 516 uses processed lip signals 577 to generate output signal 18 similar to output signals 18, 218, 318 of devices 12, 212, 312.
- lip force sensors 548 may be individually controlled (e.g. gated or the like) by corresponding signals from signal conditioning electronics 550 and/or instrument electronics 516. These control signals may be similar to control signals 215L, 215C, 215R and/or control signals 207L, 207C, 207R described above for mouth-operated input device 212.
- lip force sensor signals 576A, 576B, 576C are respectively representative of the force applied by the user's lower lip in the regions of lower surface 517 corresponding to the locations of active regions 526A, 256B, 526C.
- lip force sensor signals 576 (or processed lip signals 577) to be used to estimate a number of different characteristics of the users' lower lip which may in turn be used to control an audio synthesizer or some other electronic device (see synthesizer 13 shown in Figure 1, for example).
- lip sensor output signals 576 are used to generate three processed lip signals 577, each of which may be used independently by instrument electronics 516 to generate output signal 518. In other embodiments, lip sensor output signals 576 are used to generate a different number of processed lip signals 577.
- processed lip signals 577 may be any function ⁇ " of one or more of Hp sensor signals 576A, 576B, 576C.
- An example of a processed lip signal 577 is a signal representative of overall lip force applied to lower surface 517 of mouthpiece 514. Increasing the overall Hp force on lower surface 517 may cause corresponding changes in the resistances of each of FSRs 549A, 549B, 549C and corresponding changes to each of the Hp sensor signals 576.
- a processed Hp signal 577 representative of overall force applied by the user's Hp to lower surface 517 may be an additive combination of Hp sensor signals 576 A, 576B, 576C.
- the coefficients a, b, c, d may be selected to provide function & with a suitable dynamic range. The coefficients may also be selected to minimize zero offset.
- the signals A, B, C may be filtered, linearized (e.g.
- ⁇ " may be filtered, linearized, scaled and/or offset.
- subsets of lip sensor signals 576 A, 576B, 576C could also be used in a similar manner to generate a plurality of processed lip signals 577A, 577B, 577C representative of localized lip-force.
- FIGS 8A, 8B and 8C Another example of a processed lip signal 577 is a signal representative of the angle of the user's lower lip - see Figures 8A, 8B and 8C for schematic drawings which respectively represent flat lip angle, negative lip angle and positive lip angle.
- the coefficients a, b, c, d may be selected to provide function & with a suitable dynamic range. The coefficients may also be selected to minimize zero offset.
- lip sensor signals 576A and 576C are given equal weight and the offset parameter d is zero, then the condition A-C can be defined to represent the flat angle of the user's lower lip (Figure 8A), the condition A > C can be defined to represent the negative angle of the user's lower lip ( Figure 8B) and the condition A ⁇ C can be defined to represent the positive angle of the user's lower lip ( Figure 8C).
- the signals A and C may be filtered, linearized (e.g. using a look up table), scaled and/or offset prior to using them in a lip angle function of this nature.
- the output of the function ST may be filtered, linearized, scaled and/or offset.
- subsets of lip sensor signals 576A, 576B, 576C could also be used in a similar manner to generate one or more processed lip signals 577A, 577B, 577C representative of localized lip angle.
- a processed lip signal 577 is a signal representative of the front to back position (i.e. along longitudinal direction 30) of the user's lower lip.
- the coefficients a, b, c, d may be selected to provide function & with a suitable dynamic range. The coefficients may also be selected to minimize zero offset.
- the signals A, B and C may be filtered, linearized (e.g. using a look up table), scaled and/or offset prior to using them in a lip position function of this nature.
- the output of the function & ⁇ may be filtered, linearized, scaled and/or offset.
- subsets of lip sensor signals 576A, 576B 576C could also be used in a similar manner to generate one or more processed lip signals 577A, 577B, 577C representative of localized lip front to back position.
- the active regions 526 A, 526B, 526C of FSRs 549A, 549B, 549C may be aligned along the same transverse axis (see transverse direction 528 of Figure X).
- another example of a processed lip signal 577 is a signal representative of the transverse curvature of the user's lower lip.
- FIGS 7A, 7B and 7C are schematic drawings which respectively represent flat lip curvature, negative lip curvature and positive lip curvature.
- the coefficients a, b, c, d may be selected to provide function & with a suitable dynamic range. The coefficients may also be selected to minimize zero offset.
- the signals A, B, C may be filtered, linearized (e.g. using a look up table), scaled and/or offset prior to using them in a curvature function of this nature.
- the output of the function & ⁇ may be filtered, linearized, scaled and/or offset.
- Processed lip signals 577 may generally comprise any function & ⁇ of one or more of lip sensor output signals 576A, 576B, 576C.
- Individual processed lip signals 577 A, 577B, 577C may be defined such that they can be substantially independently controlled by a skilled user.
- a user who becomes adept at using mouth-controlled input device 512 may be able to substantially independently control the force that they apply to their lower lip and the angle of their lower lip, so as to use these characteristics to independently control processed lip signals 577 A, 577B, 577C.
- mouth-operated input device 512 may be similar to any of the other mouth-operated input devices described herein.
- mouth-operated input device 512 may comprise oral cavity sensing systems, breath pressure sensing systems and mouth force sensing systems similar to those of the other mouth-operating input devices described herein.
- mouth-operated input device 512 is designed to be used as a part of a wind-instrument simulator, but mouth-operated input device 512 is not limited to use as instrument simulators. Mouth-operated input device 512 may be used for (or may be modified for use for) any of the other applications described herein.
- Mouth-operated input device 512 may be altered or modified.
- Other types of sensors could be used in the place of FSRs 549, such sensors include, by way of non-limiting example, strain gauges, fiber optic flex sensors, optical sensors and/or mechanical compression sensors using Hall effect devices.
- FSRs 549 represent a good choice because of their low cost, simplicity and stability with respect to temperature changes. Some embodiments make use of a custom-made array of FSRs rather than discrete FSR devices 549 described above.
- Mouth-operated input device 512 may be modified to provide a larger number of lip force sensors 548 or a different pattern of lip force sensors 548.
- Mouth-operated input device 12 of Figure 1 also incorporates a sensor 42 for detecting the user's breath-pressure. Breath-pressure-sensor 42 is depicted in more detail in Figure 9. For clarity, details of device 12 which are not related to sensing the user's breath-pressure are not shown in Figure 9. As discussed above, the user's lips 26, 32 may contact mouth-engaging end 22 of device 12, such that the user may blow into mouth-engaging end 22.
- proximal surface 24 of mouth-engaging end 22 has an aperture 80 which leads through conduit 44 to breath-pressure-sensor 42.
- breath-pressure-sensor 42 Various types of breath-pressure-sensors are known in the art. Any suitable breath-pressure-sensor may be used with the device 12 of the illustrated embodiment.
- One example of a suitable breath-pressure-sensor 42 is a variable resistance pressure sensor of the type described in US patent No. 5,543,580 which is incorporated herein by reference.
- Other examples of sensors suitable for use as breath-pressure-sensor 42 include the MPXC2011 from Freescale Semiconductor, Inc. of Austin, Texas and the 26PC05SMT from Honeywell International, Inc. of Morris Township, New Jersey.
- Mouth-operated input device 12 models this aspect of a conventional instrument.
- Breath-pressure-sensor 42 senses the user's breath-pressure and creates a breath-pressure-sensor output signal 82 representative of the user's breath-pressure.
- Breath-pressure-sensor output signal 82 may be used to generate one or more processed breath-pressure signals 84.
- Instrument electronics 16 may make use of processed breath-pressure signal(s) 84 in generating output signal 18.
- Output signal 18 in turn controls the audio output of synthesizer 13 ( Figure 1).
- conduit 44 and breath-pressure- sensor 42 are sealed, such that there is substantially no air flow between the user's oral cavity 52 and sensor 42.
- the total volume of conduit 44 may be less than 500 mm 3 .
- the volume of conduit 44 is less than 200 mm 3 .
- breath-pressure-sensor output signal 82 may be processed by signal conditioning electronics 50 and/or instrument electronics 16 to output processed breath-pressure signal 84 in a manner similar to that of the other sensor output signals discussed above.
- breath-pressure-sensor output signal 82 may be processed by signal conditioning electronics 50 and/or instrument electronics 16 to output a plurality of processed breath-pressure signals 84.
- breath-pressure signals 84 may include one signal related to the measured breath-pressure and another signal related to the rate of change of the measured breath signal.
- Signal conditioning electronics 50 and instrument electronics 16 may function in a manner similar to, and have characteristics similar to, those discussed above with respect to oral cavity sensor output signal 64 and lip sensor output signal 76.
- processed breath-pressure signal 84 may be any function 2F of breath-pressure-sensor signal 82.
- the coefficients a and b may be selected to provide processed breath-pressure signal 84 with a suitably high dynamic range. These coefficients may also be selected to minimize zero offset.
- the offset coefficient b may be selected such that processed breath-pressure signal 84 is a positive signal when the user is blowing into device 12 and a negative signal when the user is sucking on device 12.
- Instrument electronics 16 may make use of processed breath-pressure signal 84 to generate output signal 18 which in turn controls the audio output of synthesizer 13.
- Processed breath-pressure signal 84 is related to the sensed breath-pressure applied by the user. Accordingly, using mouth-operated input device 12, the audio output of synthesizer 13 may be modulated by the user's breath-pressure.
- Mouth-operated input device 212 of the Figure 3 embodiment incorporates a pair of apertures 280', 280" in proximal surface 224 for admitting the user's breath. Apertures 280', 280" are depicted schematically in Figure 10. Aperture 280' leads through a conduit 244 to a breath-pressure-sensor 242.
- Breath-pressure-sensor 242 may be substantially similar to breath-pressure-sensor 42 described above. Breath- pressure-sensor 242 may generate a breath-pressure-sensor signal 282, which may be processed in the analog and/or digital domain by signal conditioning electronics 250 and instrument electronics 216 to generate one or more processed breath-pressure signal(s) 284. The processing of breath-pressure-sensor signal 282 to generate processed breath-pressure signal(s) 284 may be similar to the processing of breath- pressure signal 82 described above. [0152] Aperture 280" leads through conduit 286 to a flow regulator 288. Flow regulator 288 may be used to give instrument 10 a realistic feel by allowing the user's breath to flow through the instrument. In some embodiments, flow regulator 288 is user adjustable.
- flow regulator 288 comprises a screw (not shown) which may be adjusted to various positions which restrict the flow of air through conduit 286.
- the screw may be adjusted inwardly to a position where it occupies a large volume of space in conduit 286 and substantially restricts the flow through conduit 286 and the screw may be withdrawn to a position where it occupies only a small volume (or none) of conduit 286, such that the user's breath can flow through conduit 286.
- Conduit 286 may exhaust from device 212 at an exhaust port 290 external to the user's mouth.
- Exhaust port 290 (which is not shown in Figure 3) may be located on the side of device 212.
- Mouth-operated input device 312 (Figure 15) also comprises a pair of apertures 380', 380" which may be substantially similar to apertures 280', 280" of device 212.
- the breath pressure sensor and flow control regulator (not explicitly shown) of device 312 may be substantially similar to those of device 212.
- Mouth-operated input device 212 of the Figure 3 embodiment also comprises an optional fin 251 which projects downwardly from upper portion 236B of lower surface 236 between proximal surface 224 and intermediate surface 240.
- Fin 251 may serve to reduce the amount of radiation emitted from lip sensors 248L, 248C, 248R and reflected back to lip sensors 248L, 248C, 248R from the user's oral cavity 52 (i.e. as opposed to lip 32). Under low lip-force conditions, without fin 251, radiation emitted from lip sensors 248L, 248C, 248R may be reflected back from tongue 52 or other surfaces of oral cavity 52 and provide spurious results for lip sensors 248L, 248C, 248R.
- fin 251 may extend downwardly by an amount substantially similar to the distance between upper portion 236 A and lower portion 236B of lower surface 236.
- fin 251 is in the output path of radiation emitted from lip sensors 248L, 248C, 248R.
- fin 251 is shaped, such that most of the radiation incident on fin 251 from lip sensors 248L, 248C, 248R is reflected away from (and not spuriously detected by) lip sensors 248L, 248C, 248R.
- fin 251 is shaped such that its surface 257 facing intermediate surface 240 is skewed by greater than 30° with respect to intermediate surface 240 (or forms an angle greater than 120° with respect to the longitudinal axis 221 of input device 212). In other embodiments, fin 251 is shaped such that surface 257 forms an angle greater than 120° with the plane of incidence of radiation from lip sensors 248L, 248C, 248R. In some embodiments, fin 251 comprises (or is coated, at least on surface 257, with) a material that has a low reflectivity at the wavelength of the radiation used by lip sensors 248L, 248C, 248R.
- surface 257 of fin 251 has a reflectivity of less than 30% at the wavelength used by lip sensors 248L, 248C, 248R.
- Fin 251 may be designed such that at any lip-force, the radiation emitted from Hp sensors 248L, 248C, 248R and reflected from fin 251 is significantly less than the radiation reflected from lip 32.
- this reflectivity difference may be effected by using a sensor with a suitable wavelength, by varying the shape or orientation of fin 251 and/or by varying the material on surface 257 of fin 251.
- Mouth-operated input devices 12, 212, 312, 512 may also incorporate one or more outputs 92 (Figure 1).
- An output 92 may comprise any suitable output device for providing information to a user, such as, without limitation, an LED, an alphanumeric display, a graphical display, an audio output device and a tactile output device.
- Outputs 92 may indicate when a particular tone is flat (too low), sharp (too high) or on pitch.
- the determination of whether a note is flat, sharp or on pitch may be made by instrument electronics 16, 216, 316, 516 which may compare the output of synthesizer 13 (as measured by an acoustic sensor (not shown)) with a predetermined value that is defined as being "on-pitch".
- the predetermined on-pitch value may be determined during calibration of device 12, 212, 312, 512 and/or the acoustic sensor, for example.
- Outputs 92 may also be used to indicate the direction to a nearest harmonic. [0158]
- Outputs 92 may also be used to train musicians.
- Mouth-operated input devices 12, 212, 312, 512 may also comprise one or more hand-operated inputs 14.
- hand-operated inputs 14 comprise buttons, but hand-operated inputs 14 may generally comprise any suitable types of input mechanisms, such as multi-position switches, alphanumeric keys, sliders, knobs, joysticks or the like.
- Hand-operated inputs 14 may be used to provide ON/OFF functions for device 12, 212, 312, 512 or for various applications, such as pitch- sensing and the like. Hand-operated inputs 14 may also be used for note selection or to otherwise alter the tone of signal 18.
- Figure 11 depicts a schematic block diagram of an instrument 10 according to a particular embodiment of the invention. Instrument 10 incorporates a mouth- operated input device 12, which may generally comprise any of the mouth-operated input devices described herein (e.g. devices 12, 212, 312, 512). In the Figure 11 embodiment mouth-operated input device 12 incorporates a number of sensors, including breath-pressure-sensor 42, oral cavity sensor 46 and lip sensor 48, together with their associated signal conditioning electronics 50.
- instrument electronics 16 comprise an embedded processor 29A.
- Processor 29A may be a programmable microprocessor which has access to a memory 29B for storage of program instructions and other useful data.
- Instrument electronics 16 also receive input signals from inputs 14. Inputs 14 originate from mouth-operated input device 12 and/or instrument 10. In addition, as shown in Figure 11, instrument 10 may receive inputs 14 from external sources. In the Figure 11 embodiment, instrument 10 comprises a single output 92, but, in general, instrument 10 may comprise a plurality of outputs 92. As with inputs 14, outputs(s) 92 may be provided as a part of mouth-operated input device 12, instrument 10 and/or external output device(s).
- Instrument electronics 16 output an output signal 18 for controlling synthesizer 13. As discussed above, output signal 18 may conform to a known audio protocol. In the Figure 11 embodiment, instrument electronics 16 also receive control signal(s) 69 from synthesizer 13. Control signal(s) 69 may generally be used for any purpose. By way of non-limiting example, control signal(s) 69 could be used to configure instrument electronics 16. The configuration parameters of instrument electronics 16 may be different when instrument 10 is used to simulate a brass instrument as compared to when instrument 10 is used to simulate a woodwind instrument, for example. Those skilled in the art will appreciate that there are a wide variety of other uses for control signal(s) 69, such as tonal adjustment, sensor sampling rate adjustment and the like.
- FIG. 12 is a schematic block diagram of an instrument 10' according to another embodiment of the invention.
- Instrument 10' incorporates a mouth-operated input device 12, which may generally comprise any of the mouth-operated input devices described herein (e.g. devices 12, 212, 312, 512).
- a portion 16A of instrument electronics 16 is embedded in instrument 10' and another portion 16B of instrument electronics 16 is hosted on a computer 27.
- portion 16A of instrument electronics 16 is embedded in mouth-operated input device 12.
- a wired or wireless communication link 67 is provided between portions 16A, 16B of instrument electronics 16. Communication link 67 may be implemented using any of a variety of suitable protocols known to those skilled in the art.
- communication link 67 may be implemented in accordance with the MIDI, USB or OSC protocols.
- computer 27 which comprises portion 16B of instrument electronics 16, a microprocessor 29 A and memory 29B) controls output device 92 and communicates with synthesizer 13 via output signal 18 and control signal 69.
- portion 16A of instrument electronics 16 may also comprise a suitable processor and/or suitable memory.
- output 92 is an external output device which is controlled by portion 16B of instrument electronics 16.
- instrument 10' may generally comprise any suitable number of outputs 92 which may be provided as a part of mouth-operated input device 12, instrument 10', computer 27 and/or external output device(s). Such output(s) may additionally or alternatively be controlled by portion 16A of instrument electronics 16.
- Figure 12 instrument 10' is similar to the Figure 11 instrument 10.
- Mouth-operated input devices 12, 212, 312, 512 are described above as having application to an instrument, where devices 12, 212, 312, 512 sense characteristics of the user's mouth (e.g. characteristics associated with oral cavity, tongue position, tongue shape, lip-position and breath-pressure) and generate corresponding sensor output signals which are processed and then ultimately used to control a synthesizer 13. Mouth-operated input devices 12, 212, 312, 512 described herein may be used for other applications, such as to enable disabled individuals to control various systems with their mouths or to enable people whose hands are otherwise encumbered (e.g. people in protective suits, such as space suits or underwater suits) to control various systems.
- characteristics of the user's mouth e.g. characteristics associated with oral cavity, tongue position, tongue shape, lip-position and breath-pressure
- Mouth-operated input devices 12, 212, 312, 512 described herein may be used for other applications, such as to enable disabled individuals to control various systems with their mouths or to enable people whose hands are otherwise encumb
- Different characteristics of the way in which a user interacts with the mouth-operated input device may be used to control different system parameters. More particularly, the output signals of mouth-operated input devices 12, 212, 312, 512 may be processed to generate one or more output signals 18 which ultimately control different system parameters.
- a battery-operated wheel chair may be controlled using lip-force and tongue position. If the lip-force is greater, the mouth-operated input device may output a first output signal 18 which causes the chair's motor to move faster and if the lip- force is lower, the mouth-operated input device may change the level of the first output signal 18 such that the chair's motor may move more slowly.
- the mouth-operated input device may output a second output signal 18 which directs the chair's motor to turn the chair in a leftward direction and if the tongue position is more rightward, then the second output signal 18 may directed the chair's motor to turn the chair in a rightward direction.
- the position of the forward-facing surface a user's tongue could be used to control the X-Y position of a mouse pointer or a joystick in a computer user interface and a puff of breath or a squeeze of a lip may represent a click of the mouse/joystick button.
- These signals may also be used to control the velocity (rather than the position) of the mouse pointer, as in an isometric mouse.
- the tilt angle and force of a user's lip may control the X-Y position of a mouse pointer or a joystick in a computer user interface and a puff of breath or a squeeze of a lip may represent a click of the mouse/joystick button.
- • the angle of the user's lip and the force applied to the lip may be used as X-Y control to highlight a particular object within a two dimensional array of objects shown on a computer screen.
- a display may show an alphanumeric keyboard for example.
- a puff of breath could select the currently highlighted object on the display.
- a puff of breath could select a letter on a displayed alphanumeric keyboard.
- the mouth-operated input devices of the invention include speech synthesis or silent communication. Many of the sounds made by human vocal cords depend on motion of the tongue, jaw and lips as well as breath- pressure.
- the mouth-operated input devices of the invention may be configured to output control signals which relate to the gestures associated with speech and which could be provided to a speech synthesizer which emulates the sounds of speech based on the control signals.
- the control signals relating to the gestures associated with speech could be encoded by the device and later decoded as sounds. This would enable a person to "talk" without making sound.
- the mouth-operated input devices described herein may be provided with an infrared emitter corresponding to the IRDA standard which is commonly used to control consumer devices, such as remote controls and PDA devices.
- a mapping may be provided such that combinations of the types of gestures described herein can control consumer devices such as entertainment systems.
- Another application of the mouth-operated input devices described herein is to select between a one or two dimensional array of locations which may be predefined on the roof of the user's mouth. The user may select any of these locations with the tip of their tongue. Such locations may be discriminated using oral cavity /tongue sensors of the general type described above together with the breath pressure sensor.
- a user may emit a puff of breath similar to a "t" sound with their tongue starting at any of these locations.
- the position of the user's tongue may be detected by the oral cavity/tongue sensors and the puff of breath may be detected by the breath pressure sensor.
- This puff may cause the signal processing system to emit a signal related to a virtual button defined for that starting location.
- the tongue may be positioned in one of these locations and a puff of air similar to the "c" in “cat” or the "h” in “hat” may be emitted, causing the signal processing system to emit a signal that is associated with the current location of the tongue.
- the mouth-operated input device may be configured to distinguish between a "t", a "c” or an "h”.
- An output 92 may be used to display (to the user) the action or symbol that will result if the "t", “c” or “h” breath puff is emitted with the tongue at the current location.
- This capability may be used for example for “typing” (e.g. selecting from among virtual keys which may be alphanumeric), for playing computer games, or for pressing the keys of a virtual musical instrument while controlling the volume with the breath and ranges of pitch and/or timbre with the lips.
- the detection of such "t”, “c” and/or “h” sounds may be performed using any of the mouth-operated input devices described herein to provide desired functionality.
- FIG 17A is an exploded isometric view of a mouth-operated input device 410 according to yet another embodiment of the invention.
- Mouth-operated input device 410 functions as a joystick type input device for controlling some external system (not shown) which may be an instrument, but which may alternatively be any other suitably configured system, such as a personal computer, a video game console, a wheelchair control system or the like.
- mouth-operated joystick device 410 incorporates a mouth-operated input device 312 that is substantially similar to mouth-operated input device 312 of Figure 15.
- Mouth- operated joystick device 410 also comprises a force/joystick sensor system 412 capable of detecting the direction and/or the amount of force applied to mouth- operated input device 312 by a user's mouth (e.g. by the user's teeth, lips, tongue) or, if desired, by some other part of the user's body.
- a force/joystick sensor system 412 capable of detecting the direction and/or the amount of force applied to mouth- operated input device 312 by a user's mouth (e.g. by the user's teeth, lips, tongue) or, if desired, by some other part of the user's body.
- Force sensing system 412 which is shown in more detail in Figure 17B, includes: an upper housing element 422 and a lower housing element 417 which are moveable relative one another and a substrate 414 which houses a plurality (e.g. four) optical sensors 416A, 416B, 416C, 416D (collectively, sensors 416).
- sensors 416 are reflective optical sensors which are configured to emit radiation in the general direction of arrow 418.
- Sensors 416 may be substantially similar to reflective optical sensors 46, 48 discussed above. In other embodiments, sensors 416 could be implemented as transmissive type optical sensors.
- Force sensing system 412 also includes a reflective element 420 which, in the illustrated embodiment, is coupled to upper housing element 422 by fastener components 423A, 423B in such a manner that reflective element 420 is deformable or otherwise moveable relative to substrate 414.
- fastener component 423B projects loosely through aperture 425 in substrate 414 and is snugly received in a correspondingly shaped aperture 427 in reflective element 420 such that movement of upper housing element 420 causes corresponding movement of reflective element 420 relative to substrate 414.
- the range of movement of reflective element 420 relative to substrate 414 is less than about lmm.
- element 420 is reflective. Suitable materials for reflective element 420 include reflective spring steel, for example.
- element 420 comprises an opaque element 420 and element 420 may be fabricated from an opaque material.
- reflective/opaque element 420 is formed from a first material which is coated (at least partially) with a second material having the desired optical characteristics.
- Reflective element 420 is shown in detail in Figure 17B.
- reflective element 420 comprises a plurality (e.g. four) of tabs 426 A, 426B, 426C, 426D (collectively, tabs 426) which extend generally radially from a central region 428. When device 410 is in an ambient state, tabs 426 are generally located below corresponding sensors 416 at generally equal distances from sensors 416.
- Upper housing element 422 may be physically coupled to one or more sidewalls 430 of device 410 or may otherwise be physically coupled to device 410, such that force applied by the user to mouth-operated input device 312 results in corresponding force applied via fastener component 423B and to reflective element 420.
- Such force can move reflective element 420 relative to substrate 414 and can move one or more of tabs 426 toward or away from their corresponding sensors 416, resulting in changes to the amount of radiation detected by individual sensors 416.
- tabs 426B, 426C may move closer to their corresponding sensors 416B, 416C and tabs 426 A, 426D may move further from their corresponding sensors 416A, 416D.
- Such movement may result in sensors 416B, 416C detecting an increased amount of radiation and sensors 416A, 416D detecting a decreased amount of radiation.
- this same effect can be achieved by physically coupling substrate 414 to upper housing element 422 such that force applied by a user to mouth-operated input device 312 results in movement of substrate 414 relative to reflective element 420.
- FIG. 18 represents a schematic depiction of device 410 according to a particular embodiment of the invention.
- device 410 comprises a mouth-operated input device 312 which incorporates lip sensor 348, a pair of oral cavity sensors 346' , 346" and breath sensor 380' . These sensors produce lip sensor output signal 376, oral cavity sensor output signals 364', 364" and breath sensor signal 382, which are provided to device electronics 416.
- Mouth-operated input device 312 may also comprise one or more optional inputs 314 which provide their own signals to device electronics 416.
- Device 410 also comprises a plurality of joystick sensors 416 which provide a corresponding plurality of joystick signals 452 to device electronics 416.
- signal conditioning electronics may also be provided for joystick sensors 416.
- device 410 comprises four joystick sensors 416 (see Figure 17B) which provide four corresponding joystick signals 452 to device electronics 416.
- One or more optional additional inputs 414 may provide their own signals to device electronics 416. Such optional additional inputs may be located on device 410 or may be external to device 410.
- device electronics 416 comprises a microprocessor 429A and accessible memory 429B.
- Device electronics 416 processes the signals received from the various sensors and inputs and generates a plurality of corresponding output signals 418 which are provided to I/O port 444.
- I/O port 444 may generally be connected to any system (not shown) so that device 410 can be used to provide input to the system and thereby allow the user to control the system using their mouth.
- Output signals 418 may comprise one or more signals corresponding to each of lip sensors 348, oral cavity sensors 346', 346", breath sensors 380', joystick sensors 416, optional inputs 314, 414 and/or various combinations of these sensors and/or inputs. This can provide a wide range of control options.
- Device 410 may also be capable of receiving one or more signals 469 from the external system via port 444. Such signals 469 can be provided to device electronics 416 and can be used by device electronics 416 to control device 410 and/or to help process the various other signals received at device electronics 416. Device 410 may also comprise one or more of its own optional outputs 492 which may be similar to outputs 92 described above.
- instrument electronics 416 need not comprise a microprocessor 429A and/or memory 429B. In a manner similar to the difference between the embodiments of Figures 11 and 12, such components may be provided in the external system.
- Device 410 can function as a two axis joystick which is controllable by the user's mouth. Force applied by the user's mouth to mouth-operated input device 312 along a first axis could cause variation in the positions of opposing tabs 426B, 426D and corresponding variation in the sensor signals 452 from sensors 416B, 416D. Similarly, force applied by the user's mouth along an orthogonal axis could cause variation in the positions of opposing tabs 426A, 426C and corresponding variation in the sensor signals 452 from sensors 416A, 416C. Forces in other directions will result in various linear combinations of the movements along one of these orthogonal axes.
- Such forces may be applied to mouth-operated input device using the user's lips and/or the user's teeth, for example.
- a user can control the joystick action independently of any of the other sensors (i.e. the breath sensor(s), oral cavity sensor (s) and/or lip sensor (s).
- a user can control the joystick action with their teeth while independently interacting with the other sensors.
- a user can independently control this joystick action with an outer portion 32B of their lower lip 32 while using the inward portion 32A of their lower lip 32 to interact with lip sensor 348. This independent control may be facilitated in part by convexity 337 (see Figure 15).
- Device 410 may be used to provide a wide variety of functionalities and to provide input to a wide variety of external systems. By way of non-limiting example:
- Device 410 is usable as a joystick which provides input to an external system (e.g. a computer system, a video gaming system, a wheel chair operating system or the like).
- an external system e.g. a computer system, a video gaming system, a wheel chair operating system or the like.
- the relative position of reflective element 420 maps to a joystick direction and velocity and various tongue, lip, oral cavity or breach actions can be used to provide other joystick functionalities (e.g. buttons clicks).
- Device 410 is usable as a pointing device (i.e. in a manner similar to a conventional computer mouse) for an external system such as a computer or the like.
- the relative position of reflective element i.e. in a manner similar to a conventional computer mouse
- pointing device 420 (as detected by sensors 416) can be used to map a position of the pointing device and various tongue, lip, oral cavity or breach actions can be used to provide other pointing device functionalities (e.g. left click, right click, scroll wheel etc.).
- a certain force range on the inward portion 32A of lip 32 may be used to activate/deactivate the pointing device in a manner which would provide a functionality similar to picking up and repositioning a mouse.
- these force ranges may be subdivided into a multiple levels to provide a variety of functionalities.
- a low level of lip force may correspond to turning "OFF" the operation of the pointing device
- a moderate level of lip force may correspond to normal pointing device activity
- a high level of lip force may correspond to "dragging" a virtual object with the pointing device (i.e. similar to "clicking and dragging" with a conventional mouse).
- Device 410 could be used as a stylus type input device for an external system such as a handheld computer or the like.
- the relative position of reflective element 420 (as detected by sensors 416) can be used to map a position of the stylus and various tongue, lip, oral cavity or breach actions can be used to provide other functionalities (e.g. activating/deactivating the stylus etc.).
- Device 410 can be used as a simulator for a musical instrument such as a trombone, harmonica or Theremin, where the relative position of reflective element 420 (as detected by sensors 416) can be used to map a pitch (i.e. like the stepped side to side pitch control of a harmonica or the continuous front to back pitch control of a trombone).
- breath pressure may be used to map volume and the lip, tongue and/or oral cavity sensors can be used to control timbre or other characteristics.
- each optical sensor comprises a radiation emitter and a radiation sensor for detecting reflected radiation.
- Radiation from a single emitter may be detected by multiple radiation detectors and radiation from multiple emitters may be detected by a single radiation detector.
- lip sensors and oral cavity sensors may be pulsed at different times from one another or individual lip sensors may be pulsed at different times than one another.
- mouth-operated input device 312 of Figure 15 makes use of transmissive-type radiation sensors to implement lip sensor 348 (i.e. to detect one or more characteristics of inward portion 32A of a user's lip 32).
- Transmissive-type optical sensors may have other configurations.
- mouth-operated input device 212 may be modified to provide one or more transmissive-type optical sensors for detecting the force applied to the inward portion 32A of lip 32.
- Radiation emitters may be located inside oral cavity 52 (e.g. in fin 251) and may be directed back toward radiation detectors, which may be at or near intermediate surface 240, for example.
- Such radiation emitters and detectors could then be used to generate lip sensor output signals 276 which vary with the force applied to inward portion 32A of lip 32. For example, increased lip- force may cause inward portion 32A of lip 32 to deform around convexity 237 and into region 233, thus correspondingly reducing the amount of radiation detected by the radiation detectors.
- Similar transmissive lip sensors may be designed such that both their radiation emitters and radiation detectors are located at or near intermediate surface 40 and fin 251 is configured to reflect emitted radiation back toward the radiation detectors or such that both their radiation emitters and radiation detectors are located in fin 251 and intermediate surface 40 and is configured to reflect emitted radiation back toward the radiation detectors •
- the instrument electronics and/or signal conditioning electronics described above may comprise one or more digital processor(s) capable of performing the operations discussed above.
- processor(s) may comprise, without limitation, a microprocessor, a programmable logic array, a computer-on-a- chip, the CPU of a computer or any other suitably programmable controller.
- the lip sensors described above generate lip sensor output signals (and, ultimately, processed lip signal(s)) which represent the force applied by the user on inward portion 32 A of lower lip 32. These signals may also represent the position of inward portion 32A of lower lip 32 relative to an aspect of the mouth-operated input device (e.g. relative to the convexity of the mouth- operated input device and/or relative to the upper portion of the lower surface of the mouth-operated input device).
- the mouth-operated input devices described above have lower surfaces (e.g. lower surface 36) having lip-receiving convexities (e.g. convexity 37) provided by the stepped profile of spaced apart lower and upper portions (e.g. lower portion 36A and upper portion 36B).
- lip receiving convexities provide distinct advantages when measuring characteristics of inward portion 32A of lower lip 32 as discussed above.
- the provision of a lip sensing system on one side of a mouth-operated input device i.e. on the lower surface of the device allows for the independent measurement of the characteristics of a single lip.
- Similar lip receiving convexities may additionally or alternatively be provided on the upper surfaces of the mouth-operated input devices described above, for providing similar advantages relating to the measurement of characteristics of upper lip 26.
- Similar lip sensors may also be provided on one or both transverse sides of the mouth-operated input devices described above, for detecting the position (or other characteristics) of the user's lips relative to a transverse side of the device.
- FIG. 13A-13C schematically depict mouth- operated input devices 12 according to a number of other embodiments having various lip-receiving profiles.
- device 12 comprises at least one lip-receiving convexity 37 and a region of space 33 located adjacent convexity 37 such that a portion of lip 32 can deform around convexity 37 and into region 33.
- Region 33 may be located between first and second spaced apart surfaces 36A, 36B.
- One or both of surfaces 36A, 36B may comprise lip-receiving convexity 37.
- At least one optical sensor 48 is configured to emit radiation into region 33 and to detect radiation either transmitted through region 33 and/or reflected from lip 32 when lip 32 is located in region 33.
- Optical sensor 48 is capable of detecting a signal related to how much of lip 32 is located in region 33 which in turn is related to the force applied to the user's lip 32.
- a portion of lip 32 deforms around the lip-receiving convexity and further into region 33. Variation of the deformation of lip 32 into region 33 causes corresponding variation in the amount of transmitted and/or reflected radiation and the output level of sensor 48.
- the outputs of all of the sensors are used by the instrument electronics to generate a single output signal. This is not necessary.
- the instrument electronics may make use of one or more sensors of the types described above to generate a plurality of independent output signals.
- joystick/force sensing system 412 of device 410 represents one particular embodiment of a force sensing system.
- other types joystick/force sensing systems could be used in the place of system 412.
- Mouth-operating input devices incorporating such alternative joystick/force sensing systems could still take advantage of the functionalities afforded by the lip pressure sensing systems and oral cavity sensing systems described herein.
- the joystick/force sensing system could provide position mapping and various tongue, lip, oral cavity or breach actions could be used in to provide other pointing device functionalities (e.g. left click, right click, scroll wheel etc.).
- Device 410 described above incorporates mouth-operated input device 312.
- device 410 may comprise any of the other mouth- operated input devices described herein.
- optical lip force sensors e.g. of devices 12, 212
- resistive lip force sensors e.g. of device 512
- the optical lip force sensors may be combined in a device that incorporates multiple forms of lip force sensors.
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Physics & Mathematics (AREA)
- Biomedical Technology (AREA)
- Human Computer Interaction (AREA)
- Heart & Thoracic Surgery (AREA)
- Vascular Medicine (AREA)
- Life Sciences & Earth Sciences (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Position Input By Displaying (AREA)
Abstract
Mouth-operated input devices are provided which comprise various combinations of sensors for detecting various characteristics of a user's mouth. The characteristics of the user's mouth may then be used to control an external system. Various types of lip force sensors can be used to detect various characteristics of a user's lip(s); optical oral cavity sensors can be used to detect various characteristics of a user's tongue and oral cavity; breath pressure sensors can be used to detect breath pressure; and optical joystick sensors can be used to detect movement of the device.
Description
MOUTH-OPERATED INPUT DEVICE
Related Applications
[0001] This application claims the benefit of the priority of US application No. 60/940073 filed 24 May 2007 which is hereby incorporated herein by reference.
Technical Field
[0002] This application relates to mouth-operated input devices which may be used as input and/or control devices for a wide variety of systems. Particular embodiments of the invention provide mouth-operated input devices for electronic instrument simulators and mouth-operated input devices for operation and/or control of various components and/or systems.
Background [0003] There are a wide variety of components and systems which can make use of mouth-operated input devices. Non-limiting examples of such systems include: electronic wind instrument simulators; systems for the assistance of disabled individuals who may be without the use of their hands or who may not have hands; systems which allow people to interact with various devices when their hands are otherwise encumbered or occupied (i.e. "hands free" operation); and systems which allow mouth-operated input in addition to more traditional input means to interact with various devices.
[0004] Electronic instrument simulators, such as keyboard-activated synthesizers which use hand-operated inputs to emulate a conventional piano and other instruments, are relatively popular. Electronic instrument simulators and apparatus for electronically controlling conventional instruments have been disclosed in US patent Nos. 4,085,646; 5,543,580; 5,459,280; 5,149,904; and 7,049,503, for example. However, electronic wind instrument simulators have not achieved the same level of popularity as their conventional hand-operated counterparts. One reason for the lack of popularity of electronic wind instrument simulators is the limited ability of prior art simulators to emulate the response of the instrument to the musician's "embouchure". Embouchure may generally be described as the way that the musician's mouth (including, without limitation, lips, tongue, cheeks, throat, jaw and teeth) interact with the instrument mouthpiece. For many conventional wind instruments, the sound of the instrument is heavily influenced by embouchure.
[0005] Embouchure may have many different aspects which may related to one or more physical gestures. Non-limiting examples of various aspects of embouchure include the force with which the lips are pressed on the mouthpiece, the position of
the lips and tongue relative to the mouthpiece, the tilting and/or curvature of the lips and tongue and the spatial characteristics of the oral cavity. Through practice, musicians often learn to control various aspects of their embouchure subconsciously. In conventional instruments, embouchure typically influences the characteristics of the resonant system formed by the musician's mouth, the instrument mouthpiece and the instrument body. It has proven to be difficult to sense and/or simulate embouchure. Prior art wind instrument simulators have not been able to provide the sensitivity, dynamic range or multiple degrees of freedom necessary to simulate embouchure in a manner that is satisfactory to musicians or in a manner that allows the wind instrument simulators to faithfully reproduce the variety sounds generated by their conventional counterparts.
[0006] There is a general desire for mouth-operated input devices for electronic wind instrument simulators which are capable of sensing and/or simulating various aspects of a musicians 's embouchure and which are capable of overcoming (or at least ameliorating) some of the deficiencies of the prior art.
[0007] Mouth-operated input devices may have a wide variety of applications other than for the control of instrument simulators. Mouth-operated input devices may enable hands free operation of various components and/ or systems. Mouth-operated input devices may also be used as additional input and/or control devices for systems incorporating more conventional (e.g. hand-operated) input devices. By way of non-limiting example, mouth-operated input devices may be used by disabled individuals with limited capability of using conventional hand-operated input devices and by underwater divers or astronauts whose protective suits limit their ability to use their hands to operate input devices. Such mouth-operated input devices can be used as input and/or control devices for a wide variety of systems. Different characteristics of the way in which a user interacts with the mouth-operated input device may be used to control different system parameters. Such user-interaction characteristics may include, for example, the force with which the lips and/or tongue are pressed on the input device, the position of the lips and tongue relative to the input device, the tilting and/or curvature of the lips and tongue and the spatial characteristics of the oral cavity.
[0008] Examples of prior art mouth-operated input devices include: • the tongue-controlled microjoystick disclosed at http : //www . asel . udel . edu/robotics/chameleon/chameleon . html ; • the dual-joystick tetramouse™ device disclosed at http://tetramouse.com;
• the electricjoy™ joystick device disclosed at http : //www . genesisone . net/electricjoy . htm;
• the joystick disclosed at http:www.jouse.com;
• the joystick disclosed at http:www.lifetool.at; • the pointing device disclosed in US patent No. 6,801,231;
• the tongue-based input device disclosed in US patent No. 5,460, 186; and
• the user interface controller disclosed in US patent No. 5,365,026.
[0009] There is a general desire to provide mouth-operated input devices for various systems, wherein the mouth-operated input devices are capable of sensing a variety of user-interaction characteristics .
Brief Description of the Drawings
[0010] In drawings which depict non-limiting embodiments of the invention: Figure 1 is schematic diagram of a wind instrument simulator incorporating a mouth-operated input device according to a particular embodiment of the invention;
Figures 2A, 2B, 2C, 2D, 2E and 2F (collectively, Figure 2) schematically illustrate an oral cavity sensing system suitable for use with the Figure 1 mouth-operated input device; Figures 3 A and 3B (collectively, Figure 3) are different isometric views of a mouth-operated input device according to another embodiment of the invention;
Figure 4 is a schematic block diagram of the oral cavity sensing system of the Figure 3 mouth-operated input device; Figures 5A, 5B and 5C (collectively, Figure 5) schematically illustrate a lip sensing system according to a particular embodiment of the invention, which is suitable for use with the Figure 1 mouth-operated input device;
Figure 6 is a schematic block diagram of a lip sensing system of the Figure 3 mouth-operated input device;
Figure 7A, 7B, 7C (collectively, Figure 7) schematically depict the detection of lip curvature by the lip sensing system of Figure 6;
Figure 8A, 8B, 8C (collectively, Figure 8) schematically depict the detection of lip angle by the lip sensing system of Figure 6; Figure 9 is a schematic diagram depicting the breath-pressure-sensor of the Figure 1 mouth-operated input device;
- A -
Figure 10 is a schematic diagram depicting the breath-pressure sensing system of the Figure 3 mouth-operated input device;
Figure 11 is a schematic block diagram of an instrument according to a stand alone embodiment of the invention; Figure 12 is a schematic block diagram of an instrument according to a computer-hosted embodiment of the invention;
Figures 13A, 13B and 13C (collectively, Figure 13) schematically illustrate examples of lip sensing systems for mouth-operated input devices according to other embodiments of the invention; Figure 14 is a schematic depiction of a normalization circuit that may be used to normalize oral cavity sensor output signal(s) in accordance with a particular embodiment of the invention;
Figure 15 is a partial isometric view of a mouth-operated input device according to another embodiment of the invention; Figure 16 is a schematic block diagram of the lip sensing system of the Figure 15 mouth-operated input device;
Figure 17A is an exploded isometric view of a mouth-operated joystick device according to another embodiment of the invention;
Figure 17B is an exploded isometric view of the force/joystick sensor system of the Figure 17A device;
Figure 18 is a schematic depiction of the Figure 17 A device;
Figure 19 is schematic isometric view of a mouth-operated input device incorporating a system for enabling multi-dimensional control with a user lip according to another embodiment of the invention; Figure 20 is a schematic isometric view of the Figure 19 mouth- operated input device with the external mouthpiece component removed to show additional detail;
Figure 21 is a schematic isometric view of a force-sensitive-resistor (FSR) used in the Figure 19 mouth-operated input device; Figures 22A and 22B (collectively, Figure 22) are schematic isometric views of the external mouthpiece component of the Figure 19 mouth-operated input device; and
Figures 23 A and 23B (collectively, Figure 23) are schematic isometric views of the internal mouthpiece component of the Figure 19 mouth-operated input device.
Detailed Description
[0011] Throughout the following description, specific details are set forth in order to provide a more thorough understanding of the invention. However, the invention may be practiced without these particulars. In other instances, well known elements have not been shown or described in detail to avoid unnecessarily obscuring the invention. Accordingly, the specification and drawings are to be regarded in an illustrative, rather than a restrictive, sense.
[0012] Mouth-operated input devices are provided for interaction with various systems. In particular embodiments of the invention, mouth-operated input devices are used to control wind instrument simulators. In other embodiments, mouth- operated input devices according to the invention sense various characteristics of the user's mouth and output corresponding sensor output signals which may be used as inputs for a variety of different systems. [0013] In a particular embodiment of the invention, a mouth-operated input device comprises a convexity around which a user's lip may deform and an optical sensor for emitting radiation into a region of space on a first side of the convexity. This region of space adjacent the first side of the convexity may comprise a concavity. The optical sensor may be configured to detect radiation reflected by the user's Hp as it deforms around the convexity and into the region of space on the first side of the convexity. The optical sensor may additionally or alternatively be configured to detect radiation transmitted through the region as the user's lip deforms around the convexity and into the region of space (i.e. blocking the transmission of radiation). A first portion of the user's lip may be located on the first side of the convexity and a second portion of the user's lip may be received on an opposing side of the convexity. The application of force to the lip may cause the first portion of the user's lip to deform into the region of space adjacent the first side of the convexity. An amount of reflected or transmitted radiation detected by the sensor may be related to an amount of deformation of the user's lip into this region of space. The application of different levels of force to the lip causes different amounts of lip deformation and different amounts of extension of the first portion of the lip into the region, which in turn causes different amounts of reflected or transmitted radiation detected by the sensor. The convexity may be formed by a step profile between two spaced-apart surfaces. [0014] Another aspect of this invention provides a lip-sensing system for the mouth- operated input device which facilitates multi-dimensional control with a single one of the user's lips. Particular embodiments incorporate strategically positioned
pluralities of lip force sensors. Lip force sensors may comprise force sensitive resistors (FSRs). Such FSRs may be pre-loaded by protrusions to assist the user with reaching the FSR threshold activation pressure. Such protrusions may additionally or alternatively be shaped to concentrate lip force onto relatively small areas of the active regions of the FSRs.
[0015] Mouth-operated input devices according to particular embodiments may also comprise an optical oral cavity sensor for emitting radiation into the user's oral cavity and detecting reflected radiation from one or more surfaces of the user's oral cavity. The oral cavity sensor may be insertable into the user's oral cavity. The radiation emitted by the sensor may be oriented such that the reflected radiation detected by the sensor is correlated with a distance between the sensor and a forward facing surface of the user's tongue. The radiation emitted by the sensor may be oriented such that the reflected radiation detected by the sensor is correlated with a distance between an apex of the user's tongue and a roof of the user's mouth. [0016] Mouth-operated input devices according to particular embodiments may also comprise a force sensor which senses the force that the user's mouth is applying (in one or more directions) to the device itself. In particular embodiments, the mouth- operated input device maps this detected force to a position. [0017] Figure 1 schematically depicts a wind instrument simulator 10 incorporating a mouth-operated input device 12 according to a particular embodiment of the invention. In addition to mouth-operated input device 12, instrument simulator 10 may comprise a plurality of additional inputs 14 which may perform a variety of additional functions. By way of non-limiting example, inputs 14 may comprise finger operated keys for note selection, timbre manipulation or other control functions. Inputs 14 may additionally or alternatively be provided on mouth- operated input device 12. Instrument simulator 10 and/or mouth-operated input device 12 may also comprise instrument electronics 16 for generating an output signal 18 based on inputs 14 and/or data received from mouth-operated input device 12. [0018] In the illustrated embodiment, instrument electronics 16 comprise a digital processor for generating output signal 18 and output signal 18 may control the audio output of synthesizer 13. Output signal 18 may comprise one or more control signal(s) which conform to a known audio control protocol or to some other communication and/or control protocol. By way of non-limiting example, such audio protocols may include: MIDI (Musical Instrument Digital Interface), MIDI over USB (Universal Serial Bus), OSC (Open Sound Control), and SKINI
(Synthesis tool Kit Instrument Network Interface). In some embodiments, output signal 18 may alternatively be used as an input signal or as input parameter(s) to an analog audio synthesizer or a digital audio synthesis algorithm. [0019] In the Figure 1 embodiment, mouth-operated input device 12 comprises a housing 20 having a mouth-engaging end 22 with a proximal surface 24, an upper surface 30 and a lower surface 36. A user may interact with mouth-engaging end 22 of housing 20 by placing their upper lip 26 (and possibly one or more of their upper teeth 28) on upper surface 30 and their lower lip 32 (and possibly one or more of their lower teeth 34) on lower surface 36. [0020] Lower surface 36 and/or upper surface 30) may comprise one or more convexities 37. In the illustrated embodiment, lower surface 36 has a stepped profile comprising a first portion (upper portion 36B) and a second portion (lower portion 36A) located on either side of convexity 37. With the stepped profile of the illustrated embodiment, lower surface 36 also comprises a concavity 39 between lower portion 36A and upper portion 36B. In other embodiments, lower surface 36 may comprise more than one convexity 37 and/or more than one concavity 39. [0021] Convexity 37 may provide a relatively sharp corner. In some embodiments, a radius of curvature of convexity 37 is in a range of 0.1 mm - 3 mm. In currently preferred embodiments, the radius of curvature of convexity 37 is in a range of 0.25 mm - 1 mm. In some embodiments, convexity 37 subtends an angle in a range of 30° -150°. In currently preferred embodiments, convexity 37 subtends an angle in a range of 75°-120° .
[0022] In the illustrated embodiment, lower portion 36A and upper portion 36B of lower surface 36 are connected by intermediate surface 40 which extends therebetween. Intermediate surface 40 is shown in the illustrated embodiment as being approximately orthogonal with lower portion 36A and upper portion 36B, but this is not necessary. As explained in more detail below, the user's lower lip 32 may interact with the portion of lower surface 36 at or near convexity 37 (i.e. in a region of the transition between lower portion 36A and upper portion 36B). [0023] Housing 20 may contain a number of sensors for sensing various aspects of the interaction between the user's mouth and input device 12. In the Figure 1 embodiment, input device 12 comprises a breath-pressure-sensor 42, which is connected to the mouth-engaging end 22 by a suitable conduit 44. In the Figure 1 embodiment, input device 12 also comprises an oral cavity sensor 46 and a lower lip sensor 48. Oral cavity sensor 46 may comprise one or more optical reflectivity
sensors and lip sensor 48 comprises one or more optical reflectivity sensors and/or one or more optical transmission sensors.
[0024] Housing 20 and/or instrument 10 may also contain signal conditioning electronics 50, which condition the output signals from sensors 42, 46, 48 before providing them to instrument electronics 16. Instrument electronics 16 may also be located in housing 20 and/or in instrument 10. In the Figure 1 embodiment, sensors 42, 46, 48 share signal conditioning electronics 50. In other embodiments, device 12 and/or instrument 10 comprise independent signal conditioning electronics 50 for each sensor 42, 46, 48. As explained in more detail below, signal conditioning electronics 50 may work cooperatively with instrument electronics 16 to receive signals (not shown in Figure 1) from various sensors 42, 46, 48 and to use these sensor signals to generate output signal 18, which may in turn control the audio output of synthesizer 13. In embodiments where mouth-operated input device is used to control other systems or components, instrument electronics 16 may be replaced by other similar electronics suitable to such other systems or components. [0025] Mouth-operated input device 12 is capable of sensing a number of user- interaction characteristics related to the way in which a user interacts with device 12. Such user-interaction characteristics may be indicative of various aspects of embouchure. Figure 2 schematically depicts how device 12 is capable of sensing spatial characteristics of the user's oral cavity 52 including the user's tongue 54. For clarity, some details of device 12 not related to sensing tongue and/or oral cavity characteristics have been removed from Figure 2.
[0026] Oral cavity sensor 46 my be an optical reflection-type sensor which emits radiation and senses the intensity of reflected radiation. Oral cavity sensor 46 comprises a radiation emitter. In the Figure 2 embodiment, the radiation emitter is a LED emitter 6OA. LED emitter 6OA may emit radiation in the visible, infrared or near infrared spectrum. The approximately 400-660 nanometer wavelength common to a wide variety of LEDs represents one particular choice, as saliva is relatively transparent at such wavelengths. Oral cavity sensor 46 also comprises a radiation detector for sensing reflected radiation. In the Figure 2 embodiment, the radiation detector is a phototransistor 6OB.
[0027] Optical reflection-type sensors of this type are known in the art. Examples of such sensors include the OPB745 manufactured and sold by Optek Technology, Inc. of Carrollton, Texas. In some embodiments, sensor 46 comprises discrete components. For example, LED emitter 6OA may be implemented by the FA1105W-TR LED from Stanley Electric Sales of America, Inc. of Irvine,
California and phototransistor 60B may be implemented by the PT-IOOMC-OMP from Lumex, Inc. of Palatine, Illinois.
[0028] In particular embodiments, oral cavity sensor 46 comprises, or is optically coupled to, a light pipe 58, such that some component(s) of oral cavity sensor 46 (e.g. LED 6OA and/or phototransistor 60B) may be located remotely (i.e. away from mouth-engaging end 22 of device 12). In other embodiments, light pipe 58 is not required and LED 6OA and phototransistor 6OB (or other radiation emitting and radiation detecting components of sensor 46) may be located adjacent to proximal surface 24 of input device 12. In still other embodiments, radiation emitted from LED 6OA (or other radiation emitting device) may travel through a different light pipe than radiation that is incident on phototransistor 60B (or other radiation detection device).
[0029] As shown in the Figure 2 embodiment, additional optics 62A may be provided for shaping and/or directing the sensor output radiation and additional optics 62B may be provided for coupling remotely located LED 6OA and phototransistor 6OB (or other radiation emitting and radiation detecting components of sensor 46) to light pipe 58. As is known in the art, such additional beam shaping and/or directing optics 62 may comprise, without limitation, lenses, mirrors, prisms, gratings, beam splitters, polarizers and other beam-shaping and/or coupling optical elements. Such additional optics 62 may be static or moveable. Optics 62 may comprise, or have their outer surfaces coated with, hydrophilic material, such that saliva does not bead on their outer surfaces.
[0030] Optics 62 may comprises polarizer(s) and/or polarizing filter(s). Polarization of the sensor output beam using a polarizing filter may reduce the effect of specular reflection caused by saliva. If reflected radiation is passed through a polarizing filter orthogonal to that of the sensor output beam, then specularly reflected radiation will tend to be blocked by the orthogonal filter because specularly reflected radiation retains its polarization whereas diffusely reflected radiation does not. [0031] As shown in Figure 2A, sensor 46 may optionally be controlled by signal 23 from instrument electronics 16 and/or by signal 25 from signal conditioning electronics 50. Instrument electronics 16 and signal conditioning electronics 50 may comprise one or more controllers suitably configured for this purpose. Signals 23, 25 may comprise one way or two way control signals. For clarity, signals 23, 25 are not shown in Figures 2B-2F.
[0032] Radiation emitted from sensor 46 may be reflected from one or more surfaces of the user's mouth, including, for example, lips 26, 32, teeth 28, 34, tongue 54, roof 53, cheeks (not shown) and throat 56.
[0033] One characteristic of the user's oral cavity 52 is the position of the user's tongue 54 relative to the front of oral cavity 52 and/or relative to device 12 (i.e. the forward-backward position of tongue 54). Oral cavity sensor 46 is capable of detecting radiation reflected from a "forward-facing surface" 51 of tongue 54. "Forward-facing surface" 51 may be defined as any surface of tongue 54 capable of directly receiving radiation emitted from sensor 46. Radiation reflected from forward-facing surface 51 of tongue 54 may be sensed by oral cavity sensor 46 and may be used to obtain an oral cavity sensor output signal 64 which is related to the position of the user's tongue 54 relative to the front of oral cavity 52. [0034] In some embodiments, oral cavity sensor 46 is designed such that oral cavity sensor output signal 64 changes monotonically as forward-facing surface 51 of tongue 54 moves forward or backward within oral cavity 52. Monotonic variation of oral cavity sensor output signal 64 with the position of forward-facing surface 51 of tongue 54 may be achieved by designing and/or selecting various parameters of oral cavity sensor 46 including, by way of non-limiting example: angular beam shape or profile, beam width, beam orientation, uniformity of illumination across the beam and polarization of the beam. These parameters may also be designed and/or selected to provide oral cavity sensor output signal 64 with a suitably high dynamic range.
[0035] Figures 2A-2D illustrate various possible positions of the user's tongue 54.
In Figure 2A, the user has configured their tongue 54, such that forward-facing surface 51 is positioned relatively close to proximal surface 24 of device 12. In
Figure 2B, the user's tongue 54 is configured such that its forward-facing surface 51 is further from proximal surface 24 of device 12. In Figures 2C and 2D, the user has curved their tongue 54 such that tongue 54 is pointed upwardly and forward- facing surface 51 is actually the surface typically referred to as the undersurface of tongue 54. In Figure 2C, forward-facing surface 51 of tongue 54 is positioned relatively close to proximal surface 24 of device 12. In Figure 2D, forward-facing surface of tongue 54 is positioned relatively far from proximal surface 24 of device
12.
[0036] Another characteristic of the user's oral cavity 52 is the height of the apex 55 of the user's tongue 54 relative to roof 53 of oral cavity 52 and to device 12. This oral cavity characteristic may be correlated to a cross-sectional area of oral cavity
52 in the region of apex 55. In Figure 2E, the user has configured their tongue 54 and/or their jaw (not shown), such that the apex 55 of tongue 54 is relatively far from the roof 53 of their mouth and the cross-sectional area of the user's oral cavity 52 in the region of apex 55 is relatively large. In Figure 2F, the user has configured their tongue 54 and/or their jaw, such that the apex 55 of tongue 54 is relatively close to the roof 53 of their mouth and the cross-sectional area of the user's oral cavity 52 in the region of apex 55 is relatively small.
[0037] The height of apex 55 of tongue 54 may also be adjusted relative to device 12 when tongue 54 is pointed upwardly as shown in Figures 2C and 2D. In such a configuration, apex 55 is at or near the tip of tongue 54. In Figure 2C, the user has adjusted their tongue 54 and/or their jaw such that apex 55 is relatively far from the roof 53 of oral cavity 52 and in Figure ID, the user has adjusted their tongue 54 and/or their jaw such that apex 55 is relatively close to the roof 53 of oral cavity 52.
[0038] Comparing Figures 2A and 2B, it can be seen that when forward-facing surface 51 of tongue 54 is located closer to proximal surface 24 of device 12 (Figure 2A), there will be more radiation reflected back toward sensor 46 (i.e. optics 62A) from the user's tongue 54 (and other mouth surfaces). Consequently, sensor 46 will detect a relatively large amount of reflected radiation. When forward-facing surface 51 of tongue 54 is located further from proximal surface 24 of device 12 (Figure 2B), there will be less radiation reflected back toward sensor 46 from the user's tongue 54 (and other mouth surfaces). Consequently, sensor 46 will detect a relatively small amount of reflected radiation. [0039] Similarly, comparing Figures 2C and 2D, it can be seen that when tongue 54 is curved upwardly and forward-facing surface 51 is located closer to proximal surface 24 of device 12 (Figure 2C), sensor 46 will detect a relatively large amount of reflected radiation and when forward-facing surface 51 of tongue 54 is located further from proximal surface 24 of device 12 (Figure 2D), sensor 46 will detect a relatively small amount of reflected radiation. [0040] Comparing Figures 2E and 2F, it can be seen that when apex 55 of tongue 54 is located closer to roof 53 (Figure 2F), there will be more radiation reflected back toward sensor 46 from the user's tongue 54. Consequently, sensor 46 will detect a relatively large amount of reflected radiation. When apex 55 of tongue 54 is located further from roof 53 (Figure 2E), there will be less radiation reflected back toward sensor 46 from the user's tongue 54. Consequently, sensor 46 will detect a relatively small amount of reflected radiation. Similarly, comparing Figures 2C and
2D, it can be seen that when tongue 54 is curved upwardly and apex 55 is located closer to roof 53 (Figure 2D), sensor 46 will detect a relatively large amount of reflected radiation and when apex 55 of tongue 54 is located further from roof 53 (Figure 2C), sensor 46 will detect a relatively small amount of reflected radiation. [0041] When playing a conventional wind instrument, a musician frequently (and often subconsciously) moves the surfaces of their mouth (e.g. their lips, teeth, tongue, cheeks, roof and/or throat) to change the spatial characteristics of their oral cavity and to thereby modulate the sound emanating from the instrument. For example, by raising the apex of their tongue, a musician may change the shape and/or size of the resonant cavity in their mouth and may thereby change the characteristics of the resonant cavity formed by the combination of the instrument and the musician's mouth. These changes in the resonant cavity may cause corresponding changes in the frequency, pitch, timbre or other characteristics of the sound created by the instrument. The spatial characteristics of the musician's oral cavity represent an aspect of the musician's embouchure.
[0042] Device 12 models this aspect of a conventional instrument. Oral cavity sensor 46 senses the spatial characteristics of the user's oral cavity 52 (including the user's tongue 54) and creates an oral cavity sensor output signal 64 representative of these spatial characteristics. Signal processing electronics 50 and/or instrument electronics 16 may use oral cavity sensor output signal 64 to generate one or more processed oral cavity signals 17. Instrument electronics 16 may further process processed oral cavity signals 17 to generate output signal 18. Output signal 18 may in turn control the audio output of synthesizer 13 (Figure 1). [0043] Oral cavity sensor output signal 64 may be processed by signal conditioning electronics 50 and/or instrument electronics 16. The processing performed on oral cavity sensor output signal 64 by signal conditioning electronics 50 and/or instrument electronics 16 may take place in the analog and/or digital domain and may generally involve any of a variety of processing techniques, including, without limitation: digitizing, amplifying, inverting, filtering, scaling, offsetting, linearizing, differentiating, integrating, averaging, normalizing, performing linear and/or nonlinear mathematical transformations (e.g. Fourier transforms and logarithmic functions) and the like. The result of the processing by signal conditioning electronics 50 and/or instrument electronics 16 is one or more processed oral cavity signal(s) 17, which may generally comprise any function &~ of oral cavity sensor output signal 64. The function & used to generate processed oral
cavity signal(s) 17 may include any of the analog or digital processing techniques discussed above.
[0044] Figure 14 schematically illustrates a normalization circuit 61 according to a particular embodiment of the invention, which represents a technique for processing oral cavity sensor output signal 64 to generate a normalized oral cavity signal 64' according to a particular embodiment of the invention. In general, normalization circuit 61 may be implemented as a part of signal conditioning electronics 50 and/or instrument electronics 16. Normalization circuit 61 normalizes oral cavity sensor output signal 64 to account for variations in the amount of ambient light in the environment in which device 12 is being used. Such ambient light may penetrate the cheeks of a user and be detected by optical sensor 46, for example. Normalization circuit 61 or components of normalization circuit 61 may be controlled by a controller (not shown). [0045] Normalization circuit 61 takes two temporally spaced apart samples of oral cavity sensor output signal 64. A first sample 64 A is taken when LED 6OA (see
Figure 2) is ON and a second sample 64B is taken a short time (Δt) later when LED 6OA is OFF. Sample 64B represents the ambient light level detected by oral cavity sensor 46. The time (Δt) between the first and second samples may be less than about 1 ms. In particular embodiments, the time (Δt) is less than about 100 μs. Oral cavity sensor output signal samples 64 A, 64B are amplified by corresponding linear amplifiers to provide amplified oral cavity sensor output signal samples 64A', 64B'. Circuit 61 comprises a sample and hold circuit 71 which is used to temporarily retain one of the one of the amplified oral cavity sensor output signal samples (sample 64 A' in the illustrated embodiment). [0046] Amplified oral cavity sensor samples 64A', 64B' are then provided to difference amplifier 59, which amplifies a difference between amplified oral cavity sensor samples 64A', 64B' to provide difference signal 19. It will be appreciated that difference signal 19 is representative of the normalized oral cavity output signal (i.e. an oral cavity output signal that is attributable directly to the signal from LED 6OA and not to ambient radiation). In the illustrated embodiment, circuit 61 also comprises optional signal conditioning circuitry 73A and a D/A converter 73B which digitize difference signal 19 to output normalized oral cavity signal 64' . Optional signal conditioning circuitry 73A may comprise a low pass filter and additional amplification for example. [0047] Normalization circuit 61 represents one particular circuit for processing oral cavity sensor output signal 64 to generate a normalized oral cavity output signal 64'.
Normalized oral cavity sensor output signal 64' may be further processed using other signal processing circuits and/or processing algorithms to produce one or more different processed oral cavity output signal(s) 17 (Figure 2). In some embodiments, normalization circuit 61 is not required. Those skilled in the art will appreciate that other normalization circuits may be envisaged to reduce the impact of ambient light on processed oral cavity output signal(s) 17. In particular, the difference between the ambient oral cavity signal (i.e. when the emitter is inactive) and the oral cavity signal when the emitter is active may be obtained in the digital (rather than analog) domain. [0048] Processed oral cavity signal(s) 17 may be used by instrument electronics 16 to generate output signal 18 which may in turn control the audio output of synthesizer 13. Processed oral cavity signal(s) 17 are related to the sensed spatial characteristics of the user's oral cavity 52. Accordingly, using mouth-operated input device 12, the audio output of synthesizer 13 may be controlled by the spatial characteristics of the user's oral cavity 52. Such spatial characteristics may include: the volume of oral cavity 52, the distance of the apex 55 of tongue 54 from the proximal surface 24 of device 12, the distance between the apex 55 of tongue 54 and the roof 53 of the user's mouth, and/or the cross-sectional area of oral cavity 52 in the region of apex 55, for example. Processed oral cavity signal(s) 17 may additionally or alternatively be correlated to the rate of change of any of these sensed spatial characteristics of the user's oral cavity 52. [0049] In the illustrated embodiment, signal conditioning electronics 50 and instrument electronics 16 are separate components. Signal conditioning electronics 50 may additionally or alternatively be a part of oral cavity sensor 46, instrument electronics 16 and/or synthesizer 13. Instrument electronics 16 may also be a part of signal conditioning electronics 50 and/or synthesizer 13. Depending on the nature of signal conditioning electronics 50 and instrument electronics 16, oral cavity output signal 64 may have many forms. In the illustrated embodiment, signal conditioning electronics 50 cooperate with instrument electronics 16 to generate audio output signal 18 based at least in part on oral cavity sensor output signal 64.
[0050] The radiation emitted by oral cavity sensor 46 (or its associated light pipe 58 or beam control optics 62A) may have a conical shape with a divergence profile in a range of 20° -100° . Such a divergence profile provides oral cavity sensor 46 (and oral cavity sensor signal 64) with a suitably high dynamic range which can accurately detect small changes in the spatial characteristics of oral cavity 52. The
radiation emitted by oral cavity sensor 46 may be substantially uniform throughout the conical profile.
[0051] The radiation emitted from oral cavity sensor 46 may be oriented slightly downwardly (i.e. toward throat 56), so that sensor 46 may accurately sense reflection from tongue 54. The radiation emitted from sensor 46 may be oriented at a downward angle in a range of 0°-80° with respect to a longitudinal axis 21 (see Figure 1) of input device 12 (i.e. an axis of input device 12 which extends into the user's mouth and which may be generally parallel with a roof 53 of the user's mouth). In particular embodiments, the radiation emitted from sensor 46 is oriented at a downward angle in a range of 0°-45° with respect to the longitudinal axis 21 of input device 12. In currently preferred embodiments, this angular range is 5°-30° . The radiation emitted from oral cavity sensor 46 may alternatively be oriented slightly upwardly. In some embodiments, the radiation emitted from sensor 46 is oriented at an upward angle in a range of 0°-45° with respect to the longitudinal axis 21 of input device 12. In currently preferred embodiments, this angular range is 5°-30°.
[0052] The orientation with which the user holds instrument 10 (Figure 1) and input device 12 relative to their mouth will impact the angle of radiation emitted from oral cavity sensor 46 relative to throat 56. That is, the user may be capable of altering the orientation of the longitudinal axis 21 of input device 12 and, in so doing, may also be capable of altering the angle of radiation emitted from oral cavity sensor 46. In some embodiments, the radiation emission angle of oral cavity sensor 46 is made adjustable, for example by one or more suitable mechanical micro-manipulators. A suitable micro-manipulator may be an adjustment screw similar to the type used in corrective eyewear, for example. A user may also purposefully adjust the orientation of instrument 10 and input device 12 to influence the oral cavity reflectivity response characteristics.
[0053] Figure 3 depicts a mouth-operated input device 212 according to another embodiment of the invention. In many respects, mouth-operated input device 212 of Figure 3 is similar to mouth-operated input device 12 of Figure 1. Features of device 212 which are similar to features of device 12 are referred to using similar reference numerals that are preceded by the digit "2". One aspect of device 212 that differs from device 12 is that device 212 has an oral cavity sensing system 209 which comprises two oral cavity sensors 246', 246". In the Figure 3 embodiment, oral cavity sensors 246', 246" are transversely spaced apart from one another, although they may be spaced apart in some other direction.
[0054] Figure 4 is a schematic block diagram of oral cavity sensing system 209 of the Figure 3 input device 212 according to a particular embodiment of the invention. Each of oral cavity sensors 246', 246" may be implemented in a manner similar to, and have characteristics similar to, oral cavity sensor 46 of device 12 described above. The radiation emitted from oral cavity sensors 246', 246" may be oriented in different directions. In the illustrated embodiment, radiation emitted from sensor 246' is oriented slightly downwardly (i.e. toward tongue 54 and throat 56) and radiation emitted from sensor 246" is oriented slightly upwardly (i.e. toward roof 53 of the user's mouth). [0055] The radiation emitted from sensor 246' may be oriented at a downward angle in a range of 0°-80° with respect to the longitudinal axis 221 of input device 212 and the radiation emitted from sensor 246" may be oriented at an upward angle in a range of 0°-80° with respect to the longitudinal axis 221 of input device 212. In particular embodiments, sensor 246' is oriented at a downward angle in a range of 0°-45° and the radiation emitted from sensor 246" is oriented at an upward angle in a range of 0°-45°. In currently preferred embodiments, these angular range are 5°-30°. In this manner, mouth-operated input device 212 and its oral cavity sensors 246', 246" sense different spatial characteristics of the user's oral cavity 52. [0056] Other embodiments are possible wherein one or more additional or alternative sensors are oriented at different angles which may be the same as one another or different than one another. For example, as shown schematically in dotted outline in Figure 4, oral cavity sensing system 209 may comprise an optional, additional or alternative pair of oral cavity sensors 246 A' 246A". Additional or alternative oral cavity sensors 246 A' 246 A" may be oriented such that sensor 246 A' is oriented at a first transverse (i.e. sideways) angle with respect to the longitudinal axis 221 of device 212 and sensor 246 A" is oriented at an opposing transverse angle with respect to the longitudinal axis 221 of device 212. The opposing transverse angles of sensors 246A', 246 A" may be in a range of 0°-90° with respect to the longitudinal axis 221 of device 212. In currently preferred embodiments, thee angular range are 10° -45° .
[0057] As shown in Figure 4, oral cavity sensors 246', 246" sense spatial characteristics of the user's oral cavity 52 and generate corresponding oral cavity sensor output signals 264', 264" representative of these spatial characteristics. Oral cavity sensor output signals 264', 264" may be related to: the volume of oral cavity 52, the distance of the forward-facing surface 51 of tongue 54 from the proximal surface 24 of device 12, the distance between the apex 55 of tongue 54 and the roof
53 of the user's mouth (which may be dependent on tongue shape and jaw angle) and/or the cross-sectional area of oral cavity 52 in the region of apex 55 (which may also be dependent of tongue shape and jaw angle). Oral cavity sensor output signals 264', 264" may be normalized using a circuit similar to circuit 61 (Figure 14) or some other normalization process to substract out the effect of ambient light.
[0058] Oral cavity sensor output signals 264', 264" may be used to generate one or more processed oral cavity signals 217. Instrument electronics 216 use processed oral cavity signals 217 to generate output signal 218. In the illustrated embodiment, oral cavity sensor output signals 264', 264" are used to generate two processed oral cavity signals 217A, 217B. Instrument electronics 216 may make use of one or both of processed oral cavity signals 217A, 217B to generate output signal 218. Output signal 218 may in turn control the audio output of synthesizer 13 (Figure 1). In other embodiments, oral cavity sensor output signals 264', 264" are used to generate a different number of processed oral cavity signal(s) 217. [0059] To generate processed oral cavity signals 217A, 217B, oral cavity sensor output signals 264', 264" may be processed in the analog and/or digital domain by their corresponding signal conditioning electronics 250', 250" and/or by instrument electronics 216. Signal conditioning electronics 250', 250" and instrument electronics 216 may have features similar to signal conditioning electronics 50 and instrument electronics 16 described above.
[0060] In general, processed oral cavity signals 217A, 217B may be any function & of one or both of oral cavity sensor output signals 264', 264". The function £F used to generate processed oral cavity signals 217A, 217B may include any of the analog or digital processing techniques discussed above. In particular, oral cavity sensor output signals 264', 264" may be normalized using a circuit similar to circuit 61
(Figure 14) as a part of any processing performed by signal conditioning electronics 250', 250" and/or instrument electronics 216. For the discussion that follows, it is assumed that oral cavity output sensor output signals 264', 264" are normalized. [0061] Normalized oral cavity output signals 264', 264" may be further processed. By way of non-limiting example, processed oral cavity signal 217A may be a difference function between normalized oral cavity sensor output signals 264" and 264' and processed oral cavity signal 217B may be a scaled and possibly offset function of normalized oral cavity sensor output signal 264'. [0062] The inventor has discovered that when sensors 246', 246" are oriented as discussed above (i.e. sensor 246' oriented at a slight downward angle and sensor 246" oriented at a slight upward angle), the difference between normalized oral
cavity sensor output signals 264' and 264" is related to the distance between the apex 55 of tongue 54 and the roof 53 of the user's mouth and/or the cross-sectional area of oral cavity 52 in the region of apex 55. This parameter is also related to the angle of the user's jaw (not shown) relative to the roof 53 of their mouth. Accordingly, processed oral cavity signal 217A may be derived from a difference function having the form ^=aA-bB+c, where A and B represent normalized oral cavity output signals 264', 264"; a and b represent scaling coefficients; and c represents an offset coefficient. Furthermore, when sensor 246' is oriented as discussed above, oral cavity sensor output signal 264' is related to the distance of the forward-facing surface 51 of tongue 54 from the proximal surface 24 of device 12. Accordingly, processed oral cavity signal 217B may be derived from a function having the form ^"=dA+e, where A represents normalized oral cavity sensor output signal 264'; d represents a scaling coefficient; and e represents an offset coefficient. [0063] These functions are merely examples of the types of functions which may be used to generate processed oral cavity signals 217A, 217B from normalized oral cavity sensor output signals 264', 264". Processed oral cavity signals 217A, 217B may generally comprise any function ^ of one or both of normalized oral cavity sensor output signals 264', 264". [0064] Without being bound by any particular theory, it is thought that one of the resonant frequencies of a user's oral cavity 52 is related to the volume of the cavity between the forward-facing surface 51 of the tongue 54 and user's lips 26, 32. Those skilled in the art will appreciate that a processed oral cavity signal 217A, 217B representative of the location of forward-facing surface 51 relative to the proximal surface 24 of device 12 may be used as a basis for approximating this resonance. Similarly, it is thought that another one of the resonant frequencies of a user's oral cavity is related to the cross-sectional area of the constriction between the apex 55 of the user's tongue 54 and the roof 53 of the user's mouth. Accordingly, a processed oral cavity signal 217A, 217B representative of the height of apex 55 relative to roof 53 may be used as a basis for approximating this resonance.
[0065] Figure 4 depicts optional, additional or alternative oral cavity sensors 246A', 246A". These oral cavity sensors 246A', 246A" may generate oral cavity sensor output signals 264A', 264A", which may be processed by signal conditioning electronics 250A', 250A" and by instrument electronics 216. The processing of oral cavity sensor output signals 264A', 264A" may be similar in many respects to that
discussed above for oral cavity sensor output signals 264', 264" . In particular, oral cavity sensor output signals 264A', 264 A" may be normalized using a circuit similar to circuit 61 (Figure 14) as a part of any processing performed by signal conditioning electronics 250A', 250A" and/or instrument electronics 216. For the discussion that follows, it is assumed that oral cavity sensor output signals 264A', 264 A" are normalized.
[0066] Oral cavity sensor output signals 264A', 264A" may be used by instrument electronics 216 to help generate processed oral cavity signals 217A, 217B. For example, processed oral cavity signals 217A, 217B may generally comprise any function &~ of oral cavity sensor output signals 264', 264" and/or oral cavity sensor output signals 264A', 264A" .
[0067] Oral cavity sensor output signals 264A', 264A" may also be used by instrument electronics 216 to generate additional or alternative processed oral cavity signals 217 (not shown in Figure 4). Such additional or alternative processed oral cavity signals 217 may generally be any function ^~of oral cavity sensor output signals 264', 264" and/or oral cavity sensor output signals 264A', 264A". [0068] In one particular example, where sensors 246A', 246 A" are oriented at slightly transverse (i.e. sideways) angles (as described above), oral cavity sensor output signals 264A', 264A" may be related to the side to side position of tongue 54 in oral cavity 52. For example, the difference between oral cavity sensor output signals 264A' and 264A" may be related to the side to side position of tongue 54 Accordingly, a processed oral cavity signal 217 representative of the side to side position of tongue 54 may be derived from a difference function having the form ^=fA-gB+h, where A and B represent oral cavity output signals 264A', 264A"; f and g represent scaling coefficients; and h represents an offset coefficient.
[0069] Another example of generating a processed oral cavity signal 217 involves applying a high pass filter to one of oral cavity sensor output signal 264', 264", 264A', 264 A" and/or some combination of oral cavity sensor output signals 264', 264", 264A', 264A". A processed oral cavity signal 217 which has been high-pass filtered may be related to the gesture of "flutter tongue" . Flutter tongue is a gesture used by conventional wind instrument musicians, where the musician vibrates their tongue at a relatively high frequency in a manner similar to that involved in rolling an 'R' sound in speech. [0070] Another example of generating a processed oral cavity signal 217 involves applying a threshold filter to one of oral cavity sensor output signal 264', 264", 264A', 264 A" and/or some combination of oral cavity sensor output signals 264',
264", 264A', 264A". A processed oral cavity signal 217 which is above a certain threshold may indicate that the user's tongue is touching proximal surface 24 which may be related to the gesture of "tonguing". Tonguing is a gesture which may used by conventional wind instrument musicians to articulate spaces between notes by influencing the flow of air through the instrument. Another form of tonguing is a gesture which may be used by players of conventional reed instruments to stop the vibration of the reed with the surface of their tongue.
[0071] In another embodiment, oral cavity sensing system 209 incorporates an additional sensor (not shown) for detecting a signal representative of the short range proximity of the end of the user's tongue 54. Such a sensor may also be an optical reflection-type sensor and the sensor and/or its signal conditioning electronic may be specifically designed and/or selected for short range proximity detection or threshold proximity detection. Such a sensor may be oriented sharply downwardly from proximal surface 224 of device 212 and can be used to detect when tongue 54 touches mouthpiece 212 at or near proximal surface 224. This sensor may also be capable of providing a sensor output signal related to the concept of "tonguing" . [0072] Instrument electronics 216 may make use of processed oral cavity signals 217A, 217B (and any additional processed oral cavity signals 217 which are not shown in Figure 4) to generate output signal 218 which may in turn control the audio output of synthesizer 13 (Figure 1). In one particular embodiment, processed oral cavity signals 217A, 217B are related to a pair of sensed spatial characteristics of the user's oral cavity 52 (e.g. the distance between the apex 55 of tongue 54 and the roof 53 of the user's mouth and the distance of the forward-facing surface 51 of tongue 54 from the proximal surface 24 of device 12). In this embodiment, mouth-operated input device 212 may be used to independently control the audio output of synthesizer 13 by way of two different spatial characteristics of the user's oral cavity
52.
[0073] As discussed above, oral cavity sensor output signals 264', 264", 264A',
264A" may be processed to generate any number of processed oral cavity signal(s) 217, each of which may (alone or in combination with other oral cavity sensor output signals 264', 264", 264A', 264A") be related to a different spatial characteristic of the user's oral cavity. The provision of multiple oral cavity sensors 246', 246", 246A', 246 A" allows for a model of the user's oral cavity that has multiple degrees of freedom (i.e. multiple processed oral cavity signals 217 that may relate to a corresponding plurality of physical gestures/characteristics). Processed oral cavity
signal(s) 217 may additionally or alternatively be correlated to the rate of change of any of these sensed spatial characteristics of the user's oral cavity 52. [0074] Figure 5 schematically depicts how mouth-operated input device 12 (Figure 1) is capable of sensing the force applied to the mouth-engaging end 22 of device 12 by lower lip 32 according to a particular embodiment of the invention. For clarity, details of device 12 which are not related to sensing the force applied by lower lip 32 are not shown in Figures 5. As discussed above, in the illustrated embodiment the user's lower lip 32 contacts mouth-engaging end 22 of device 12 at or near convexity 37. Lower lip 32 comprises an inward portion 32A (located on an inward side 37A of convexity 37) and an outward portion 32B (located on an outward side 37B of convexity 37). Outward portion 32B of lip 32 may be received on lower portion 36A of lower surface 36.
[0075] In Figure 5A, the user is applying a relatively small force with inward portion 32A of lower lip 32, such that lip 32 is not substantially deformed. With the low level of force applied in Figure 5 A, inward portion 32A and outward portion 32B of lip 32 both extend upwardly approximately to the level of lower portion 36A of lower surface 36.
[0076] In Figure 5B, the user is applying an intermediate force with inward portion 32A of lower lip 32, such that inward portion 32A of lip 32 deforms around convexity 37 and extends upwardly above lower portion 36A and into a region of space 33 on inward side 37A of convexity 37 and adjacent intermediate surface 40. Region 33 may be located between convexity 37 and concavity 39. Region 33 may be located between a plane substantially parallel with a portion of lower portion 36A immediately outside 37B of convexity 37 and a plane substantially parallel with a portion of upper portion 36B on the inside 37A of convexity 37.
[0077] It should be noted that the location and shape of outward portion 32B of lower lip 32 need not change significantly as the user applies more lip-force to inner portion 32A of lower lip 32. Outward portion 32B of lower lip 32B may continue to rest on lower portion 36A of lower surface 36 and may be independently controlled by the user for a different purpose as discussed in more detail below. When outward portion 32B of lower lip 32 does not move, the aperture between the outward portions of upper lip 26 and lower lip 32 is maintained at a relatively constant size, which is approximately equal to the circumference of device 12. [0078] In Figure 5C, the user is applying a relatively large force with inward portion 32 A of lower lip 32, such that inward portion 32A of lip 32 deforms around convexity 37 and extends a significant distance above lower portion 36A and into
region 33. Inward portion 32A of lip 32 may also deform such that it is closer to intermediate surface 40. When enough force is applied to lower lip 32, inward portion 32A of lip 32 may reach upper portion 36B of lower surface 36 and may become flattened against upper portion 36B. Similarly, inward portion 32A of lip 32 may reach intermediate surface 40 and may become flattened against intermediate surface 40. Once again, the location of outward portion 32B of lower lip 32 and the aperture between the outward portions of lips 26 and 32 need not change as the user applies more lip-force with inward portion 32A of lower lip 32. [0079] In the Figure 5 embodiment, mouth-operated input device 12 comprises a lip sensor 48. Lip sensor 48 may be a optical reflection-type sensor which emits radiation and senses the intensity of reflected radiation of the same general type as oral cavity sensor 46 discussed above. Lip sensor 48 may be oriented to direct radiation into region 33 on inward side 37A of convexity 37. In the Figure 5 embodiment, lip sensor 48 is positioned to direct radiation out of intermediate surface 40 and into region 33. In other embodiments, lip sensor 48 is positioned to direct radiation out of upper surface 36B and into region 33. [0080] As with oral cavity sensor 46, lip sensor 48 may comprise, or may be optically coupled to, a light pipe 66, such that some components of lip sensor 48 (e.g. radiation emitter 7OA and radiation detector 70B) may be located remotely. In other embodiments, light pipe 66 is not required and radiation emitter 7OA and radiation detector 7OB may be located and/or oriented to direct radiation directly into region 33. Lip sensor 48 and/or light pipe 66 may comprise additional optics 68A for shaping and/or directing the sensor output radiation and/or additional optics 68B for coupling remotely located radiation emitter 7OA and detector 7OB to light pipe 66. Optics 68 may be similar to optics 62 discussed above.
[0081] Radiation emitted from lip sensor 48 into region 33 is reflected from inward portion 32A of the user's lower lip 32. As will be discussed in more detail below, the shape of the mouth-engaging end 22 of device 12 (i.e. providing region 33 on inward side 37A of convexity 37) allows for optical sensing of the lip-force applied by inward portion 32A of lower lip 32 with a high degree of dynamic range. In the illustrated embodiment, region 33 is created by the stepped profile of lower surface 36 (i.e. spaced-apart lower and upper portions 36A, 36B), which permit lower lip 32 to deform around convexity such that inward portion 32A of lip 32 extends into region 33. [0082] Figure 5 A shows that when the force applied by the user to the inward portion 32A of lower lip 32 is relatively small, lower portion 36A and convexity 37
maintain the inward portion 32 A of lower lip 32 at a location that is, for the most part, at or below the level of lower portion 36A (i.e. not extending substantially into region 33). When inward portion 32A of lower lip 32 does not extend substantially into region 33, lip sensor 48 detects relatively little reflected radiation, because a significant portion of radiation 72 emitted by sensor 48 either does not impinge on lower lip 32 or reflects from lower lip 32, but is not received by sensor 48. [0083] Lip sensor 48 detects a larger amount of reflected radiation when the force applied to inward portion 32A of lower lip 32 is at the intermediate level of Figure 5B, because inward portion 32A of lower lip 32 deforms around convexity 37, extends above lower portion 36A and into region 33 on inward side 37A of convexity 37. When inward portion 32A of lip 32 extends into region 33, it reflects more radiation back toward sensor 48.
[0084] Figure 5C shows that when the force applied to inward portion 32A of lower lip 32 is at a relatively high level, inward portion 32A of lower lip 32 deforms around convexity 37, extends significantly above lower portion 36A and into region 33. When inward portion 32A of lip 32 is forced harder, it may deform further by flattening against upper portion 36B and/or against intermediate surface 40. Under the lip-force conditions of Figure 5C, lip sensor 48 detects even more reflected radiation, because a larger percentage of the radiation emitted by lip sensor 48 is reflected by inward portion 32A of lip 32 and detected by sensor 48.
[0085] When playing a conventional wind instrument, a musician frequently (and often subconsciously) changes the force applied by their lip(s) on the mouthpiece of the instrument to modulate the sound emanating from the instrument. For example, by applying different levels of force, a musician may alter the frequency, pitch, timbre or other characteristics of the instrument and thereby change the sound created by the instrument. The force applied by the musician's lip(s) represents an aspect of the musician's embouchure.
[0086] Device 12 models this aspect of a conventional instrument. Lip sensor 48 senses the force applied by inward portion 32A of the user's lip 32 and creates a lip sensor output signal 76 representative of this force. Lip sensor output signal 76 may be used by signal conditioning electronics 50 and/or instrument electronics 16 to generate one or more processed lip signals 77. Instrument electronics 16 may then use processed lip signals 77 to generate output signal 18. Output signal 18 may in turn be used to control the audio output of synthesizer 13 (Figure 1). [0087] Those skilled in the art will appreciate that the size of region 33 (i.e. the distance between upper portion 36B and lower portion 36A) and/or other parameters
of lip sensor 48 may be designed so that lip sensor output signal 76 is a monotonically increasing function of the force applied by inward portion 32 A of lip 32 to mouth-engagement end 22 of device 12. In particular embodiments, the spacing between upper and lower portions 36 A, 36B is in a range of 2-15 mm. Other aspects of device 12 which may be used to provide a monotonic relationship between lip sensor output signal 76 and the force applied by inward portion 32A of lip 32 include, without limitation: angular beam shape or profile, beam width, beam orientation, uniformity of illumination across the beam, polarization of the beam, the radius of curvature of convexity 37 and reflectivity of mouthpiece 12. These parameters may also be designed to provide Hp sensor output signal 76 with a suitably high dynamic range which varies between about zero applied force to the maximum lip-force that a normal user may apply.
[0088] In the illustrated embodiment, lip sensor output signal 76 is processed by signal conditioning electronics 50 and/or instrument electronics 16 to output a single processed lip signal 77. In other embodiments, lip sensor output signal 76 may be processed by signal conditioning electronics 50 and/or instrument electronics 16 to output a plurality of processed lip signal(s) 77. Signal conditioning electronics 50 and instrument electronics 16 may function in a manner similar to, and have characteristics similar to, those discussed above with respect to oral cavity sensor output signal 64. In particular, signal conditioning electronics and/or instrument electronics 16 may comprise a normalization circuit that operates to provide normalized lip sensor output signals 76. Such a normalization circuit may operate in a manner substantially similar to normalization circuit 61 (Figure 14) discussed above for oral cavity sensor output signal 64. For the discussion that follows, it is assumed that lip sensor output signal 76 is normalized.
[0089] Processed lip signal 77 may generally be any function W of lip sensor signal 76. Instrument electronics 16 may use processed lip signal 77 to generate output signal 18 which in turn controls the audio output of synthesizer 13. Processed lip signal 77 is related to the sensed lip-force applied by the user to inward portion 32A of lower lip 32. Accordingly, mouth-operated input device 12 may be used to modulate the audio output of synthesizer 13 based on the lip-force applied by a user. Processed lip signal(s) 77 may additionally or alternatively be correlated to the rate of change of the sensed lip-force applied by the user. [0090] Convexity 37 and/or the spaced apart lower and upper portions 36A, 36B of lower surface 36 may also function to provide a user with tactile feedback. The user will be able to feel their lip 32 deforming around convexity 37 and into region 33 as they apply greater lip-force. Convexity 37 and/or the spaced apart lower and upper portions 36A, 36B of lower surface 36 also provide the user with an identifiable
"home position" around which the user can control their lip-force and lip-position with high precision. In addition to, or as an alternative to, forcing lip 32 against convexity 37 to deform inward portion 32A of lip 32 into region 33, the user may change the position of their lip 32 with respect to region 33 by moving their lip 32 or by moving instrument 10 to reflect differing amounts of radiation into lip sensor 48 and to thereby influence lip sensor output signal 76, processed lip signal 77, output signal 18 and the sound created by instrument 10 (Figure 1). [0091] A user may also move their lower lip 32 and/or instrument 10 to change the relative size of lower lip portions 32A, 32B (i.e. the amount of lip 32 that is on either side of convexity 37). If lower lip 32 is positioned relative to convexity 37, such that inward portion 32A is relatively small and outward portion 32B is relatively large, then a relatively large amount of force will be required to cause inward portion 32 A to deform a certain distance into region 33 and to thereby effect a certain lip sensor output signal 76. Conversely, if lower lip 32 is positioned relative to convexity 37, such that inward portion 32 A is relatively large and outward portion 32B is relatively small, then a relatively small amount of force will be required to cause the same amount of deformation and to effect the same lip sensor output signal 76. [0092] Mouth-operated input device 212 of the Figure 3 embodiment incorporates a lip sensing system 211 which comprises three lip sensors 248L, 248C and 248R. Lip sensors 248L, 248C, 248R are oriented to direct radiation into region 233 on inward side 237A of convexity 237. In the illustrated embodiment, lip sensors 248L, 248C, 248R are provided at transversely spaced apart locations and are configured to direct radiation out of intermediate surface 240. Lip sensors 248L, 248C, 248R may be used to sense the force applied to inward portion 32A of lip 32 in a manner similar to lip sensor 48 of device 12. Lip sensors 248L, 248C, 248R may be also able to sense other characteristics of a user's lower lip 32 as discussed in more detail below. [0093] Figure 6 is a schematic block diagram of lip sensing system 211 of the Figure 3 input device 212 according to a particular embodiment of the invention. Each of lip sensors 248L, 248C, 248R may be implemented in a manner similar to, and have characteristics similar to, lip sensor 48 of device 12 described above. [0094] Lip sensors 248L, 248C, 248R may optionally be controlled by signals 207L, 207C, 207R from instrument electronics 216 and/or by signals 215L, 215C, 215R from signal conditioning electronics 250L, 250C, 250R. Instrument electronics 216 and signal conditioning electronics 250L, 250C, 250R may comprise one or more controllers suitably configured for this purpose. Signals 207L, 207C, 207R, 215L, 215C, 215R may comprise one way or two way control signals.
[0095] As shown in Figure 6, lip sensors 248L, 248C, 248R sense characteristics of the user's lower lip 32 and create corresponding lip sensor output signals 276L, 276C, 276R representative of these characteristics. Lip sensor output signals 276L, 276C, 276R may be correlated to the force that the user applies to their lower lip 32, the position of lower lip 32 with respect to device 212, the shape of lower lip 32 (e.g. the curvature of lip 32 or the tilt angle of lip 32) and other characteristics of lower lip 32. Lip sensor output signals 276L, 276C, 276R may be processed by their respective signal conditioning circuits 250L, 250C, 250R and/or by instrument electronics 216 to generate one or more processed lip signals 277. [0096] Instrument electronics 216 may use one of more processed lip signals 277 to generate output signal 218 on the basis of these characteristics. In the illustrated embodiment, lip sensor output signals 276L, 276C, 276R are used to generate three processed lip signals 277A, 277B, 277C, each of which may be used independently by instrument electronics 216 in the generation of output audio signal 218. In other embodiments, lip sensor output signals 276L, 276C, 276R are used to generate a different number of processed lip signal(s) 277. Output signal 218 in turn controls the audio output of synthesizer 13 (Figure 1).
[0097] To generate processed lip signals 277 A, 277B, 277C, lip sensor output signals 276L, 276C, 276R may be processed in the analog and/or digital domain by their corresponding signal conditioning electronics 250L, 250C, 250R and/or instrument electronics 216. Signal conditioning electronics 250L, 250C, 250R and instrument electronics 216 may have features similar to signal conditioning electronics 50 and instrument electronics 16 described above. The processing performed on lip sensor output signals 276L, 276C, 276R by signal conditioning electronics 250L, 250C, 250R and/or instrument electronics 216 may involve any of a variety of analog and digital signal processing techniques similar to those described above for oral cavity sensor output signals 264', 264". In particular, lip sensor output signals 276L, 276C, 276R may be normalized using a circuit similar to circuit 61 (Figure 14) as a part of any processing performed by signal conditioning electronics 250L, 250C, 250R and/or instrument electronics 216. For the discussion that follows, it is assumed that lip sensor output signals 276L, 276C, 276R are normalized.
[0098] In general, processed lip signals 277 A, 277B, 277C may be any function & of one or more of lip sensor output signals 276L, 276C, 276R. An example of a processed lip signal 277 A, 277B, 277C is a signal representative of overall force applied to inward portion 32A of lip 32. The sensing of lip-force for a mouth- operated input device 12 incorporating a single lip sensor 48 was discussed above (Figure 5). Increasing the force applied to inward portion 32A of lower lip 32 causes
increasing reflection detected by lip sensor 48 and corresponding increases in lip sensor output signal 76. In mouth-operated input device 212, increasing the force applied to inward portion 32A of lower lip 32 causes increasing reflection detected by each of lip sensors 248L, 248C, 248R and corresponding increases in lip sensor output signals 276L, 276C, 276R.
[0099] A processed lip signal 277 representative of overall force applied to inward portion 32A of lip 32 may be an additive combination of lip sensor output signals 276L, 276C, 276R. Such an additive combination may have the form: <^~=aL+bC+cR+d, where L, C, R respectively represent the lip sensor output signals 276L, 276C, 276R; a, b and c represent scaling/weighting coefficients; and d represents an offset parameter. The coefficients a, b, c, d may be selected to provide function & with a suitable dynamic range. The coefficients may also be selected to minimize zero offset. The signals L, C, R may be filtered, linearized (e.g. using a look up table), scaled and/or offset prior to using them in an additive combination of this nature. Similarly, the output of the function & may be filtered, linearized, scaled and/or offset. Those skilled in the art will appreciate that subsets of lip sensor output signals 276L, 276C, 276R could also be used in a similar manner to generate a plurality of processed lip signals 277 A, 277B, 277C representative of localized lip- force. [0100] Another example of a processed lip signal 277 is a signal representative of the curvature of inward portion 32A of lip 32. In general, the force applied to, and position of, inward portion 32A of lip 32 may be measured by a plurality of sensors 248 which are configured to measure certain parts of the lip and to generate corresponding lip sensor output signals 276 and the lip sensor output signals 276 may be combined to provide a processed lip signal 277 which varies with the curvature of inward portion 32A of lip 32. Figures 7A, 7B and 7C are schematic drawings which respectively represent flat lip curvature, negative lip curvature and positive lip curvature. In one particular embodiment, a processed lip signal 277 representative of curvature may have the form ^=(aL+bR)/2-cC +d, where L, C, R, respectively represent the lip sensor output signals 276L, 276C, 276R; a, b and c represent weighting/scaling coefficients; and d represents an offset parameter. The coefficients a, b, c, d may be selected to provide function &" with a suitable dynamic range. The coefficients may also be selected to minimize zero offset. [0101] If all of the lip sensor output signals are given equal weight and the offset parameter d is zero, then the condition (L+R)/2=C can be defined to represent the flat curvature of Figure 7A, the condition (L+R)/2 < C can be defined to represent the negative curvature of Figure 7B and the condition (L+R)/2 > C can be defined to represent the positive curvature of Figure 7C. The signals L, R, C may be filtered,
linearized (e.g. using a look up table), scaled and/or offset prior to using them in a curvature function of this nature. Similarly, the output of the function ^" may be filtered, linearized, scaled and/or offset.
[0102] Yet another example of a processed lip signal 277 is a signal representative of the angle of inward portion 32A of lip 32. Figures 8A, 8B and 8C are schematic drawings which respectively represent flat lip angle, negative lip angle and positive lip angle. In one particular embodiment, a processed lip signal 277 representative of the angle of inward portion 32A of lip 32 may have the form ^"=c(aL-bR)+d, where L and R, respectively represent the Hp sensor output signals 276L, 276R; a, b and c represent weighting/scaling coefficients; and d represents an offset parameter. The coefficients a, b, c, d may be selected to provide function & with a suitable dynamic range. The coefficients may also be selected to minimize zero offset. [0103] If lip sensor output signals 276L and 276R are given equal weight and the offset parameter d is zero, then the condition L=R can be defined to represent the flat angle of inward portion 32A of lip 32 (Figure 8A), the condition L > R can be defined to represent the negative angle of inward portion 32A of lip 32 (Figure 8B) and the condition L < R can be defined to represent the positive angle of inward portion 32A of lip 32 (Figure 8C). The signals L and R may be filtered, linearized (e.g. using a look up table), scaled and/or offset prior to using them in a lip angle function of this nature. Similarly, the output of the function & may be filtered, linearized, scaled and/or offset. Those skilled in the art will appreciate that subsets of lip sensor output signals 276L, 276C, 276R could also be used in a similar manner to generate one or more processed lip-position signals 277 A, 277B, 277C representative of localized lip angle. [0104] The above-described force, curvature and angle functions are merely examples of the types of functions which may be used to generate processed lip signals 277 from lip sensor output signals 276L, 276C, 276R. Processed lip signals 277 may generally comprise any function &~ of one or more of lip sensor output signals 276L, 276C, 276R. Individual processed lip signals 277 A, 277B, 277C may be defined such that they can be substantially independently controlled by a skilled user. For example, where one of processed lip signals 277 A, 277B, 277C relates to the force applied by inward portion 32A of lower lip 32 and another one of processed lip signals 277 A, 277B, 277C relates to the angle of inward portion 32 A of lower lip 32, then a user who becomes adept at using input device 212 may be able to substantially independently control the force that they apply to inward portion 32 A of their lower lip 32 and the angle of inward portion 32A of lower lip 32, so as to use these characteristics to independently control processed lip signals 211 A, 277B, 277C and to thereby independently influence the generation of output signal 218.
[0105] Instrument electronics 216 may use processed lip signals 211 A, 277B, 277C to generate output signal 218 which in turn controls the audio output of synthesizer 13. As discussed above, lip sensor output signals 276L, 276C, 276R may be processed to generate a different number of processed lip signals 277, each of which may (alone or in combination with other processed lip signals 277) be related to sensed spatial characteristics of the user's lower lip 32. The provision of multiple lip sensors 248L, 248C, 248R allows for a model of the user's lip that has multiple degrees of freedom (i.e. multiple processed lip signals 277 that may relate to a corresponding plurality of physical gestures/characteristics). Processed lips signals 277 may also be related to the rate of change of such sensed spatial characteristics of the user's lower lip 32.
[0106] Figure 15 is a partial isometric view of a mouth-operated input device 312 according to another embodiment of the invention. In many respects, mouth-operated input device 312 of Figure 15 is similar to mouth-operated input device 12 (Figure 1) and mouth-operated input device 212 (Figure 3). Features of device 312 which are similar to features of device 12 and device 212 are referred to using similar reference numerals that are preceded by the digit "3". For example, oral cavity sensing system 309 of mouth-operated input device 312 is substantially similar to oral cavity sensing system 209 of mouth-operated input device 212. [0107] One aspect of device 312 that differs from devices 12, 212 is that device 312 incorporates a transmissive optical sensor 348 for detecting characteristic(s) of inward portion 32A of a user's lower lip 32. Transmissive optical lip sensor 348 is depicted schematically in Figure 16. For clarity, some details of device 312 not related to sensing characteristic(s) of inward portion 32A of a user's lower lip 32 have been removed from Figure 16.
[0108] Lip sensor 348 is an optical transmission-type sensor which emits radiation and senses the intensity of transmitted radiation detected at a remote location. Lip sensor 348 comprises a radiation emitter 370A which may be substantially similar to the radiation emitters described above. Lip sensor 348 also comprises a radiation detector 370B for sensing transmitted radiation which may be substantially similar to the radiation detectors described above. Detector 370B is spaced apart from emitter 370A. As shown in Figure 15, emitter 370A may be located on a first transverse side of region 333 (i.e. between convexity 337 and concavity 339 and between lower portion 336A and upper portion 336B of lower surface 336). Although not shown explicitly in the Figure 15 view, detector 370B may be located on a transversely opposing second side of region 333 (i.e. so as to detect radiation that is transmitted through region 333 by emitter 370A). In the illustrated embodiment, emitter 370A is located on lobe 34 IA on one transverse side of region 333 and detector 370B is
located on lobe 34 IB on the opposing transverse side of region 333, although other configurations are possible.
[0109] In some embodiments, emitter 370A comprises, or is optically coupled to, a light pipe 366 A and detector 370B comprises, or is optically coupled to, a light pipe 366B, such that emitter 370A and/or detector 370B may be located remotely (i.e. away from mouth-engaging end 322 of device 312). As shown in Figure 16, additional optics 368A may be provided for shaping and/or directing the radiation emitted from emitter 370A and/or light pipe 366A and additional optics 368B may be provided for receiving radiation at remotely located detector 370B and/or light pipe 366B. Such optics may be similar to the coupling optics described above.
[0110] Lip sensor 348 (including emitter 370A and/or detector 370B) may optionally be controlled by signal 307 from instrument electronics 316 and/or by signals 315 from signal conditioning electronics 350. Instrument electronics 316 and signal conditioning electronics 350 may comprise one or more controllers suitably configured for this purpose. Signals 307, 315 may comprise one way or two way control signals.
[0111] Lip sensor 348 detects the force that a user is applying to the inward portion 32 A of lip 32. When a user applies a relatively small amount of force to the inward portion 32 A of their lower lip 32, the deformation of inward portion 32A around convexity 337 and into region 333 will be relatively low (see Figure 5A which schematically depicts this situation). Consequently, a relatively large amount of the radiation emitted by emitter 370A will be detected by detector 370B on the transversely opposing side of region 333 and lip sensor output signal 376 will be relatively high. [0112] As the user applies an intermediate level of force to the inward portion 32A of their lower lip 32, the deformation of inward portion 32A around convexity 337 and into region 333 will be moderate (see Figure 5B which schematically depicts this situation). Under these conditions, inward portion 32A of lip 32 will intercept some of the radiation from emitter 370A before such radiation crosses region 333 to reach detector 370B. As a result, the amount of radiation detected by detector 370B will be at an intermediate level and lip sensor output signal 376 will be at an intermediate level (i.e. less than the low force situation described above). [0113] As the user applies a relatively high level of force to the inward portion 32 A of their lower lip 32, the deformation of inward portion 32A around convexity 337 and into region 333 will be relatively high (see Figure 5C which schematically depicts this situation). Under these conditions, inward portion 32A of lip 32 will intercept a significant amount of the radiation from emitter 370A before such radiation crosses region 333 to reach detector 370B. As a result, the amount of
radiation detected by detector 370B will be at a relatively low level and lip sensor output signal 376 will be at a relatively low level (i.e. less than the low force and intermediate force situations described above).
[0114] It will be appreciated from the above that lip sensor 348 creates a lip sensor output signal 376 that is inversely representative of the force applied to inward portion 32A of lip 32. As shown best in Figure 15, lobes 341A, 341B, upper portion 336B of lower surface 336 and intermediate surface 340 enclose region 333 and thereby improve the dynamic range of transmissive sensor 348. In a manner similar to that described above, lip sensor output signal 376 may be used by signal conditioning electronics 350 and/or instrument electronics 316 to generate one or more processed lip signals 377. Instrument electronics 316 may then use processed lip signals 377 to generate output signal 318. Output signal 318 may in turn be used to control the audio output of synthesizer 13 (Figure 1). [0115] In other respects, lip sensing system 311 of device 312 may be similar to, or may be varied in manners similar to, the lip sensing systems 11, 211 discussed above. For example, the size of region 333 (i.e. the distance between upper portion 336B and lower portion 336A) and/or other parameters of lip sensor 348 may be designed so that lip sensor output signal 376 is a monotonically changing function of the force applied by inward portion 32A of lip 32 to mouth-engagement end 322 of device 312. As another example, lip sensor output signal 376 may be processed by signal conditioning electronics 350 and/or instrument electronics 316 to output a plurality of processed lip signals 377 and such processed lip signal(s) may generally be any function & of lip sensor signal 376. [0116] In some embodiments, mouth operated input device 312 may make use of a plurality of transmissive sensors for detecting various characteristics related to the inward portion 32 A of a user's lip 32. Such a plurality of transmissive sensors may be used in a manner similar to the three lip sensors 248L, 248C, 248R of device 212. [0117] Figures 19-23 schematically depict a mouth-operated input device 512 according to another embodiment of the invention. In many respects, mouth-operated input device 512 is similar to mouth-operated input device 12 (Figure 1) and mouth- operated input device 212 (Figure 3). Mouth-operated input device 512 differs from mouth-operated input devices 12 and 212 in that mouth-operated input device 512 incorporates a non-optical lip-sensing system 513. In one particular embodiment, lip- sensing system 513 of mouth-operated input device 512 is based on sensing lip force using one or more force-sensitive resistors (FSRs).
[0118] Mouth-operated input device 512 incorporates a lip-sensing system 513. In the illustrated embodiment, lip-sensing system 513 is configured to sense various characteristics of a user's lower lip. This is not necessary. Mouth-operated input
device 512 could be modified to incorporate features similar to those of lip-sensing system 513 for sensing the characteristics of the user's upper lip in addition to or as an alternative to sensing the characteristics of the user's lower lip. [0119] Lip-sensing system 513 includes a mouthpiece 514 which can be placed at least partially in the user's mouth. In the illustrated embodiment, where lip-sensing system 513 is designed for detecting characteristics of the user's lower lip, the user's lower lip (not shown) may come into contact with lower surface 517 of mouthpiece 514 when the mouthpiece 514 is partially inserted into the user's mouth. The user's upper lip and/or upper teeth (not shown) may rest on upper surface 518 of mouthpiece 514. In the illustrated embodiment, where mouth-operated input device 512 is used as a part of an instrument simulator, the angle between lower surface 517 and upper surface 518 of mouthpiece 514 may be in a range of 0°-50° . In particular embodiments, this angular range may be 15° -45°. [0120] Lip-sensing system 513 incorporates a plurality of lip force sensors 548 A, 548B, 548C (collectively, lip force sensors 548) which are configured (as described in more detail below) to detect the localized force applied to various regions of lower surface 517 by various portions of the user's lower lip. Each of lip force sensors 548 is capable of generating a corresponding lip force sensor output signal 576A, 576B, 576C (collectively, lip force sensor signals 576) which is representative of the force detected by that particular lip force sensor 548.
[0121] In the illustrated embodiment, lip force sensors 548 are implemented by force-sensing resistors (FSRs) 549 A, 549B, 549C (collectively, FSRs 549). By way of non-limiting example, lip force sensors 548 may be implemented using the FSRs sold by Interlink Electronics, Inc. of Camarillo, California under part number 400 or similar FSRs sold by Tekscan, Inc. of South Boston, Massachusetts. A single FSR 549 is shown in Figure 21. FSRs 549 have a resistance which varies with the force/pressure applied to their active regions 526. In the illustrated embodiment, active regions 526 of FSRs 549 are generally circular in cross-section. In particular embodiments, active region 526 is dimensioned to be less than 10 mm across (e.g. in diameter). FSRs 549 have tail portions 527 which are mounted on circuit board 532 and which are suitably connected to provide corresponding lip force sensor output signals 576. By way of non-limiting example, FSRs 549 may be connected as part of a voltage divider circuit. [0122] In the illustrated embodiment, the active regions 526A, 526B, 526C of FSRs 549A, 549B, 549C are located at spaced apart locations within mouthpiece 514. In the illustrated embodiment, lip-sensing system comprises three FSRs 549 wherein FSRs 549A, 549C have active regions 526 A, 526C which are spaced apart from one another along a transverse dimension (see arrow 528 of Figure 20) and FSR 549B
has an active region 526B which is spaced apart from active regions 526 A, 526C along a longitudinal dimension (see arrow 530 of Figure 20). Longitudinal dimension 530 and transverse dimension 528 may be generally orthogonal to one another. As explained in more detail below, this pattern of active regions 526 of FSRs 549 allows measurement of various characteristics of the user's lower lip.
[0123] FSRs 549 have a resistance that may vary with applied force. FSRs 549 typically have a threshold activation pressure (i.e. a threshold activation force per unit area). When FSRs 549 experience forces per unit area that are below their activation pressure, then their resistance does not vary with the applied force (e.g. FSR 549 may represent an open circuit). This threshold activation pressure may be in a range of .005-.125 N/mm2, for example. The forces that can be applied by most users' lips are relatively small and may not be able to overcome the threshold activation pressure of typical FSRs. Additionally, in some applications, such as instrument simulation, it may be desirable to have lip-sensing system 513 be responsive to a range of lip forces which may be below the threshold activation pressure of typical FSRs.
[0124] Mouthpiece 514 may be designed to overcome this characteristic of FSRs 549 by configuring mouthpiece 514 to "pre-load" active regions 526 of FSRs 549 and/or to concentrate applied lip force onto a relatively small area of active regions 526 of FSRs 549. Mouthpiece 514 is shown in more detail in Figures 22A, 22B, 23A and 23B. In the illustrated embodiment, mouthpiece 514 comprises an external mouthpiece component 534 (Figures 22A, 22B) and an internal mouthpiece component 536 (Figures 23 A, 23B). Internal mouthpiece component 536 may receive a portion of circuit board 532 in its cavity 538. Internal mouthpiece component 536 may be fabricated from a material that is opaque. Such opacity may assist with the operation of optical sensors which may be part of mouth-operated input device 512 and which may be used to sense other characteristics of the user's mouth (e.g. the optical sensors used to sense characteristics of the user's oral cavity). [0125] External mouthpiece component 534 includes lower surface 517 and upper surface 518 described above and receives internal mouthpiece component 536 in its cavity 540 (Figure 22B). In particular embodiments, external mouthpiece component 534 may be fabricated in whole, or in suitable part(s), from an elastomeric material. This elastomeric material may be used to seal mouthpiece 514 and to transfer lip force to FSRs 549. In some embodiments, the elastomeric material has a hardness rating in a range of Shore A 5 to Shore A 120. In some embodiments, external mouthpiece component 534 is relatively more deformable than internal mouthpiece component 536. Internal mouthpiece component 536 may support portions of external mouthpiece component 534. In some embodiments, external mouthpiece
component 534 is transparent or translucent (i.e. facilitates transmission of light therethrough). This characteristic of external mouthpiece component 534 is useful for the operation of optical sensors which may be part of mouth-operated input device 512 and which may be used to sense other characteristics of the user's mouth (e.g. the optical sensors used to sense characteristics of the user's oral cavity).
[0126] In the illustrated embodiment, external mouthpiece component 34 comprises a plurality of protrusions 547 A, 547B, 547C (collectively, protrusions 547) which extend from an interior surface 542 of external mouthpiece component 534 and into cavity 540. Protrusions 547 may be located at locations corresponding to active regions 526 of FSRs 549 so as to transfer force from various portions of the users lower lip, through lower surface 517 and to FSRs 549. Protrusions 547 may also be shaped and/or sized to pre-load FSRs 549 and/or to concentrate force onto relatively small areas of active regions 526 of FSRs 549. In particular, protrusions 547 may be sized such that when circuit board 532 is inserted into mouthpiece 514, protrusions 547 interact with (i.e. contact) corresponding active regions 526 of FSRs 549 to create force therebetween. Such forces may cause deformation of protrusions 547 (which may be elastomeric) and/or FSRs 549. The forces between protrusions 547 and active regions 526 of FSRs 549 pre-load FSRs 549 with forces that may be at or near the activation threshold pressure of FSRs 549. In particular embodiments, the pre-load force per unit area between protrusions 547 and active regions 526 is in a range of 50% -150% of the threshold activation pressure for FSRs 524. In some embodiments, this range is 80% -120% .
[0127] Protrusions 547 may additionally or alternatively be shaped such that the areas of protrusions 547 that contact corresponding active regions 526 of FSRs 549 represent a relatively small portion of the area of the active regions 526. In this manner, protrusions 547 can concentrate lower lip force from a relatively large area onto a relatively small area of active regions 526, thereby increasing the pressure. In the illustrated embodiment, protrusions 547 are frustro-conical in shape to achieve this effect. In other embodiments, protrusions 547 may have other shapes where the distal regions of protrusions 547 have smaller cross-sectional areas than the regions of protrusions 547 that are relatively close to interior surface 542. In some embodiments, the areas of protrusions 547 that contact corresponding active regions 526 of FSRs 549 are in a range of 10% -75% of the area of active regions 526. In some embodiments, this range is 20% -50% . It should be noted that when protrusions 547 are formed from elastomeric material, the area of protrusions 547 in contact with active regions 526 may increase under application of larger amounts of external force.
[0128] As discussed above, lip force sensor signals 576 may be processed by suitable signal conditioning electronics 550 and/or instrument electronics 516. Signal conditioning electronics 550 and instrument electronics 516 may be similar to (and include components similar to) signal conditioning electronics 50, 250L, 250R, 250C, 350 and instrument electronics 16, 216, 316 of the other mouth-operated input devices 12, 212, 312 described herein. In the illustrated embodiment, signal conditioning electronics 550 process lip force sensor signals 576 to provide processed lip signals 577A, 577B, 577C (collectively processed lip signals 577) similar to processed lip signals 77, 277A, 277B, 277C, 377 of devices 12, 212, 312 and instrument electronics 516 uses processed lip signals 577 to generate output signal 18 similar to output signals 18, 218, 318 of devices 12, 212, 312. These processed lip signals 577 and/or output signal 18 may be used to estimate a number of characteristics of the user's lower lip. [0129] Although not expressly shown, lip force sensors 548 may be individually controlled (e.g. gated or the like) by corresponding signals from signal conditioning electronics 550 and/or instrument electronics 516. These control signals may be similar to control signals 215L, 215C, 215R and/or control signals 207L, 207C, 207R described above for mouth-operated input device 212. [0130] As discussed above, lip force sensor signals 576A, 576B, 576C are respectively representative of the force applied by the user's lower lip in the regions of lower surface 517 corresponding to the locations of active regions 526A, 256B, 526C. This allows various functions (e.g. linear combinations) of lip force sensor signals 576 (or processed lip signals 577) to be used to estimate a number of different characteristics of the users' lower lip which may in turn be used to control an audio synthesizer or some other electronic device (see synthesizer 13 shown in Figure 1, for example). In the illustrated embodiment, lip sensor output signals 576 are used to generate three processed lip signals 577, each of which may be used independently by instrument electronics 516 to generate output signal 518. In other embodiments, lip sensor output signals 576 are used to generate a different number of processed lip signals 577.
[0131] In general, processed lip signals 577 may be any function ^" of one or more of Hp sensor signals 576A, 576B, 576C. An example of a processed lip signal 577 is a signal representative of overall lip force applied to lower surface 517 of mouthpiece 514. Increasing the overall Hp force on lower surface 517 may cause corresponding changes in the resistances of each of FSRs 549A, 549B, 549C and corresponding changes to each of the Hp sensor signals 576.
[0132] A processed Hp signal 577 representative of overall force applied by the user's Hp to lower surface 517 may be an additive combination of Hp sensor signals
576 A, 576B, 576C. Such an additive combination may have the form: ^=aA+bB+cC+d, where A, B, C respectively represent the lip sensor output signals 576A, 576B, 576C; a, b and c represent scaling/weighting coefficients; and d represents an offset parameter. The coefficients a, b, c, d may be selected to provide function & with a suitable dynamic range. The coefficients may also be selected to minimize zero offset. The signals A, B, C may be filtered, linearized (e.g. using a look up table), scaled and/or offset prior to using them in an additive combination of this nature. Similarly, the output of the function ^" may be filtered, linearized, scaled and/or offset. Those skilled in the art will appreciate that subsets of lip sensor signals 576 A, 576B, 576C could also be used in a similar manner to generate a plurality of processed lip signals 577A, 577B, 577C representative of localized lip-force. [0133] Another example of a processed lip signal 577 is a signal representative of the angle of the user's lower lip - see Figures 8A, 8B and 8C for schematic drawings which respectively represent flat lip angle, negative lip angle and positive lip angle. In one particular embodiment, a processed lip signal 577 representative of the angle of the user's lower lip may have the form ^=c(aA-bC)+d, where A and C, respectively represent the lip sensor signals 576A, 576C; a, b and c represent weighting/scaling coefficients; and d represents an offset parameter. The coefficients a, b, c, d may be selected to provide function & with a suitable dynamic range. The coefficients may also be selected to minimize zero offset.
[0134] If lip sensor signals 576A and 576C are given equal weight and the offset parameter d is zero, then the condition A-C can be defined to represent the flat angle of the user's lower lip (Figure 8A), the condition A > C can be defined to represent the negative angle of the user's lower lip (Figure 8B) and the condition A < C can be defined to represent the positive angle of the user's lower lip (Figure 8C). The signals A and C may be filtered, linearized (e.g. using a look up table), scaled and/or offset prior to using them in a lip angle function of this nature. Similarly, the output of the function ST may be filtered, linearized, scaled and/or offset. Those skilled in the art will appreciate that subsets of lip sensor signals 576A, 576B, 576C could also be used in a similar manner to generate one or more processed lip signals 577A, 577B, 577C representative of localized lip angle.
[0135] Another example of a processed lip signal 577 is a signal representative of the front to back position (i.e. along longitudinal direction 30) of the user's lower lip. In one particular embodiment, a processed lip signal 577 representative of the front to back position of the user's lower lip may have the form ^"=(aA+cC)/2-bB+d, where A, B and C, respectively represent the lip sensor signals 576A, 576B, 576C; a, b and c represent weighting/scaling coefficients; and d represents an offset parameter. The coefficients a, b, c, d may be selected to provide function & with a
suitable dynamic range. The coefficients may also be selected to minimize zero offset.
[0136] If lip sensor signals 576A, 576B and 576C are given equal weight and the offset parameter d is zero, then the condition (A + C)/2-B=0 can be defined to represent the condition where the user's lower lip is positioned at a midpoint between the front to back position of active regions 526A, 526C and the front to back position of active region 526B, the condition (A+C)/2-B <0 can mean that the user's lip is positioned relatively close to active region 526B (as compared to the midpoint) and the condition (A+C)/2-B > 0 can mean that the user's lip is positioned relatively close to active regions 526A, 526C (as compared to the midpoint). The signals A, B and C may be filtered, linearized (e.g. using a look up table), scaled and/or offset prior to using them in a lip position function of this nature. Similarly, the output of the function &~ may be filtered, linearized, scaled and/or offset. Those skilled in the art will appreciate that subsets of lip sensor signals 576A, 576B 576C could also be used in a similar manner to generate one or more processed lip signals 577A, 577B, 577C representative of localized lip front to back position.
[0137] In other embodiments, the active regions 526 A, 526B, 526C of FSRs 549A, 549B, 549C may be aligned along the same transverse axis (see transverse direction 528 of Figure X). In such embodiments, another example of a processed lip signal 577 is a signal representative of the transverse curvature of the user's lower lip.
Figures 7A, 7B and 7C are schematic drawings which respectively represent flat lip curvature, negative lip curvature and positive lip curvature. In one particular embodiment, a processed lip signal 577 representative of curvature may have the form ^"=(aA+cC)/2-bB +d, where A, B, C, respectively represent the lip sensor output signals 576A, 576B, 576C of the right, middle and left FSRs 549; a, b and c represent weighting/scaling coefficients; and d represents an offset parameter. The coefficients a, b, c, d may be selected to provide function & with a suitable dynamic range. The coefficients may also be selected to minimize zero offset. [0138] If left and right lip sensor signals 576A, 576C are given equal weight and the offset parameter d is zero, then the condition (A+C)/2=B can be defined to represent the flat curvature (see Figure 7A), the condition (A+C)/2 <B can be defined to represent the negative curvature (see Figure 7B) and the condition (A+C)/2>B can be defined to represent the positive curvature (see Figure 7C). The signals A, B, C may be filtered, linearized (e.g. using a look up table), scaled and/or offset prior to using them in a curvature function of this nature. Similarly, the output of the function &~ may be filtered, linearized, scaled and/or offset.
[0139] The above-described force, angle and curvature functions are merely examples of the types of functions which may be used to generate processed lip
signals 577 from lip sensor output signals 576A, 576B, 576C. Processed lip signals 577 may generally comprise any function &~ of one or more of lip sensor output signals 576A, 576B, 576C. Individual processed lip signals 577 A, 577B, 577C may be defined such that they can be substantially independently controlled by a skilled user. For example, where one of processed lip signals 577 A, 577B, 577C relates to the force applied by the user's lower lip and another one of processed lip signals 577 A, 577B, 577C relates to the angle of the user's lower lip, then a user who becomes adept at using mouth-controlled input device 512 may be able to substantially independently control the force that they apply to their lower lip and the angle of their lower lip, so as to use these characteristics to independently control processed lip signals 577 A, 577B, 577C.
[0140] The provision of multiple lip sensors 548A, 548B, 548C allows for a model of the user's lip that has multiple degrees of freedom (i.e. multiple processed lip signals 577 that may relate to a corresponding plurality of physical gestures/characteristics). Processed lips signals 577 may also be related to the rate of change of such sensed spatial characteristics of the user's lower lip. [0141] Other than for lip-sensing system 513, mouth-operated input device 512 may be similar to any of the other mouth-operated input devices described herein. By way of non-limiting example, mouth-operated input device 512 may comprise oral cavity sensing systems, breath pressure sensing systems and mouth force sensing systems similar to those of the other mouth-operating input devices described herein. In the illustrated embodiment, mouth-operated input device 512 is designed to be used as a part of a wind-instrument simulator, but mouth-operated input device 512 is not limited to use as instrument simulators. Mouth-operated input device 512 may be used for (or may be modified for use for) any of the other applications described herein.
[0142] Mouth-operated input device 512 may be altered or modified. By way of non- limiting example Other types of sensors could be used in the place of FSRs 549, such sensors include, by way of non-limiting example, strain gauges, fiber optic flex sensors, optical sensors and/or mechanical compression sensors using Hall effect devices. FSRs 549 represent a good choice because of their low cost, simplicity and stability with respect to temperature changes. Some embodiments make use of a custom-made array of FSRs rather than discrete FSR devices 549 described above. Mouth-operated input device 512 may be modified to provide a larger number of lip force sensors 548 or a different pattern of lip force sensors 548. More lip force sensors 548 or a different pattern of lip force sensors 548 may provide more or different degrees of freedom. It will be appreciated that there may be a limited to the number of degrees of freedom that a user could consciously control with their lips.
[0143] Mouth-operated input device 12 of Figure 1 also incorporates a sensor 42 for detecting the user's breath-pressure. Breath-pressure-sensor 42 is depicted in more detail in Figure 9. For clarity, details of device 12 which are not related to sensing the user's breath-pressure are not shown in Figure 9. As discussed above, the user's lips 26, 32 may contact mouth-engaging end 22 of device 12, such that the user may blow into mouth-engaging end 22. In the illustrated embodiment, proximal surface 24 of mouth-engaging end 22 has an aperture 80 which leads through conduit 44 to breath-pressure-sensor 42. [0144] Various types of breath-pressure-sensors are known in the art. Any suitable breath-pressure-sensor may be used with the device 12 of the illustrated embodiment. One example of a suitable breath-pressure-sensor 42 is a variable resistance pressure sensor of the type described in US patent No. 5,543,580 which is incorporated herein by reference. Other examples of sensors suitable for use as breath-pressure-sensor 42 include the MPXC2011 from Freescale Semiconductor, Inc. of Austin, Texas and the 26PC05SMT from Honeywell International, Inc. of Morris Township, New Jersey. [0145] When playing a conventional wind instrument, a musician frequently (and often subconsciously) changes their breath-pressure to modulate the sound emanating from the instrument. For example, by applying different levels of breath-pressure, a musician may alter the frequency, pitch, timbre or other characteristics of the instrument and thereby change the sound created by the instrument.
[0146] Mouth-operated input device 12 models this aspect of a conventional instrument. Breath-pressure-sensor 42 senses the user's breath-pressure and creates a breath-pressure-sensor output signal 82 representative of the user's breath-pressure. Breath-pressure-sensor output signal 82 may be used to generate one or more processed breath-pressure signals 84. Instrument electronics 16 may make use of processed breath-pressure signal(s) 84 in generating output signal 18. Output signal 18 in turn controls the audio output of synthesizer 13 (Figure 1). [0147] In the illustrated embodiment of Figure 9, conduit 44 and breath-pressure- sensor 42 are sealed, such that there is substantially no air flow between the user's oral cavity 52 and sensor 42. The total volume of conduit 44 may be less than 500 mm3. In some embodiments, the volume of conduit 44 is less than 200 mm3. These features of conduit 44 provide breath-pressure-sensor 42 with a transient response that has sufficiently high speed for most musical applications. In addition, these features minimize condensation from the user's breath and minimize the amount of saliva in conduit 44.
[0148] In the illustrated embodiment, breath-pressure-sensor output signal 82 may be processed by signal conditioning electronics 50 and/or instrument electronics 16 to output processed breath-pressure signal 84 in a manner similar to that of the other
sensor output signals discussed above. In other embodiments, breath-pressure-sensor output signal 82 may be processed by signal conditioning electronics 50 and/or instrument electronics 16 to output a plurality of processed breath-pressure signals 84. By way of non-limiting example, breath-pressure signals 84 may include one signal related to the measured breath-pressure and another signal related to the rate of change of the measured breath signal. Signal conditioning electronics 50 and instrument electronics 16 may function in a manner similar to, and have characteristics similar to, those discussed above with respect to oral cavity sensor output signal 64 and lip sensor output signal 76. [0149] In general, processed breath-pressure signal 84 may be any function 2F of breath-pressure-sensor signal 82. In one embodiment, processed breath-pressure signal 84 may have the form ^"=aP+b, where P represents breath-pressure-sensor signal 82; a represents a scaling coefficient; and b represents an offset coefficient. The coefficients a and b may be selected to provide processed breath-pressure signal 84 with a suitably high dynamic range. These coefficients may also be selected to minimize zero offset. The offset coefficient b may be selected such that processed breath-pressure signal 84 is a positive signal when the user is blowing into device 12 and a negative signal when the user is sucking on device 12. [0150] Instrument electronics 16 may make use of processed breath-pressure signal 84 to generate output signal 18 which in turn controls the audio output of synthesizer 13. Processed breath-pressure signal 84 is related to the sensed breath-pressure applied by the user. Accordingly, using mouth-operated input device 12, the audio output of synthesizer 13 may be modulated by the user's breath-pressure. [0151] Mouth-operated input device 212 of the Figure 3 embodiment incorporates a pair of apertures 280', 280" in proximal surface 224 for admitting the user's breath. Apertures 280', 280" are depicted schematically in Figure 10. Aperture 280' leads through a conduit 244 to a breath-pressure-sensor 242. Breath-pressure-sensor 242 may be substantially similar to breath-pressure-sensor 42 described above. Breath- pressure-sensor 242 may generate a breath-pressure-sensor signal 282, which may be processed in the analog and/or digital domain by signal conditioning electronics 250 and instrument electronics 216 to generate one or more processed breath-pressure signal(s) 284. The processing of breath-pressure-sensor signal 282 to generate processed breath-pressure signal(s) 284 may be similar to the processing of breath- pressure signal 82 described above. [0152] Aperture 280" leads through conduit 286 to a flow regulator 288. Flow regulator 288 may be used to give instrument 10 a realistic feel by allowing the user's breath to flow through the instrument. In some embodiments, flow regulator 288 is user adjustable. In one particular embodiment, flow regulator 288 comprises a
screw (not shown) which may be adjusted to various positions which restrict the flow of air through conduit 286. For example, the screw may be adjusted inwardly to a position where it occupies a large volume of space in conduit 286 and substantially restricts the flow through conduit 286 and the screw may be withdrawn to a position where it occupies only a small volume (or none) of conduit 286, such that the user's breath can flow through conduit 286. Conduit 286 may exhaust from device 212 at an exhaust port 290 external to the user's mouth. Exhaust port 290 (which is not shown in Figure 3) may be located on the side of device 212. [0153] Mouth-operated input device 312 (Figure 15) also comprises a pair of apertures 380', 380" which may be substantially similar to apertures 280', 280" of device 212. The breath pressure sensor and flow control regulator (not explicitly shown) of device 312 may be substantially similar to those of device 212. [0154] Mouth-operated input device 212 of the Figure 3 embodiment also comprises an optional fin 251 which projects downwardly from upper portion 236B of lower surface 236 between proximal surface 224 and intermediate surface 240. Fin 251 may serve to reduce the amount of radiation emitted from lip sensors 248L, 248C, 248R and reflected back to lip sensors 248L, 248C, 248R from the user's oral cavity 52 (i.e. as opposed to lip 32). Under low lip-force conditions, without fin 251, radiation emitted from lip sensors 248L, 248C, 248R may be reflected back from tongue 52 or other surfaces of oral cavity 52 and provide spurious results for lip sensors 248L, 248C, 248R.
[0155] In some embodiments, fin 251 may extend downwardly by an amount substantially similar to the distance between upper portion 236 A and lower portion 236B of lower surface 236. In the illustrated embodiment, fin 251 is in the output path of radiation emitted from lip sensors 248L, 248C, 248R. In some embodiments, fin 251 is shaped, such that most of the radiation incident on fin 251 from lip sensors 248L, 248C, 248R is reflected away from (and not spuriously detected by) lip sensors 248L, 248C, 248R. In one particular embodiment, fin 251 is shaped such that its surface 257 facing intermediate surface 240 is skewed by greater than 30° with respect to intermediate surface 240 (or forms an angle greater than 120° with respect to the longitudinal axis 221 of input device 212). In other embodiments, fin 251 is shaped such that surface 257 forms an angle greater than 120° with the plane of incidence of radiation from lip sensors 248L, 248C, 248R. In some embodiments, fin 251 comprises (or is coated, at least on surface 257, with) a material that has a low reflectivity at the wavelength of the radiation used by lip sensors 248L, 248C, 248R. In one particular embodiment, surface 257 of fin 251 has a reflectivity of less than 30% at the wavelength used by lip sensors 248L, 248C, 248R.
[0156] Fin 251 may be designed such that at any lip-force, the radiation emitted from Hp sensors 248L, 248C, 248R and reflected from fin 251 is significantly less than the radiation reflected from lip 32. By way of non-limiting example, this reflectivity difference may be effected by using a sensor with a suitable wavelength, by varying the shape or orientation of fin 251 and/or by varying the material on surface 257 of fin 251.
[0157] Mouth-operated input devices 12, 212, 312, 512 (or instruments 10 incorporating such mouth-operated input devices) may also incorporate one or more outputs 92 (Figure 1). An output 92 may comprise any suitable output device for providing information to a user, such as, without limitation, an LED, an alphanumeric display, a graphical display, an audio output device and a tactile output device. Outputs 92 may indicate when a particular tone is flat (too low), sharp (too high) or on pitch. The determination of whether a note is flat, sharp or on pitch may be made by instrument electronics 16, 216, 316, 516 which may compare the output of synthesizer 13 (as measured by an acoustic sensor (not shown)) with a predetermined value that is defined as being "on-pitch". The predetermined on-pitch value may be determined during calibration of device 12, 212, 312, 512 and/or the acoustic sensor, for example. Outputs 92 may also be used to indicate the direction to a nearest harmonic. [0158] Outputs 92 may also be used to train musicians. For example, to train a musician to apply a particular gesture, an ideal gesture may be graphically displayed on an output 92 alongside a graphical representation of the user's current gesture (as measured by the various sensors of device 12, 212, 312, 512). By comparing the graphical gestures, the user can refine their gesture to be closer to that of the ideal. [0159] Mouth-operated input devices 12, 212, 312, 512 (or instruments 10 and systems incorporating such devices) may also comprise one or more hand-operated inputs 14. In the Figure 1 embodiment, hand-operated inputs 14 comprise buttons, but hand-operated inputs 14 may generally comprise any suitable types of input mechanisms, such as multi-position switches, alphanumeric keys, sliders, knobs, joysticks or the like. Hand-operated inputs 14 may be used to provide ON/OFF functions for device 12, 212, 312, 512 or for various applications, such as pitch- sensing and the like. Hand-operated inputs 14 may also be used for note selection or to otherwise alter the tone of signal 18. [0160] Figure 11 depicts a schematic block diagram of an instrument 10 according to a particular embodiment of the invention. Instrument 10 incorporates a mouth- operated input device 12, which may generally comprise any of the mouth-operated input devices described herein (e.g. devices 12, 212, 312, 512). In the Figure 11 embodiment mouth-operated input device 12 incorporates a number of sensors,
including breath-pressure-sensor 42, oral cavity sensor 46 and lip sensor 48, together with their associated signal conditioning electronics 50. The conditioned signals from sensors 42, 46, 48 are provided to instrument electronics 16. In the Figure 11 embodiment, instrument electronics 16 comprise an embedded processor 29A. Processor 29A may be a programmable microprocessor which has access to a memory 29B for storage of program instructions and other useful data. [0161] Instrument electronics 16 also receive input signals from inputs 14. Inputs 14 originate from mouth-operated input device 12 and/or instrument 10. In addition, as shown in Figure 11, instrument 10 may receive inputs 14 from external sources. In the Figure 11 embodiment, instrument 10 comprises a single output 92, but, in general, instrument 10 may comprise a plurality of outputs 92. As with inputs 14, outputs(s) 92 may be provided as a part of mouth-operated input device 12, instrument 10 and/or external output device(s). [0162] Instrument electronics 16 output an output signal 18 for controlling synthesizer 13. As discussed above, output signal 18 may conform to a known audio protocol. In the Figure 11 embodiment, instrument electronics 16 also receive control signal(s) 69 from synthesizer 13. Control signal(s) 69 may generally be used for any purpose. By way of non-limiting example, control signal(s) 69 could be used to configure instrument electronics 16. The configuration parameters of instrument electronics 16 may be different when instrument 10 is used to simulate a brass instrument as compared to when instrument 10 is used to simulate a woodwind instrument, for example. Those skilled in the art will appreciate that there are a wide variety of other uses for control signal(s) 69, such as tonal adjustment, sensor sampling rate adjustment and the like. [0163] Figure 12 is a schematic block diagram of an instrument 10' according to another embodiment of the invention. Instrument 10' incorporates a mouth-operated input device 12, which may generally comprise any of the mouth-operated input devices described herein (e.g. devices 12, 212, 312, 512). In the Figure 12 embodiment, a portion 16A of instrument electronics 16 is embedded in instrument 10' and another portion 16B of instrument electronics 16 is hosted on a computer 27. In some embodiments, portion 16A of instrument electronics 16 is embedded in mouth-operated input device 12. A wired or wireless communication link 67 is provided between portions 16A, 16B of instrument electronics 16. Communication link 67 may be implemented using any of a variety of suitable protocols known to those skilled in the art. By way of non-limiting example, communication link 67 may be implemented in accordance with the MIDI, USB or OSC protocols. In the Figure 12 embodiment, computer 27 (which comprises portion 16B of instrument electronics 16, a microprocessor 29 A and memory 29B) controls output device 92 and
communicates with synthesizer 13 via output signal 18 and control signal 69. While not shown explicitly in Figure 12, portion 16A of instrument electronics 16 may also comprise a suitable processor and/or suitable memory.
[0164] In the Figure 12 embodiment, output 92 is an external output device which is controlled by portion 16B of instrument electronics 16. Although not explicitly shown in Figure 12, instrument 10' may generally comprise any suitable number of outputs 92 which may be provided as a part of mouth-operated input device 12, instrument 10', computer 27 and/or external output device(s). Such output(s) may additionally or alternatively be controlled by portion 16A of instrument electronics 16.
[0165] In other respects, the Figure 12 instrument 10' is similar to the Figure 11 instrument 10.
[0166] Mouth-operated input devices 12, 212, 312, 512 are described above as having application to an instrument, where devices 12, 212, 312, 512 sense characteristics of the user's mouth (e.g. characteristics associated with oral cavity, tongue position, tongue shape, lip-position and breath-pressure) and generate corresponding sensor output signals which are processed and then ultimately used to control a synthesizer 13. Mouth-operated input devices 12, 212, 312, 512 described herein may be used for other applications, such as to enable disabled individuals to control various systems with their mouths or to enable people whose hands are otherwise encumbered (e.g. people in protective suits, such as space suits or underwater suits) to control various systems. Different characteristics of the way in which a user interacts with the mouth-operated input device (e.g. changes in oral cavity, tongue position, tongue shape, lip-position and breath-pressure) may be used to control different system parameters. More particularly, the output signals of mouth-operated input devices 12, 212, 312, 512 may be processed to generate one or more output signals 18 which ultimately control different system parameters. For example, a battery-operated wheel chair may be controlled using lip-force and tongue position. If the lip-force is greater, the mouth-operated input device may output a first output signal 18 which causes the chair's motor to move faster and if the lip- force is lower, the mouth-operated input device may change the level of the first output signal 18 such that the chair's motor may move more slowly. If the tongue position is more leftward, then the mouth-operated input device may output a second output signal 18 which directs the chair's motor to turn the chair in a leftward direction and if the tongue position is more rightward, then the second output signal 18 may directed the chair's motor to turn the chair in a rightward direction. For such applications, it may be advantageous to position the mouth-operated input device in close proximity to the user's mouth using a bracket or the like.
[0167] There are a wide variety of ways to map user gestures and the corresponding mouthpiece signals to control signals which may be used to control various systems. By way of non-limiting example:
• the position of the forward-facing surface a user's tongue could be used to control the X-Y position of a mouse pointer or a joystick in a computer user interface and a puff of breath or a squeeze of a lip may represent a click of the mouse/joystick button. These signals may also be used to control the velocity (rather than the position) of the mouse pointer, as in an isometric mouse. • the tilt angle and force of a user's lip may control the X-Y position of a mouse pointer or a joystick in a computer user interface and a puff of breath or a squeeze of a lip may represent a click of the mouse/joystick button.
• the angle of the user's lip and the force applied to the lip may be used as X-Y control to highlight a particular object within a two dimensional array of objects shown on a computer screen. Such a display may show an alphanumeric keyboard for example. A puff of breath could select the currently highlighted object on the display. For example a puff of breath could select a letter on a displayed alphanumeric keyboard.
[0168] Another application of the mouth-operated input devices of the invention include speech synthesis or silent communication. Many of the sounds made by human vocal cords depend on motion of the tongue, jaw and lips as well as breath- pressure. Thus, the mouth-operated input devices of the invention may be configured to output control signals which relate to the gestures associated with speech and which could be provided to a speech synthesizer which emulates the sounds of speech based on the control signals. Similarly, the control signals relating to the gestures associated with speech could be encoded by the device and later decoded as sounds. This would enable a person to "talk" without making sound. [0169] The mouth-operated input devices described herein may be provided with an infrared emitter corresponding to the IRDA standard which is commonly used to control consumer devices, such as remote controls and PDA devices. A mapping may be provided such that combinations of the types of gestures described herein can control consumer devices such as entertainment systems. [0170] Another application of the mouth-operated input devices described herein is to select between a one or two dimensional array of locations which may be predefined on the roof of the user's mouth. The user may select any of these locations with the tip of their tongue. Such locations may be discriminated using oral cavity /tongue
sensors of the general type described above together with the breath pressure sensor. A user may emit a puff of breath similar to a "t" sound with their tongue starting at any of these locations. The position of the user's tongue may be detected by the oral cavity/tongue sensors and the puff of breath may be detected by the breath pressure sensor. This puff may cause the signal processing system to emit a signal related to a virtual button defined for that starting location. Alternatively, the tongue may be positioned in one of these locations and a puff of air similar to the "c" in "cat" or the "h" in "hat" may be emitted, causing the signal processing system to emit a signal that is associated with the current location of the tongue. In some embodiments, the mouth-operated input device may be configured to distinguish between a "t", a "c" or an "h". An output 92 may be used to display (to the user) the action or symbol that will result if the "t", "c" or "h" breath puff is emitted with the tongue at the current location. This capability may be used for example for "typing" (e.g. selecting from among virtual keys which may be alphanumeric), for playing computer games, or for pressing the keys of a virtual musical instrument while controlling the volume with the breath and ranges of pitch and/or timbre with the lips. The detection of such "t", "c" and/or "h" sounds may be performed using any of the mouth-operated input devices described herein to provide desired functionality.
[0171] Figure 17A is an exploded isometric view of a mouth-operated input device 410 according to yet another embodiment of the invention. Mouth-operated input device 410 functions as a joystick type input device for controlling some external system (not shown) which may be an instrument, but which may alternatively be any other suitably configured system, such as a personal computer, a video game console, a wheelchair control system or the like. In the illustrated embodiment, mouth-operated joystick device 410 incorporates a mouth-operated input device 312 that is substantially similar to mouth-operated input device 312 of Figure 15. Mouth- operated joystick device 410 also comprises a force/joystick sensor system 412 capable of detecting the direction and/or the amount of force applied to mouth- operated input device 312 by a user's mouth (e.g. by the user's teeth, lips, tongue) or, if desired, by some other part of the user's body.
[0172] Force sensing system 412, which is shown in more detail in Figure 17B, includes: an upper housing element 422 and a lower housing element 417 which are moveable relative one another and a substrate 414 which houses a plurality (e.g. four) optical sensors 416A, 416B, 416C, 416D (collectively, sensors 416). In the illustrated embodiment, sensors 416 are reflective optical sensors which are configured to emit radiation in the general direction of arrow 418. Sensors 416 may be substantially similar to reflective optical sensors 46, 48 discussed above. In other
embodiments, sensors 416 could be implemented as transmissive type optical sensors.
[0173] Force sensing system 412 also includes a reflective element 420 which, in the illustrated embodiment, is coupled to upper housing element 422 by fastener components 423A, 423B in such a manner that reflective element 420 is deformable or otherwise moveable relative to substrate 414. In the illustrated embodiment, fastener component 423B projects loosely through aperture 425 in substrate 414 and is snugly received in a correspondingly shaped aperture 427 in reflective element 420 such that movement of upper housing element 420 causes corresponding movement of reflective element 420 relative to substrate 414. In particular embodiments, the range of movement of reflective element 420 relative to substrate 414 is less than about lmm. In the illustrated embodiment, element 420 is reflective. Suitable materials for reflective element 420 include reflective spring steel, for example. In embodiments where sensors 416 are transmissive type sensors, element 420 comprises an opaque element 420 and element 420 may be fabricated from an opaque material. In some embodiments, reflective/opaque element 420 is formed from a first material which is coated (at least partially) with a second material having the desired optical characteristics. [0174] Reflective element 420 is shown in detail in Figure 17B. In the illustrated embodiment, reflective element 420 comprises a plurality (e.g. four) of tabs 426 A, 426B, 426C, 426D (collectively, tabs 426) which extend generally radially from a central region 428. When device 410 is in an ambient state, tabs 426 are generally located below corresponding sensors 416 at generally equal distances from sensors 416. Accordingly, when device 410 is in an ambient state, the reflected radiation detected from each of sensors 416 and the corresponding sensor output signals is approximately equal. Due to variations in the characteristics of individual optical sensors 416, the output signals from sensors 416 may require calibration (e.g. adjustment by scaling, offset or other suitable technique) to achieve this equality. [0175] Upper housing element 422 may be physically coupled to one or more sidewalls 430 of device 410 or may otherwise be physically coupled to device 410, such that force applied by the user to mouth-operated input device 312 results in corresponding force applied via fastener component 423B and to reflective element 420. Such force can move reflective element 420 relative to substrate 414 and can move one or more of tabs 426 toward or away from their corresponding sensors 416, resulting in changes to the amount of radiation detected by individual sensors 416. For example, if a user's mouth applies a force to mouth-operated input device 312 in a direction of arrow 433 (Figure 17B), tabs 426B, 426C may move closer to their corresponding sensors 416B, 416C and tabs 426 A, 426D may move further from
their corresponding sensors 416A, 416D. Such movement may result in sensors 416B, 416C detecting an increased amount of radiation and sensors 416A, 416D detecting a decreased amount of radiation.
[0176] In other embodiments, this same effect can be achieved by physically coupling substrate 414 to upper housing element 422 such that force applied by a user to mouth-operated input device 312 results in movement of substrate 414 relative to reflective element 420.
[0177] Figure 18 represents a schematic depiction of device 410 according to a particular embodiment of the invention. As shown in Figure 18, device 410 comprises a mouth-operated input device 312 which incorporates lip sensor 348, a pair of oral cavity sensors 346' , 346" and breath sensor 380' . These sensors produce lip sensor output signal 376, oral cavity sensor output signals 364', 364" and breath sensor signal 382, which are provided to device electronics 416. For clarity, separate signal processing electronics are not shown in Figure 18 but may be present in device 410. Mouth-operated input device 312 may also comprise one or more optional inputs 314 which provide their own signals to device electronics 416. [0178] Device 410 also comprises a plurality of joystick sensors 416 which provide a corresponding plurality of joystick signals 452 to device electronics 416. Although not shown explicitly in Figure 18, signal conditioning electronics may also be provided for joystick sensors 416. In the illustrated embodiment, device 410 comprises four joystick sensors 416 (see Figure 17B) which provide four corresponding joystick signals 452 to device electronics 416. One or more optional additional inputs 414 may provide their own signals to device electronics 416. Such optional additional inputs may be located on device 410 or may be external to device 410.
[0179] In the illustrated embodiment, device electronics 416 comprises a microprocessor 429A and accessible memory 429B. Device electronics 416 processes the signals received from the various sensors and inputs and generates a plurality of corresponding output signals 418 which are provided to I/O port 444. I/O port 444 may generally be connected to any system (not shown) so that device 410 can be used to provide input to the system and thereby allow the user to control the system using their mouth. Output signals 418 may comprise one or more signals corresponding to each of lip sensors 348, oral cavity sensors 346', 346", breath sensors 380', joystick sensors 416, optional inputs 314, 414 and/or various combinations of these sensors and/or inputs. This can provide a wide range of control options.
[0180] Device 410 may also be capable of receiving one or more signals 469 from the external system via port 444. Such signals 469 can be provided to device electronics 416 and can be used by device electronics 416 to control device 410
and/or to help process the various other signals received at device electronics 416. Device 410 may also comprise one or more of its own optional outputs 492 which may be similar to outputs 92 described above.
[0181] In some alternative embodiments, instrument electronics 416 need not comprise a microprocessor 429A and/or memory 429B. In a manner similar to the difference between the embodiments of Figures 11 and 12, such components may be provided in the external system.
[0182] Device 410 can function as a two axis joystick which is controllable by the user's mouth. Force applied by the user's mouth to mouth-operated input device 312 along a first axis could cause variation in the positions of opposing tabs 426B, 426D and corresponding variation in the sensor signals 452 from sensors 416B, 416D. Similarly, force applied by the user's mouth along an orthogonal axis could cause variation in the positions of opposing tabs 426A, 426C and corresponding variation in the sensor signals 452 from sensors 416A, 416C. Forces in other directions will result in various linear combinations of the movements along one of these orthogonal axes. Such forces may be applied to mouth-operated input device using the user's lips and/or the user's teeth, for example. A user can control the joystick action independently of any of the other sensors (i.e. the breath sensor(s), oral cavity sensor (s) and/or lip sensor (s). For example, a user can control the joystick action with their teeth while independently interacting with the other sensors. With practice, a user can independently control this joystick action with an outer portion 32B of their lower lip 32 while using the inward portion 32A of their lower lip 32 to interact with lip sensor 348. This independent control may be facilitated in part by convexity 337 (see Figure 15). [0183] Device 410 may be used to provide a wide variety of functionalities and to provide input to a wide variety of external systems. By way of non-limiting example:
• Device 410 is usable as a joystick which provides input to an external system (e.g. a computer system, a video gaming system, a wheel chair operating system or the like). In a joystick embodiment, the relative position of reflective element 420 (as detected by sensors 416) maps to a joystick direction and velocity and various tongue, lip, oral cavity or breach actions can be used to provide other joystick functionalities (e.g. buttons clicks).
• Device 410 is usable as a pointing device (i.e. in a manner similar to a conventional computer mouse) for an external system such as a computer or the like. In such an embodiment, the relative position of reflective element
420 (as detected by sensors 416) can be used to map a position of the pointing device and various tongue, lip, oral cavity or breach actions can be used to provide other pointing device functionalities (e.g. left click, right click, scroll
wheel etc.). In one particular embodiment, a certain force range on the inward portion 32A of lip 32 (or some other aspect of tongue position, oral cavity configuration or breath pressure) may be used to activate/deactivate the pointing device in a manner which would provide a functionality similar to picking up and repositioning a mouse. In some embodiments, these force ranges may be subdivided into a multiple levels to provide a variety of functionalities. For example, a low level of lip force may correspond to turning "OFF" the operation of the pointing device, a moderate level of lip force may correspond to normal pointing device activity and a high level of lip force may correspond to "dragging" a virtual object with the pointing device (i.e. similar to "clicking and dragging" with a conventional mouse).
• Device 410 could be used as a stylus type input device for an external system such as a handheld computer or the like. In such an embodiment, the relative position of reflective element 420 (as detected by sensors 416) can be used to map a position of the stylus and various tongue, lip, oral cavity or breach actions can be used to provide other functionalities (e.g. activating/deactivating the stylus etc.).
• Device 410 can be used as a simulator for a musical instrument such as a trombone, harmonica or Theremin, where the relative position of reflective element 420 (as detected by sensors 416) can be used to map a pitch (i.e. like the stepped side to side pitch control of a harmonica or the continuous front to back pitch control of a trombone). In such an embodiment, breath pressure may be used to map volume and the lip, tongue and/or oral cavity sensors can be used to control timbre or other characteristics. [0184] As will be apparent to those skilled in the art in the light of the foregoing disclosure, many alterations and modifications are possible in the practice of this invention without departing from the spirit or scope thereof. For example:
• In the embodiments, described above, each optical sensor comprises a radiation emitter and a radiation sensor for detecting reflected radiation. In general, a one-to-one correspondence between radiation emitters and radiation detectors is not necessary. Radiation from a single emitter may be detected by multiple radiation detectors and radiation from multiple emitters may be detected by a single radiation detector. In some embodiments, it is desirable to reduce "cross-talk" between different optical sensors or different optical sensing systems. Such cross-talk may arise because of detection of radiation in a particular sensor which may originate from the emitter(s) of other sensor(s). In such cases, it may be desirable to apply rapid non-overlapping pulses to various sensors or various groups of sensors, so as to mutliplex the
sensor signals in the time domain. For example, lip sensors and oral cavity sensors may be pulsed at different times from one another or individual lip sensors may be pulsed at different times than one another. In other embodiments, it may be desirable to have different sensors (or groups of sensors) operate at different wavelengths, so as to multiplex the sensor signals in the frequency domain.
• It is conceivable that one could provide a two dimensional array of oral cavity sensors to capture oral cavity data which may be similar to an image of the interior of a user's oral cavity and may use such two dimensional data to control various systems. In such embodiments, one or more lenses or other optical elements may be provided on the sensor array for imaging the oral cavity (or a portion thereof).
• As discussed above, mouth-operated input device 312 of Figure 15 makes use of transmissive-type radiation sensors to implement lip sensor 348 (i.e. to detect one or more characteristics of inward portion 32A of a user's lip 32).
In such transmissive-type radiation sensors, radiation is transmitted from an emitter to a radiation detector that is located remotely, but in the optical path of the transmitted beam, such that the amount of radiation received at the remotely located detector is reduced by interference from various parts of the user's mouth. Transmissive-type optical sensors may have other configurations. In one particular example, mouth-operated input device 212 may be modified to provide one or more transmissive-type optical sensors for detecting the force applied to the inward portion 32A of lip 32. Radiation emitters may be located inside oral cavity 52 (e.g. in fin 251) and may be directed back toward radiation detectors, which may be at or near intermediate surface 240, for example. Such radiation emitters and detectors could then be used to generate lip sensor output signals 276 which vary with the force applied to inward portion 32A of lip 32. For example, increased lip- force may cause inward portion 32A of lip 32 to deform around convexity 237 and into region 233, thus correspondingly reducing the amount of radiation detected by the radiation detectors. Similar transmissive lip sensors may be designed such that both their radiation emitters and radiation detectors are located at or near intermediate surface 40 and fin 251 is configured to reflect emitted radiation back toward the radiation detectors or such that both their radiation emitters and radiation detectors are located in fin 251 and intermediate surface 40 and is configured to reflect emitted radiation back toward the radiation detectors
• The instrument electronics and/or signal conditioning electronics described above may comprise one or more digital processor(s) capable of performing the operations discussed above. Such processor(s) may comprise, without limitation, a microprocessor, a programmable logic array, a computer-on-a- chip, the CPU of a computer or any other suitably programmable controller.
• The lip sensors described above generate lip sensor output signals (and, ultimately, processed lip signal(s)) which represent the force applied by the user on inward portion 32 A of lower lip 32. These signals may also represent the position of inward portion 32A of lower lip 32 relative to an aspect of the mouth-operated input device (e.g. relative to the convexity of the mouth- operated input device and/or relative to the upper portion of the lower surface of the mouth-operated input device).
• The mouth-operated input devices described above have lower surfaces (e.g. lower surface 36) having lip-receiving convexities (e.g. convexity 37) provided by the stepped profile of spaced apart lower and upper portions (e.g. lower portion 36A and upper portion 36B). Such lip receiving convexities provide distinct advantages when measuring characteristics of inward portion 32A of lower lip 32 as discussed above. Also, the provision of a lip sensing system on one side of a mouth-operated input device (i.e. on the lower surface of the device) allows for the independent measurement of the characteristics of a single lip. Those skilled in the art will appreciate that similar lip receiving convexities may additionally or alternatively be provided on the upper surfaces of the mouth-operated input devices described above, for providing similar advantages relating to the measurement of characteristics of upper lip 26. Similar lip sensors may also be provided on one or both transverse sides of the mouth-operated input devices described above, for detecting the position (or other characteristics) of the user's lips relative to a transverse side of the device.
• In general, the lip-receiving convexities on the lower surfaces of the mouth- operated input devices described above need not be strictly be implemented using the stepped profile (e.g. the stepped profile of lower portion 36 A and an upper portion 36B (Figure I)). Figures 13A-13C schematically depict mouth- operated input devices 12 according to a number of other embodiments having various lip-receiving profiles. In each of the embodiments of Figure 13, device 12 comprises at least one lip-receiving convexity 37 and a region of space 33 located adjacent convexity 37 such that a portion of lip 32 can deform around convexity 37 and into region 33. Region 33 may be located between first and second spaced apart surfaces 36A, 36B. One or both of
surfaces 36A, 36B may comprise lip-receiving convexity 37. At least one optical sensor 48 is configured to emit radiation into region 33 and to detect radiation either transmitted through region 33 and/or reflected from lip 32 when lip 32 is located in region 33. Optical sensor 48 is capable of detecting a signal related to how much of lip 32 is located in region 33 which in turn is related to the force applied to the user's lip 32. When a user applies a larger force to their lip 32, a portion of lip 32 deforms around the lip-receiving convexity and further into region 33. Variation of the deformation of lip 32 into region 33 causes corresponding variation in the amount of transmitted and/or reflected radiation and the output level of sensor 48.
• In some of the above described embodiments, the outputs of all of the sensors are used by the instrument electronics to generate a single output signal. This is not necessary. In some embodiments, the instrument electronics may make use of one or more sensors of the types described above to generate a plurality of independent output signals.
• Those skilled in the art will appreciate that joystick/force sensing system 412 of device 410 represents one particular embodiment of a force sensing system. In some embodiments, other types joystick/force sensing systems could be used in the place of system 412. Mouth-operating input devices incorporating such alternative joystick/force sensing systems could still take advantage of the functionalities afforded by the lip pressure sensing systems and oral cavity sensing systems described herein. For example, the joystick/force sensing system could provide position mapping and various tongue, lip, oral cavity or breach actions could be used in to provide other pointing device functionalities (e.g. left click, right click, scroll wheel etc.).
• Device 410 described above incorporates mouth-operated input device 312. In alternative embodiments, device 410 may comprise any of the other mouth- operated input devices described herein.
• The optical lip force sensors (e.g. of devices 12, 212) and the resistive lip force sensors (e.g. of device 512) may be combined in a device that incorporates multiple forms of lip force sensors.
Claims
1. A mouth-operated input device comprising: a lip-receiving component comprising a lip-receiving surface for receiving a user's lip; and one or more force-sensitive resistors, each force sensitive resistor having an active region coupled to a corresponding region of the lip-receiving surface for generating a corresponding lip-force output signal; wherein the lip-force output signal generated by each force-sensitive resistor varies with an amount of lip force applied by the user's lip to the corresponding region of the lip-receiving surface.
2. An input device according to claim 1 comprising, for each force-sensitive resistor, a coupling mechanism located so as to transmit a lip force applied by the user's lip to the corresponding region of the lip-receiving surface to a sensor force applied by the coupling mechanism to the active region of the force-sensitive resistor, wherein the force per unit area of the sensor force is greater than the force per unit area of the lip force.
3. An input device according to claim 2 wherein, for each force-sensitive resistor, the coupling mechanism comprises a projection extending between the corresponding region of the lip-receiving surface and the active region of the force-sensitive resistor and wherein a first cross-sectional area of the projection at an end adjacent the corresponding region of the lip-receiving surface is greater than a second cross-sectional area of the projection at an end adjacent the active region of the force-sensitive resistor.
4. An input device according to claim 1 comprising, for each force-sensitive resistor, a coupling mechanism extending between the corresponding region of the lip-receiving surface and the active region of the force-sensitive resistor, the coupling mechanism shaped to pre-load the active region of the force-sensitive resistor with a force per unit area that is in a range of 50%- 150% of a threshold activation pressure of the force-sensitive resistor.
5. An input device according to claim 1 comprising, for each force-sensitive resistor, a coupling mechanism extending between the corresponding region of the lip-receiving surface and the active region of the force-sensitive resistor, the coupling mechanism shaped to pre-load the active region of the force-sensitive resistor with a force per unit area that is in a range of 80%- 120% of a threshold activation pressure of the force-sensitive resistor.
6. An input device according to any one of claims 1 to 5 wherein the lip- receiving component comprises an elastomeric material which is deformable in response to application of lip force to the lip-receiving surface.
7. An input device according to claim 6 wherein the elastomeric material has a hardness in a range of Shore A 5 to Shore A 120.
8. An input device according to any one of claims 1 to 7 wherein the lip- receiving component comprises an external mouthpiece component which includes the lip-receiving surface and an internal mouthpiece component which supports the one or more force-sensitive resistors and wherein the external mouthpiece component is relatively more deformable than the internal mouthpiece component.
9. An input device according to any one of claims 1 to 7 wherein the lip- receiving component comprises an external mouthpiece component which includes the lip-receiving surface and an internal mouthpiece component which supports the one or more force-sensitive resistors and wherein the external mouthpiece component is relatively more transparent than the internal mouthpiece component.
10. An input device according to any one of claims 1 to 9 comprising a plurality of force-sensitive resistors, each force sensitive resistor having an active region coupled to a corresponding region of the lip-receiving surface for generating a corresponding lip-sensor output signal, wherein the lip-sensor output signal generated by each force-sensitive resistor varies with an amount of lip force applied by the user's lip to the corresponding region of the lip- receiving surface.
11. An input device according to claim 10 comprising electronics connected to receive the corresponding lip-sensor output signals and to generate therefrom one or more processed lip signals, each of the processed lip signals varying in response to changes in lip force applied to the lip-receiving surface by one or more different parts of the lip.
12. An input device according to claim 10 comprising electronics connected to receive the corresponding lip-sensor output signals and to generate therefrom one or more processed lip signals, each of the processed lip signals varying in response to changes in position of one or more different parts of the lip on the lip-receiving surface.
13. An input device according to claim 10 comprising electronics connected to receive the corresponding lip-sensor output signals and to generate therefrom one or more processed lip signals, each of the processed lip signals varying in response to changes in a different physical characteristic of the lip.
14. An input device according to claim 13 wherein the different characteristics of the lip include one or more of: force applied to the lip; angle of the lip as it extends transversely across the lip-receiving surface; curvature of the lip as it extends transversely across the lip-receiving surface; and position of the lip.
15. An input device according to claim 13 wherein the different characteristics of the lip include two or more of: force applied to the lip; angle of the lip as it extends transversely across the lip-receiving surface; curvature of the lip as it extends transversely across the lip-receiving surface; and position of the lip.
16. A mouth-operated input device comprising: a lip-receiving component comprising a lip-receiving surface for receiving a user's lip; and a plurality of force sensors, each force sensor configured to generate a corresponding lip-sensor output signal based on an amount of lip force applied by the user's lip to a corresponding region of the lip-receiving surface.
17. An input device according to claim 16 comprising electronics connected to receive the corresponding lip-sensor output signals and to generate therefrom one or more processed lip signals, each of the processed lip signals varying in response to changes in lip force applied to the lip-receiving surface by one or more different parts of the lip.
18. An input device according to claim 16 comprising electronics connected to receive the corresponding lip-sensor output signals and to generate therefrom one or more processed lip signals, each of the processed lip signals varying in response to changes in position of one or more different parts of the Hp on the lip-receiving surface.
19. An input device according to claim 16 comprising electronics connected to receive the corresponding lip-sensor output signals and to generate therefrom one or more processed lip signals, each of the processed lip signals varying in response to changes in a different physical characteristic of the lip.
20. An input device according to claim 19 wherein the different characteristics of the lip include one or more of: force applied to the lip; angle of the lip as it extends transversely across the lip-receiving surface; curvature of the lip as it extends transversely across the lip-receiving surface; and position of the lip.
21. An input device according to claim 19 wherein the different characteristics of the lip include two or more of: force applied to the lip; angle of the lip as it extends transversely across the lip-receiving surface; curvature of the lip as it extends transversely across the lip-receiving surface; and position of the lip.
22. An input device according to any one of claims 16 to 21 wherein each force sensor comprises: a force-sensitive resistor comprising an active region; and a coupling mechanism located so as to transmit a lip force applied by the user's lip to the corresponding region of the lip-receiving surface to a sensor-force applied by the coupling mechanism to the active region of the force-sensitive resistor wherein the force per unit area of the sensor-force is greater than the force per unit area of the lip force.
23. An input device according to claim 22 wherein, for each force-sensitive resistor, the coupling mechanism comprises a projection extending between the corresponding region of the lip-receiving surface and the active region of the force-sensitive resistor and wherein a first cross-sectional area of the projection at an end adjacent the corresponding region of the lip-receiving surface is greater than a second cross-sectional area of the projection at an end adjacent the active region of the force-sensitive resistor.
24. An input device according to any one of claims 16 to 21 wherein each force sensor comprises: a force-sensitive resistor comprising an active region; and a coupling mechanism extending between the corresponding region of the lip-receiving surface and the active region of the force-sensitive resistor, the coupling mechanism shaped to pre-load the active region of the force- sensitive resistor with a force per unit area that is in a range of 50% -150% of a threshold activation pressure of the force-sensitive resistor.
25. An input device according to any one of claims 16 to 21 wherein each force sensor comprises: a force-sensitive resistor comprising an active region; and a coupling mechanism extending between the corresponding region of the lip-receiving surface and the active region of the force-sensitive resistor, the coupling mechanism shaped to pre-load the active region of the force- sensitive resistor with a force per unit area that is in a range of 80% -120% of a threshold activation pressure of the force-sensitive resistor.
26. An input device according to any one of claims 16 to 25 wherein the Hp- receiving component comprises an elastomeric material which is deformable in response to application of lip force to the lip-receiving surface.
27. An input device according to claim 26 wherein the elastomeric material has a hardness in a range of Shore A 5 to Shore A 120.
28. An input device according to any one of claims 16 to 27 wherein the lip- receiving component comprises an external mouthpiece component which includes the lip-receiving surface and an internal mouthpiece component which supports the plurality of force sensors and wherein the external mouthpiece component is relatively more deformable than the internal mouthpiece component.
29. An input device according to any one of claims 16 to 27 wherein the lip- receiving component comprises an external mouthpiece component which includes the lip-receiving surface and an internal mouthpiece component which supports the plurality of force sensors and wherein the external mouthpiece component is relatively more transparent than the internal mouthpiece component.
30. A mouth-operated input device comprising: a lip receiving element for receiving a user's lip, the lip receiving element comprising at least one convexity such that application of lip-force against the lip receiving element causes a first portion of the lip to deform around the convexity into a region of space adjacent the convexity; an optical lip-force sensor comprising an emitter for directing radiation into the region and a detector for detecting radiation emanating from the region and for generating a lip-force output signal based on an amount of detected radiation; wherein the lip-force output signal varies in response to changes in the lip- force.
31. An input device according to claim 30 wherein the lip-force sensor is a reflective-type optical lip-force sensor for detecting radiation reflected from the first portion of the lip located in the region and the lip-force output signal is positively correlated with an amount of the first portion of the lip located in the region.
32. An input device according to claim 30 wherein the lip-force sensor is a transmissive-type optical lip-force sensor for detecting radiation transmitted through the region and the lip-force output signal is negatively correlated with an amount of the first portion of the lip located in the region.
33. An input device according to claim 32 wherein the emitter and the detector are located on opposing sides of the region which are spaced-apart from one another in a transverse direction, the transverse direction generally parallel to an elongated dimension of the lip.
34. An input device according to claim 32 wherein the emitter and the detector are located on a same side of the region and the device comprises a reflective surface located on an opposing side of the region.
35. An input device according to any one of claims 30 to 34 wherein the lip receiving element comprises a lip receiving surface for receiving a second portion of the lip, the lip receiving surface located on an opposite side of the convexity from the region.
36. An input device according to claim 35 wherein the lip receiving element comprises a second surface, the second surface spaced apart from the lip receiving surface so as to form a step profile with the convexity located between the lip receiving surface and the second surface.
37. An input device according to claim 36 wherein the region is located between a first plane substantially parallel with at least a portion of the lip receiving surface and a second plane substantially parallel with at least a portion of the second surface.
38. An input device according to any one of claims 30 to 37 comprising normalization electronics configured to: obtain a first sample of the lip-force output signal at a first time when the emitter is active to direct radiation; obtain a second sample of the lip force output signal at a second time when the emitter is not active to direct radiation; and output a difference between the first sample and the second sample as the lip-force output signal.
39. An input device according to any one of claims 30 to 37 comprising normalization electronics, the normalization electronics comprising: a sample and hold component for obtaining a first sample of the lip- force output signal at a first time; and a difference amplifier for obtaining a second sample of the lip-force output signal at a second time and for outputting a difference between the first sample and the second sample as the lip-force output signal; wherein the emitter is active to direct radiation at one of the first and second times and is not active to direct radiation at the other one of the first and second times.
40. An input device according to claim 30 comprising a plurality of optical lip sensors, each optical lip sensor comprising: a corresponding emitter for directing radiation into the region; and a corresponding detector for detecting radiation emanating from the region and for generating a corresponding lip sensor output signal based on a corresponding amount of detected radiation.
41. An input device according to claim 40 comprising electronics connected to receive the corresponding lip sensor output signals and to generate therefrom one or more processed lip signals, each of the processed lip signals varying in response to at least one of: changes in lip-force applied at one or more different parts of the lip; and changes in position of the one or more different parts of the lip.
42. An input device according to claim 40 comprising electronics connected to receive the corresponding lip sensor output signals and to generate therefrom one or more processed lip signals, each of the processed lip signals varying in response to changes in one or more of: force applied to the lip; angle of the first portion of the lip as it extends through the region; curvature of the first portion of the lip as it extends through the region; and position of the first portion of the lip.
43. An input device according to any one of claims 1 and 30 used in one of: an instrument simulator comprising an audio synthesizer wherein the instrument simulator is configured to generate one or more control signals for controlling the audio output of the audio synthesizer based at least in part on the lip-force output signal; a wheelchair controller wherein the wheelchair controller is configured to generate one or more control signals for controlling movement of a corresponding wheelchair based at least in part on the lip-force output signal; and an input controller for a computer system wherein the input controller is configured to generate one or more control signals for controlling operation of the computer system based at least in part on the lip- force output signal.
44. An input device according to any one of claims 11 to 15, 17 to 21, 41 and 42 used in one of: an instrument simulator comprising an audio synthesizer wherein the instrument simulator is connected to receive the one or more processed lip signals and is configured to generate one or more control signals for controlling an audio output of the audio synthesizer based at least in part on the one or more processed lip signals; a wheelchair controller wherein the wheelchair controller is connected to receive the one or more processed lip signals and is configured to generate one or more control signals for controlling movement of a corresponding wheelchair based at least in part on the one or more processed lip signals; and an input controller for a computer system wherein the computer system is connected to receive the one or more processed lip signals and is configured to generate one or more control signals for controlling operation of the computer system based at least in part on the one or more processed lip signals.
45. An input device according to any one of claims 1 to 44 comprising an oral cavity sensor comprising an oral cavity emitter for directing oral cavity radiation into a user's oral cavity and an oral cavity detector for detecting oral cavity radiation reflected from one or more surfaces of the oral cavity and for generating an oral cavity sensor output signal based on an amount of detected oral cavity radiation.
46. An input device according to claim 45 wherein the oral cavity emitter is configured to emit a beam of radiation having a center that is oriented at an angle in a range of 0°-30° below a longitudinal axis oriented generally parallel to a roof of the oral cavity and wherein the oral cavity sensor output signal varies in response to changes in a distance between the device and a forward facing surface of the user's tongue.
47. An input device according to claim 45 wherein the oral cavity emitter is configured to emit a beam of radiation having a center that is oriented at an angle in a range of 0°-30° above a longitudinal axis oriented generally parallel to a roof of the oral cavity and wherein the oral cavity sensor output signal varies in response to changes in a distance between an apex of the user's tongue and a roof of the user's mouth.
48. An input device according to any one of claims 45 to 47 comprising normalization electronics configured to: obtain a first sample of the oral cavity sensor output signal at a first time when the oral cavity emitter is active to direct radiation; obtain a second sample of the oral cavity sensor output signal at a second time when the oral cavity emitter is not active to direct radiation; and output a difference between the first sample and the second sample as the oral cavity sensor output signal.
49. An input device according to any one of claims 45 to 47 comprising normalization electronics, the normalization electronics comprising: an oral cavity sample and hold component for obtaining a first sample of the oral cavity sensor output signal at a first time; and an oral cavity difference amplifier for obtaining a second sample of the oral cavity sensor output signal at a second time and for outputting a difference between the first sample and the second sample as the oral cavity sensor output signal; wherein the oral cavity emitter is active to direct radiation at one of the first and second times and is not active to direct radiation at the other one of the first and second times.
50. An input device according to any one of claims 1 to 44 comprising a plurality of oral cavity sensors, each oral cavity sensor comprising a corresponding oral cavity emitter for directing oral cavity radiation into a user's oral cavity and a corresponding oral cavity detector for detecting corresponding oral cavity radiation reflected from one or more surfaces of the oral cavity and for generating a corresponding oral cavity sensor output signal based on a corresponding amount of detected oral cavity radiation.
51. An input device according to claim 50 comprising electronics connected to receive the corresponding oral cavity sensor output signals and configured to generate therefrom one or more processed oral cavity signals, each of the processed oral cavity signals varying in response to changes one or more of: a distance between the device and a forward-facing surface of a user's tongue; a distance between the apex of the user's tongue and a roof of a user's mouth; and a side to side position of the user's tongue.
52. An input device according to claim 51 wherein the corresponding oral cavity emitter of at least one first oral cavity sensor is configured to emit a beam of radiation having a center that is oriented in a first direction toward a user's tongue and the corresponding oral cavity emitter of at least one second oral cavity sensor is configured to emit a beam of radiation having a center that is oriented in a second direction toward a roof of a user's mouth.
53. An input device according to claim 52 wherein the first direction is oriented at an angle in a range of 0°-30° below a longitudinal axis and the second direction is oriented at an angle in a range of 0°-30° above the longitudinal axis, the longitudinal axis oriented generally parallel to the roof of the user's mouth.
54. An input device according to claim 52 wherein one of the processed oral cavity signals varies in response to changes in a distance between the device and a forward facing surface of a user's tongue and is determined based, at least in part, on the corresponding oral cavity sensor output signal associated with the first oral cavity sensor and wherein another one of the processed oral cavity signals varies in response to changes in a distance between an apex of a user's tongue and the roof of the user's mouth and is determined based, at least in part, on a difference function between the corresponding oral cavity sensor output signals associated with the first oral cavity sensor and the second oral cavity sensor.
55. An input device according to claim 50 wherein the corresponding emitter of at least one first optical oral cavity sensor is configured to emit a beam of radiation having a center that is oriented in a first direction at an angle in a range of 10°-45° on one transverse side of a longitudinal axis and the corresponding emitter of at least one second optical oral cavity sensor is configured to emit a beam of radiation having a center that is oriented in a second direction at an angle in a range of 10° -45° on an opposing transverse side of the longitudinal axis, the longitudinal axis oriented generally parallel to the user's cheeks.
56. An input device according to claim 55 wherein one of the processed oral cavity signals varies in response to changes in a side to side position of a user's tongue and is determined based, at least in part, on a difference function between the corresponding oral cavity sensor output signals associated with the first oral cavity sensor and the second oral cavity sensor.
57. An input device according to any one of claims 1 to 56 comprising a breath- pressure-sensor for detecting a user's breath-pressure and generating a breath- pressure-sensor output signal based on an amount of detected breath-pressure.
58. An input device according to claim 57 comprising an adjustable breath flow regulator for regulating an amount of breath flow through the device.
59. An input device according to any one of claims 1 to 58 wherein the input device is mounted on a base and wherein the input device comprises a plurality of optical joystick sensors located between the input device and the base for detecting an amplitude and direction of movement of the device relative to the base.
60. An input device according to claim 59 wherein each of the plurality of optical joystick sensors comprises: a joystick emitter for directing joystick radiation toward a reflective member; and a corresponding joystick detector for detecting joystick radiation reflected from the reflective member and for outputting a corresponding joystick sensor signal that varies in response to changes in an amount of detected joystick radiation and wherein movement of the device relative to the base causes corresponding relative movement between the reflective member and the joystick emitters.
61. An input device according to claim 60 wherein the plurality of optical joystick sensors comprises a pair of optical joystick sensors aligned with a first axis and a second pair of optical joystick sensors aligned with a second axis, the second axis orthogonal to the first axis.
62. An input device according to any one of claims 1 and 30 comprising a force sensing system including one or more force sensors for sensing force applied to the device by the user's mouth in at least two orthogonal directions and producing at least one force-sensor output signal, the at least one force-sensor output signal varying with the force applied to the device by the user's mouth, the device configured as a pointing device wherein the at least one force- sensor output signal maps to a two-dimensional position of a virtual pointing element and the lip-force output signal maps to one or more discrete functionalities, the one or more discrete functionalities comprising one or more of: activating and deactivating the two-dimensional position mapping of the at least one force-sensor output signal; activating and deactivating dragging of virtual objects pointed at by the virtual pointing element; and executing virtual objects pointed at by the virtual pointing element.
63. A method for generating control signals using a user's mouth, the method comprising: providing one or more force-sensitive resistors; and for each of the one or more force-sensitive resistors: coupling an active region of the force-sensitive resistor to a corresponding region of a lip-receiving surface; receiving lip force applied by the user's lip to the corresponding region of the lip-receiving surface; and generating a lip-force output signal which varies with an amount of lip force applied to the corresponding region of the lip- receiving surface.
64. A method according to claim 63 wherein, for each force-sensitive resistor, coupling the active region of the force-sensitive resistor to the corresponding region of a lip-receiving surface comprises locating a coupling mechanism to transmit the lip force applied by the user's lip to the corresponding region of the lip-receiving surface to a sensor force applied by the coupling mechanism to the active region of the force-sensitive resistor, wherein the force per unit area of the sensor force is greater than the force per unit area of the lip force.
65. A method according to claim 64 wherein, for each force-sensitive resistor, locating the coupling mechanism comprises extending a projection between the corresponding region of the lip-receiving surface and the active region of the force-sensitive resistor and wherein a first cross-sectional area of the projection at an end adjacent the corresponding region of the lip-receiving surface is greater than a second cross-sectional area of the projection at an end adjacent the active region of the force-sensitive resistor.
66. A method according to claim 64 wherein, for each force-sensitive resistor, locating the coupling mechanism comprises pre-loading the active region of the force-sensitive resistor with a force per unit area that is in a range of 50% -150% of a threshold activation pressure of the force-sensitive resistor.
67. A method according to claim 64 wherein, for each force-sensitive resistor, locating the coupling mechanism comprises pre-loading the active region of the force-sensitive resistor with a force per unit area that is in a range of 80% -120% of a threshold activation pressure of the force-sensitive resistor.
68. A method according to any one of claims 63 to 67 wherein, for each force- sensitive resistor, receiving the lip force applied by the user's lip to the corresponding region of the lip-receiving surface comprises deforming the corresponding region of the lip-receiving surface.
69. A method according to any one of claims 63 to 68 comprising: providing a plurality of force-sensitive resistors; and for each of the plurality of force-sensitive resistors: coupling an active region of the force-sensitive resistor to a corresponding region of a lip-receiving surface; receiving lip force applied by the user's lip to the corresponding region of the lip-receiving surface; and generating a lip-sensor output signal which varies with an amount of lip force applied to the corresponding region of the lip- receiving surface.
70. A method according to claim 69 comprising: receiving the corresponding lip- sensor output signals; and generating therefrom one or more processed lip signals, each of the processed lip signals varying in response to changes in lip force applied to the lip-receiving surface by one or more different parts of the lip.
71. A method according to claim 69 comprising: receiving the corresponding lip- sensor output signals; and generating therefrom one or more processed lip signals, each of the processed lip signals varying in response to changes in position of one or more different parts of the lip on the lip-receiving surface.
72. A method according to claim 69 comprising: receiving the corresponding lip- sensor output signals; and generating therefrom one or more processed lip signals, each of the processed lip signals varying in response to changes in a different physical characteristic of the lip.
73. A method according to claim 72 wherein the different characteristics of the lip include one or more of: force applied to the lip; angle of the lip as it extends transversely across the lip-receiving surface; curvature of the lip as it extends transversely across the lip-receiving surface; and position of the lip.
74. A method according to claim 72 wherein the different characteristics of the lip include two or more of: force applied to the lip; angle of the lip as it extends transversely across the lip-receiving surface; curvature of the lip as it extends transversely across the lip-receiving surface; and position of the lip.
75. A method for generating control signals using a user's mouth, the method comprising: providing a plurality of force sensors; and for each force sensor: receiving lip force applied by the user's lip to the corresponding region of the lip-receiving surface; detecting the lip force applied by the user's lip to the corresponding region of the lip-receiving surface; and generating a lip-sensor output signal which varies with an amount of lip force applied to the corresponding region of the lip- receiving surface.
76. A method according to claim 75 comprising: receiving the corresponding lip- sensor output signals; and generating therefrom one or more processed lip signals, each of the processed lip signals varying in response to changes in lip force applied to the lip-receiving surface by one or more different parts of the lip.
77. A method according to claim 75 comprising: receiving the corresponding lip- sensor output signals; and generating therefrom one or more processed lip signals, each of the processed lip signals varying in response to changes in position of one or more different parts of the lip on the lip-receiving surface.
78. A method according to claim 75 comprising: receiving the corresponding lip- sensor output signals; and generating therefrom one or more processed lip signals, each of the processed lip signals varying in response to changes in a different physical characteristic of the lip.
79. A method according to claim 78 wherein the different characteristics of the lip include one or more of: force applied to the lip; angle of the lip as it extends transversely across the lip-receiving surface; curvature of the lip as it extends transversely across the lip-receiving surface; and position of the lip.
80. A method according to claim 78 wherein the different characteristics of the lip include two or more of: force applied to the lip; angle of the lip as it extends transversely across the lip-receiving surface; curvature of the lip as it extends transversely across the lip-receiving surface; and position of the lip.
81. A method according to any one of claims 75 to 80 wherein each of the force sensors comprises a force-sensitive resistor.
82. A method for generating control signals using a user's mouth, the method comprising: providing a lip receiving element for receiving a user's lip, the lip receiving element comprising at least one convexity such that application of lip-force against the lip receiving element causes a first portion of the lip to deform around the convexity into a region of space adjacent the convexity; directing radiation into the region; detecting radiation emanating from the region; and generating a lip-force output signal based on an amount of detected radiation; wherein the lip-force output signal varies in response to changes in the lip- force.
83. A method according to claim 82 wherein detecting radiation emanating from the region comprises detecting radiation reflected from the first portion of the lip.
84. A method according to claim 82 wherein detecting radiation emanating from the region comprises detecting radiation transmitted through the region.
85. A method according to claim 84 wherein directing radiation into the region comprises directing a beam of radiation having a center that is oriented in a transverse direction, the transverse direction generally parallel to an elongated dimension of the lip.
86. A method according to claim 82 wherein directing radiation into the region comprises directing a plurality of radiation beams into different portions of the region and, for each radiation beam: detecting corresponding radiation emanating from the region; and generating a corresponding lip sensor output signal based on a corresponding amount of detected radiation and wherein the method further comprises processing the corresponding lip sensor output signals to generate therefrom one or more processed lip signals, each of the processed lip signals varying in response to changes in one or more of: force applied to the lip; angle of the first portion of the lip as it extends through the region; curvature of the first portion of the lip as it extends through the region; and position of the first portion of the lip.
87. A method according to any one fo claims 83 to 85 comprising: at a first time, obtaining a first sample of the lip-force output signal while directing radiation into the region; at a second time, temporarily refraining from directing radiation into the region and obtaining a second sample of the lip force output signal; and outputting a difference between the first sample and the second sample as the lip-force output signal.
88. A method according to any one of claims 63 to 87 comprising: directing oral cavity radiation into a user's oral cavity; detecting oral cavity radiation reflected from one or more surfaces of the oral cavity; and generating an oral cavity sensor output signal based on an amount of detected oral cavity radiation.
89. A method according to any one of claims 63 to 87 comprising directing a plurality of oral cavity radiation beams into a user's oral cavity and, for each radiation beam: detecting corresponding oral cavity radiation reflected from one or more surfaces of the oral cavity; and generating a corresponding oral cavity sensor output signal based on a corresponding amount of detected oral cavity radiation; and wherein the method further comprises processing the corresponding oral cavity sensor output signals to generate therefrom one or more processed oral cavity signals, each of the processed oral cavity signals varying in response to changes in a different physical characteristics of the oral cavity.
90. A method according to claim 89 wherein directing a plurality of oral cavity radiation beams into the user's oral cavity comprises emitting a first beam of oral cavity radiation having a center that is oriented in a first direction toward a user's tongue and emitting a second beam of oral cavity radiation having a center that is oriented in a second direction toward a roof of a user's mouth.
91. A method according to claim 90 wherein one of the processed oral cavity signals varies in response to changes in a distance between the device and a forward facing surface of a user's tongue and is determined based, at least in part, on the corresponding oral cavity output signal associated with the first beam of oral cavity radiation and wherein another one of the processed oral cavity signals varies in response to changes in a distance between an apex of a user's tongue and the roof of the user's mouth and is determined based, at least in part, on a difference function between the corresponding oral cavity output signals associated with the first beam of oral cavity radiation and the second beam of oral cavity radiation.
92. A method according to claim 89 wherein directing a plurality of oral cavity radiation beams into the user's oral cavity comprises emitting a first beam of oral cavity radiation having a center that is oriented in a first direction at an angle in a range of 10° -45° on one transverse side of a longitudinal axis and emitting a second beam of oral cavity radiation having a center that is oriented in a second direction at an angle in a range of 10° -45° on an opposing transverse side of the longitudinal axis, the longitudinal axis oriented generally parallel to the user's cheeks.
93. A method according to claim 92 wherein one of the processed oral cavity signals varies in response to changes in a side to side position of a user's tongue and is determined based, at least in part, on a difference function between the oral cavity output signals associated with the first beam of oral cavity radiation and the second beam of oral cavity radiation.
94. A method according to claim 88 comprising: at a first time, obtaining a first sample of the oral cavity sensor output signal while directing radiation into the region; at a second time, temporarily refraining from directing radiation into the region and obtaining a second sample of the oral cavity sensor output signal; and outputting a difference between the first sample and the second sample as the oral cavity sensor output signal.
95. A mouth-operated input device comprising: a plurality of cavity sensors, each cavity sensor comprising an emitter for directing radiation into a user's oral cavity and a detector for detecting corresponding oral cavity radiation reflected from one or more surfaces of the oral cavity and for generating a corresponding oral cavity sensor output signal based on a corresponding amount of detected oral cavity radiation; and electronics connected to receive the oral cavity sensor output signals and configured to generate therefrom one or more processed oral cavity signals, each of the processed oral cavity signals varying in response to changes in a different physical characteristics of the oral cavity; wherein the emitter of at least one first oral cavity sensor is configured to emit a beam of radiation having a center that is oriented in a first direction at an angle in a range of 0°-45° above a longitudinal axis of the device and the emitter of at least one second oral cavity sensor is configured to emit a beam of radiation having a center that is oriented in a second direction at an angle in a range of 0°-45° below the longitudinal axis of the device, the longitudinal axis oriented generally parallel to a roof of a user's mouth.
96. A mouth-operated input device comprising: a plurality of cavity sensors, each cavity sensor comprising an emitter for directing radiation into a user's oral cavity and a detector for detecting corresponding oral cavity radiation reflected from one or more surfaces of the oral cavity and for generating a corresponding oral cavity sensor output signal based on a corresponding amount of detected oral cavity radiation; and electronics connected to receive the oral cavity sensor output signals and configured to generate therefrom one or more processed oral cavity signals, each of the processed oral cavity signals varying in response to changes in a different physical characteristics of the oral cavity; wherein the emitter of at least one first optical oral cavity sensor is configured to emit a beam of radiation having a center that is oriented in a first direction at an angle in a range of 0°-45° on one transverse side of a longitudinal axis of the device and the emitter of at least one second optical oral cavity sensor is configured to emit a beam of radiation having a center that is oriented in a second direction at an angle in a range of 0°-45° on an opposing transverse side of the longitudinal axis of the device, the longitudinal axis oriented generally parallel to the user's cheeks.
97. A mouth-operated input device comprising: an optical sensor for detecting one or more characteristics of a user's mouth, then optical sensor comprising an emitter for directing radiation toward at least a portion of the mouth and a detector for detecting radiation emanating from the portion of mouth and for generating a sensor output signal based on an amount of detected radiation; and normalization electronics configured to: obtain a first sample of the sensor output signal at a first time when the emitter is active to direct radiation; obtain a second sample of the sensor output signal at a second time when the emitter is not active to direct radiation; and output a difference between the first sample and the second sample as an output signal representative of the one or more characteristics of the mouth.
98. Apparatus and/or systems comprising any features, combination of features or sub-combination of features described herein.
99. Methods incorporating comprising any features, combination of features or sub-combination of features described herein.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US94007307P | 2007-05-24 | 2007-05-24 | |
| US60/940,073 | 2007-05-24 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2008141459A1 true WO2008141459A1 (en) | 2008-11-27 |
Family
ID=40031370
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CA2008/000986 Ceased WO2008141459A1 (en) | 2007-05-24 | 2008-05-23 | Mouth-operated input device |
Country Status (1)
| Country | Link |
|---|---|
| WO (1) | WO2008141459A1 (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2434478A3 (en) * | 2010-09-28 | 2012-04-04 | Yamaha Corporation | Tone generating style notification control for wind instrument having mouthpiece section |
| WO2016018425A1 (en) * | 2014-08-01 | 2016-02-04 | Jbl Radical Innovations, Llc | Closed system heated mouthpiece activates a formulation |
| CN105989820A (en) * | 2015-03-19 | 2016-10-05 | 卡西欧计算机株式会社 | Electronic wind instrument |
| JP2017015809A (en) * | 2015-06-29 | 2017-01-19 | カシオ計算機株式会社 | Reed member, mouthpiece, and electronic wind instrument |
| US10170091B1 (en) * | 2017-06-29 | 2019-01-01 | Casio Computer Co., Ltd. | Electronic wind instrument, method of controlling the electronic wind instrument, and computer readable recording medium with a program for controlling the electronic wind instrument |
| EP3422340A1 (en) * | 2017-06-29 | 2019-01-02 | Casio Computer Co., Ltd. | Electronic wind instrument capable of performing a tonguing process |
| JP2019045681A (en) * | 2017-09-01 | 2019-03-22 | カシオ計算機株式会社 | Detection apparatus, electronic musical instrument, detection method, and control program |
| US20210201872A1 (en) * | 2018-05-25 | 2021-07-01 | Roland Corporation | Electronic wind instrument (electronic musical instrument) and manufacturing method thereof |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6222524B1 (en) * | 1997-08-25 | 2001-04-24 | International Business Machines Corporation | Mouth operated input device for an electronically responsive device |
| US20040164881A1 (en) * | 2003-02-21 | 2004-08-26 | Kelvin Kwong Loun Mok | Mouth activated input device for an electronically responsive device |
| US6786863B2 (en) * | 2001-06-07 | 2004-09-07 | Dadt Holdings, Llc | Method and apparatus for remote physical contact |
| EP1760690A2 (en) * | 2005-08-30 | 2007-03-07 | Yamaha Corporation | Apparatus for assisting in playing musical instrument |
| WO2007059614A1 (en) * | 2005-11-23 | 2007-05-31 | Photon Wind Research Ltd. | Mouth-operated input device |
-
2008
- 2008-05-23 WO PCT/CA2008/000986 patent/WO2008141459A1/en not_active Ceased
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6222524B1 (en) * | 1997-08-25 | 2001-04-24 | International Business Machines Corporation | Mouth operated input device for an electronically responsive device |
| US6786863B2 (en) * | 2001-06-07 | 2004-09-07 | Dadt Holdings, Llc | Method and apparatus for remote physical contact |
| US20040164881A1 (en) * | 2003-02-21 | 2004-08-26 | Kelvin Kwong Loun Mok | Mouth activated input device for an electronically responsive device |
| EP1760690A2 (en) * | 2005-08-30 | 2007-03-07 | Yamaha Corporation | Apparatus for assisting in playing musical instrument |
| WO2007059614A1 (en) * | 2005-11-23 | 2007-05-31 | Photon Wind Research Ltd. | Mouth-operated input device |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2434478A3 (en) * | 2010-09-28 | 2012-04-04 | Yamaha Corporation | Tone generating style notification control for wind instrument having mouthpiece section |
| EP2650870A1 (en) * | 2010-09-28 | 2013-10-16 | Yamaha Corporation | Tone generating style notification control for wind instrument having mouthpiece section |
| US8581087B2 (en) | 2010-09-28 | 2013-11-12 | Yamaha Corporation | Tone generating style notification control for wind instrument having mouthpiece section |
| WO2016018425A1 (en) * | 2014-08-01 | 2016-02-04 | Jbl Radical Innovations, Llc | Closed system heated mouthpiece activates a formulation |
| CN105989820A (en) * | 2015-03-19 | 2016-10-05 | 卡西欧计算机株式会社 | Electronic wind instrument |
| US9653057B2 (en) * | 2015-03-19 | 2017-05-16 | Casio Computer Co., Ltd. | Electronic wind instrument |
| JP2017015809A (en) * | 2015-06-29 | 2017-01-19 | カシオ計算機株式会社 | Reed member, mouthpiece, and electronic wind instrument |
| EP3422340A1 (en) * | 2017-06-29 | 2019-01-02 | Casio Computer Co., Ltd. | Electronic wind instrument capable of performing a tonguing process |
| US10170091B1 (en) * | 2017-06-29 | 2019-01-01 | Casio Computer Co., Ltd. | Electronic wind instrument, method of controlling the electronic wind instrument, and computer readable recording medium with a program for controlling the electronic wind instrument |
| CN109215624A (en) * | 2017-06-29 | 2019-01-15 | 卡西欧计算机株式会社 | The control method and storage medium of electronic wind instrument and electronic wind instrument |
| US10297239B2 (en) | 2017-06-29 | 2019-05-21 | Casio Computer Co., Ltd. | Electronic wind instrument capable of performing a tonguing process |
| CN109215624B (en) * | 2017-06-29 | 2023-06-16 | 卡西欧计算机株式会社 | Electronic wind instrument, control method and storage medium for electronic wind instrument |
| JP2019045681A (en) * | 2017-09-01 | 2019-03-22 | カシオ計算機株式会社 | Detection apparatus, electronic musical instrument, detection method, and control program |
| JP7008941B2 (en) | 2017-09-01 | 2022-01-25 | カシオ計算機株式会社 | Detection device, electronic musical instrument, detection method and control program |
| US20210201872A1 (en) * | 2018-05-25 | 2021-07-01 | Roland Corporation | Electronic wind instrument (electronic musical instrument) and manufacturing method thereof |
| US11682371B2 (en) * | 2018-05-25 | 2023-06-20 | Roland Corporation | Electronic wind instrument (electronic musical instrument) and manufacturing method thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2008141459A1 (en) | Mouth-operated input device | |
| CN109559720B (en) | Electronic musical instrument and control method | |
| KR100946680B1 (en) | Improved wireless control device | |
| US7855715B1 (en) | Switch with depth and lateral articulation detection using optical beam | |
| US20120183156A1 (en) | Microphone system with a hand-held microphone | |
| US8994648B2 (en) | Processor interface | |
| US8841537B2 (en) | Systems and methods for a digital stringed instrument | |
| US7897866B2 (en) | Systems and methods for a digital stringed instrument | |
| US20020061217A1 (en) | Electronic input device | |
| US8169401B2 (en) | Haptic interface | |
| US11460293B2 (en) | Surface quality sensing using self-mixing interferometry | |
| US20140340326A1 (en) | Drawing apparatus and drawing system | |
| CN112513974B (en) | Input device with variable tension of joystick along with travel when operating musical instrument and method of using the same | |
| KR20050031458A (en) | Data input device for individuals with limited hand function | |
| WO2007059614A1 (en) | Mouth-operated input device | |
| JP2012503244A (en) | Device worn on finger, interaction method and communication method | |
| US11868532B2 (en) | Shape changeable apparatus, shape control method, and tactile sense providing apparatus | |
| KR102319228B1 (en) | Electronic device, sensing method of playing string instrument and feedback method of playing string instrument | |
| KR20250063648A (en) | Method and system for enhancing physical abilities in extended reality using physical characteristics | |
| US20060191397A1 (en) | Button of wind instrument | |
| GB2633997A (en) | Gesture-based audio syntheziser controller | |
| US20180246586A1 (en) | Electronic Input Device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 08748341 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 08748341 Country of ref document: EP Kind code of ref document: A1 |