NL2029031B1 - Nose-operated head-mounted device - Google Patents

Abstract

Methods and systems are disclosed for hands-free operation of a head- mounted device using a nasal input signal. The method comprises receiving a signal from a sensor, where the signal comprises a nasal electromyography (EMG) component representing an activity of a nasal muscle. Preferably, the signal represents a voltage over the nasal muscle. The method further comprises providing the signal to an input of a neural network, the neural network being trained to classify the input signal into one of a plurality of predefined categories, each category corresponding to a gesture associated with a control command; and operating the head-mounted device based on the control command associated with the determined gesture.

Description

NL33204/Sv-TD

Nose-operated head-mounted device

Field of the invention

The invention relates to operating a head-mounted device using a nasal electromyography signal; and, in particular, though not exclusively, to methods and systems for operating a head-mounted device using a nasal electromyography signal and a computer program product enabling a computer system to perform such methods.

Background of the invention

Head-mounted devices (HMDs), such as heads-up displays (HUDs), are increasingly popular. Such devices may be used for, e.g., virtual-reality (VR) and augmented- reality (AR) applications. Well-known examples are Oculus Rift (VR) and Google Glass (AR).

These devices are typically controlled by a control panel on the device itself, e.g., a touch surface such as a track pad, or by external devices such as a ring or joystick. In many cases, however, it is advantageous to operate the head-mounted device in a hands-free manner.

That allows a user to perform a primary physical task that requires both hands, while the user is also able to effectively interact with the head-mounted device.

Several methods for hands-free operation of a head-mounted device are known in the art. They typically use sensors to detect electro-encephalogram (EEG) signals (i.e., signals caused by brain activity); electro-oculogram (EOG) signals (i.e., signals caused by eye movement) and/or gaze/pupil tracking; and/or electromyogram (EMG) signals (i.e., signals caused by muscle contractions).

For example, US 2006/0061544 A1 discloses a method for inputting keys using biological signals, in which EOG-based eye-movement tracking is used to move a cursor. This is combined with an EMG signal provided by jaw muscles to provide a 'select' or ‘affirm’ command. However, such a device requires a relatively large amount of sensors at specific locations, severely limiting the form factor of the product. Furthermore, by using jaw muscles, such a system is bound to provide a high number of false positives when the user is, for example, eating or chewing gum, limiting potential applications.

US 2017/0060256 A1 discloses a facial gesture controller for controlling an electronic device based on measuring EMG signals caused by facial expressions, e.g. raised eyebrows, a wrinkled nose, or a smirk. However, the document does not disclose how the

EMG signals may be processed in order to reliable generate control signals.

There is therefore a need in the art for a method for efficient and reliable, hands-free interaction with a head-mounted device that reduces or at least mitigates the aforementioned drawbacks.

Summary of the invention

In a first aspect, the invention may relate to a method for hands-free operation of a head-mounted device using a nasal input signal. The method comprises receiving a signal from a sensor, where the signal comprises a nasal electromyography (EMG) component representing an activity of a nasal muscle. Preferably, the signal represents a voltage over the nasal muscle. The method further comprises providing the signal to an input of a neural network, the neural network being trained to classify the input signal into one of a plurality of predefined categories, each category corresponding to a gesture associated with a control command; and operating the head-mounted device based on the control command associated with the determined gesture.

This way, a head-mounted device may be operated using nasal gestures produced by nasal flaring. As used herein, nasal flaring is defined as the act of voluntarily contracting one’s nasal muscles, and includes e.g. wrinkling of the nose. The use of the nose as an input provider is particularly advantageous, as a head-mounted device (HMD) will practically always be in contact with the user's nose via the nose-bridge pads, so nasal sensors may be incorporated essentially regardless of the shape or design of the HMD.

Furthermore, most people are capable of voluntarily activating their nose muscle, while this muscle is not commonly used for other activities (such as eating or talking). Although the nasal muscle may be used for some more or less common facial expressions, such as disgust, the nasal activity associated with such expressions can easily be distinguished from nasal input signals, when a properly trained neural network is employed. The trained neural network may further reduce false positives from, e.g., sneezing or other involuntary nasal activity. Moreover, the network can be calibrated to the individual user, thus further increasing its accuracy.

The nasal muscle is preferably the nasalis muscle, also known as the transverse nasalis. However, the signal component may, additionally or alternatively, represent an activity from a different nasal muscle, such as the dilator naris, compressor narium, alar nasalis or anomalous nasi muscles. The nasal muscle is typically a human nasal muscle.

It is a further advantage of the claimed method that it may be used in a socially acceptable way. This relates both to the hardware that may be used to implement the method and to the nasal gestures themselves. Regarding the hardware, the HMD does not need to be bulky due to the limited number of sensors required and their placement: in a typical embodiment, one sensor on either side of the nose suffices. The gestures can be very subtle and do not require contact with another body part.

Compared to methods based on eye-tracking, the claimed method is also safer, as the user interaction is not limited by the direction of the eye gaze. Thus, the user can keep his eyes on more important sights, such as traffic, a patient being operated upon, et cetera. This is particularly important for augmented-reality applications.

It is a further advantage of this method that it enables the device in real-time, i.e., without a noticeable delay. As used herein, real-time may refer to a reaction time of less than 0.05 s, preferably less than 0.02 s. The reaction time may refer to the time required for processing the signal, i.e., the time for classifying the signal into a category corresponding to a (nasal) gesture and, where applicable, filtering the signal. The reaction time typically does not include a delay that may be implemented after a signal to see whether an additional signal is provided; such a delay may be required to differentiate between, e.g., a single flare (contracting the nasal muscle once) and a double flare (contracting the nasal muscle twice).

In this context, electromyography relates to the detection and processing or processing of electrical activity produced by skeletal muscles. Thus, electromyography detects the electric potential generated by muscle cells when these cells are electrically or neurologically activated.

In an embodiment, the method further comprises filtering the received signal to reduce or eliminate signal components not related to the activity of the nasal muscle. In general, the received signal may comprise signal components related to physical activity of the user, such as muscle activity from other (facial) muscles, brain activity, the movement of the eyeballs, et cetera, as well as signal components related to environmental factors, such as a 50 Hz signal related to the power grid. Filtering may reduce the contribution of these other signal components, leading to a higher Signal-to-Noise Ratio (SNR), resulting in a clearer and therefore easier and more reliably to classify signal, and thus in a more reliable control of the head-mounted device.

Filtering the signal may comprise filtering the signal using a high-pass filter, preferably the high-pass filter having a threshold value of more than 60 Hz, preferably at least 70 Hz, more preferably at least 80 Hz. This way, irrelevant parts of the signal may be removed, such as EEG signals caused by brain activity, and involuntary EMG signals such as eye-blinks. Alternatively or additionally, filtering the signal may comprise applying a noise reduction algorithm to the signal, preferably a wavelet noise reduction algorithm. Noise reduction may result in an easier and/or more accurately classified signal.

In an embodiment, the neural network is a one-dimensional convolutional neural network, preferably a deep one-dimensional neural network. Such a network has been shown to be particularly effective for classifying nasal EMG signals.

The number of predefined categories, corresponding to (nasal) gestures, into which the neural network classifies the input signals may vary, depending on, e.g., use case.

In an embodiment, the plurality of categories comprises at least three categories, preferably at least four categories. Each one of the plurality of categories may correspond to one of: no voluntary nasal motion, a single nasal flare, a double nasal flare, and a prolonged nasal flare.

This way, two, three, or four commands may be given by using an easy to detect and easy to perform voluntary nasal muscle contraction.

A nasal flare may be associated with a contraction of a nasal muscle, particularly the nasalis muscle, for a predetermined duration, for example between 0.1 and 0.4 seconds, preferably between 0.2 and 0.3 seconds. A long nasal flare may be associated with a muscle contraction that is substantially longer, e.g., at least 0.5 seconds. In an embodiment, the duration may be customised for each user. For example, a user may be asked to provide a number of sample input gestures during an initialisation phase. The customisation may be reflected in parameters of the neural network. In an embodiment, the parameters may be updated based on a periodic or continuous analysis of user input. The parameters can be updated periodically, for example, when the reliability of the neural network appears to be decreasing, upon request of the user, or for a new user (in the case two or more users share the same HMD).

In an embodiment, the method may further comprise operating a further device, preferably a remote device, the further device being communicatively coupled to the head-mounted device. This way, the (remote) further device may be operated in an efficient, reliable, and hands-free manner, via the head-mounted device. The further device can be a so-called internet-of-things (loT) device. The further device can be, for example, a thermostat, a lamp, a television, a lock, or any other IoT-enabled or interconnected device.

In a second aspect, the invention may relate to a method for training a neural network, preferably a convolutional neural network, more preferably a one-dimensional convolutional neural network, most preferably a deep one-dimensional convolutional neural network, to classify an input signal into one of a plurality of predefined categories. Each category from the plurality of predefined categories may correspond to a gesture from a plurality of predefined gestures, preferably nasal gestures. The method comprises receiving training data and using the training data to train the neural network to classify sensor data into one of the plurality of predefined categories. The training data may comprise sets of sensor data representing activity of a nasal muscle. Preferably, a set of sensor data includes a voltage over the nasal muscle. Each set of sensor data may be associated with a gesture from the plurality of predefined gestures.

In another aspect, the invention may relate to a system for operating a head- mounted device using a nasal electromyography (EMG) signal, the system comprising a 5 sensor for obtaining a signal from a nasal muscle, the signal representing a voltage over the nasal muscle, and a computer-readable storage medium having computer readable program code embodied therewith, the program code including at least one trained neural network, and at least one processor, preferably a microprocessor, coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code the atleast one processor is configured to perform executable operations. The executable operations may comprise: receiving a signal from a sensor, the signal comprising a nasal electromyography (EMG) component representing an activity of a nasal muscle, preferably the nasalis muscle, the signal preferably representing a voltage over the nasal muscle; providing the signal to an input of a neural network, the neural network being trained to classify the input signal into one of a plurality of predefined categories, each category corresponding to a gesture associated with a control command; and operating the head- mounted device based on the control command associated with the determined gesture.

Such a system may implemented the methods described above.

In an embodiment, the system comprises at least two sensors, preferably a first and a second sensor configured to contact either side of a user's nose and a third sensor configured to contact the user's mastoid process. The first and second sensors may be included in the nose bridge of an HMD. The third sensor may be used as a reference sensor. The third sensor may be placed at least 2 cm from the nose, preferably at least 5 cm.

Advantageously, when in use, the third sensor may be positioned on an anatomical region with little muscle activity, such as the mastoid process. The third sensor may be included in a temple or temple tip of the HMD.

In an embodiment, the sensors are included in a sensor device different from the head-mounted device, and communicatively coupled to the head-mounted device. This way, a head-mounted device without (nasal) sensors can be adapted to implement the claimed method. The hardware and/or software for classifying the sensor input into gestures may be included in the sensor device, in the head-mounted device, or distributed between the two devices. In a different embodiment, the system is a single head-mounted device, preferably a head-mounted display device.

In an embodiment, the system further comprises a further device, preferably a remote device, communicatively coupled to the head-mounted device, the system being configured to operate the further device based on the determined category.

In an aspect, the invention may relate to an input device configured to be attached to a nose bridge of a head-mounted device. The input device may comprise a first sensor and a second sensor. When in use, the first and second sensors may be configured to contact either side of a user's nose. The first and second sensors may be configured to obtain an input signal from a nasal muscle, the input signal representing a voltage over the nasal muscle. The input device further comprises a communication interface for providing the input signal, or a signal derived from the input signal, to the head-mounted device. As was described above, such an input device may be used to convert a head-mounted device without nasal gesture input in a device suitable for nasal gesture input.

In an embodiment, the input device further comprises a third sensor, the third sensor being configured to provide a reference signal, preferably the third sensor being configured to contact the user's mastoid process. The use of a reference signal may lead to a clearer, and hence easier to classify signal.

One aspect of this disclosure relates to a computer comprising a computer readable storage medium having computer readable program code embodied therewith, and a processor, preferably a microprocessor, coupled to the computer readable storage medium, wherein responsive to executing the computer readable program code, the processor is configured to perform any of the methods described herein.

One aspect of this disclosure relates to a computer program or suite of computer programs comprising at least one software code portion or a computer program product storing at least one software code portion, the software code portion, when run on a computer system, being configured for executing any of the methods described herein.

One aspect of this disclosure relates to a non-transitory computer-readable storage medium storing at least one software code portion, the software code portion, when executed or processed by a computer, is configured to perform any of the methods described herein.

As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, a method or a computer program product.

Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro- code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system." Functions described in this disclosure may be implemented as an algorithm executed by a processor/microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied, e.g., stored, thereon.

Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

More specific examples of a computer readable storage medium may include, but are not limited to, the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fibre, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of the present invention, a computer readable storage medium may be any tangible medium that can contain, or store, a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fibre, cable, RF, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java(TM), Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

Aspects of the present invention are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor, in particular a microprocessor or a central processing unit (CPU), of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer, other programmable data processing apparatus, or other devices create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Moreover, a computer program for carrying out the methods described herein, as well as a non-transitory computer readable storage-medium storing the computer program are provided. A computer program may, for example, be downloaded (updated) to the existing data processing systems or be stored upon manufacturing of these systems.

Elements and aspects discussed for or in relation with a particular embodiment may be suitably combined with elements and aspects of other embodiments, unless explicitly stated otherwise. Embodiments of the present invention will be further illustrated with reference to the attached drawings, which schematically will show embodiments according to the invention. It will be understood that the present invention is not in any way restricted to these specific embodiments.

Brief description of the drawings

Fig. 1A and 1B depict head-mounted devices according to an embodiment;

Fig. 2A-C schematically depict nose-operated input devices for a HMD according to embodiments;

Fig. 3 depicts a device for operating a further (remote) device using nasal gestures, according to an embodiment;

Fig. 4A and 4B depict a neural-network-based classifier to classify nasal input gestures for use in an embodiment;

Fig. 5 depicts a signal processing steps for operating a head-mounted device according to an embodiment;

Fig. 6 depicts a method for hands-free operation of a head-mounted device using a nasal input signal according to an embodiment; and

Fig. 7 is a block diagram illustrating an exemplary data processing system that may be used for executing methods and software products described in this application.

Detailed description

The embodiments in this disclosure describe a method for hands-free operation of a Head-Mounted Device (HMD) based on nasal gestures, as well as devices for performing such methods.

The described devices and methods for operating them allow for socially acceptable hands-free operation of an HMD. This relates both to the devices to be operated and to the operation method and gestures themselves. Regarding the hardware, the HMD does not need to be bulky but can have almost any form factor. The embodiments are therefore compatible with many currently available and announced HMDs. The only restriction is that at least a part of the device is in contact with the user's nose; in a typical embodiment, one sensor on either side of the nose suffices. Nevertheless, in other embodiments more sensors may be used and they may be positioned at other convenient locations such as near a user's ear. The gestures can be very subtle and unobtrusive, and do not require contact with another body part.

Fig. 1A depicts a head-mounted device according to an embodiment. The head-mounted device (HMD) 100 comprises two sensors 10242, which may be integrated in the bridge 104 of the HMD. The sensors are for example electrodes for measuring a voltage.

Such electrodes are generally known in the art and may be known as electromyogram (EMG) sensors. When in use, a first electrode obtains an electric signal from a first side of the nose 1124, and a second electrode obtains an electric signal from a second side of the nose 1122. The electrodes may be adapted to detect at least EMG signals caused by contractions of a nasal muscle. The HMD may comprise a third sensor (not shown) which may provide a reference signal. Preferably, a reference sensor is placed on an anatomical region with little muscle activity, such as the mastoid process. The reference sensor may be integrated in, e.g., a temple tip (not shown) of the HMD. Other embodiments may use a different number of sensors or differently placed sensors, although usually at least one sensor is provided against either side of a user's nose.

In the embodiment depicted in Fig. 1A, the electrodes are integrated in the

HMD, in this case in the bridge part. In other embodiments, the electrodes may be included in an add-on device which may be communicatively coupled to the HMD. In these embodiments, the add-on device can either include its own signal filtering units and/or processing unit, or use signal filtering units and/or processing units included in the HMD.

Such an add-on device may typically be physically coupled to the HMD, e.g. attached to the nose bridge of the HMD. The add-on device may be communicatively coupled to the HMD using a wired or wireless data connection.

As will be described in more detail with reference to Fig. 1B, the output of the sensors may be categorised into one of a plurality of predefined categories, each category corresponding to a gesture. A gesture may be associated with a control command. These control commands may be used to control the HMD. Thus, the HMD may be controlled in a hands-free manner, based on nasal gestures. Examples of nasal gestures are a single flare, a double flare, and a long flare, but other nasal gestures leading to other patterns of nasal muscle contraction and relaxation can easily be imagined. Additionally, one category typically corresponds to an absence of a (voluntary) nasal gesture.

Using these control commands, which are based on nasal gestures, a user may interact with a user interface of the HMD. The user interface can be a graphical user interface 106, which may, e.g., be projected in the field of view of the user in front of the

HMD, on a screen or glass of the HMD, or into the user's eye. The commands can be context-sensitive. In some embodiments, the HMD can only or mainly be operated using nasal gestures, whereas in other embodiments, nasal gestures may be used in addition to different input modes, or may be restricted to certain applications.

In the depicted example, the user is browsing through a user-interface menu comprising icons 1084-3, each icon associated with, e.g., an app or a menu option. The user may, e.g., select one of the icons to launch the associated app. The app currently in focus may be visually marked, e.g., using icon colour, icon size, and/or other markers. The user may change the focus or select the icon in focus by providing nasal input gestures, e.g., a single flare to move the focus to the next app and a double flare to select the app in focus.

Additional gestures may be mapped to additional functions, e.g., a long flare for returning to the previous menu, and/or not providing an input signal for a predetermined amount of time to exit the user interface.

The use of the nose as an input provider is particularly advantageous, as the

HMD will practically always be in contact with the user's nose (when the HMD is in use), so that nasal sensors may be incorporated essentially regardless of the shape or design of the

HMD. A typical sensor placement would be in the nose bridge. Furthermore, most people are capable of voluntarily activating their nose muscle, while this muscle is not commonly used for other activities (such as eating or talking).

The nasal muscle whose activity is monitored is preferably the nasalis muscle, also known as the transverse nasalis. However, the signal component may, additionally or alternatively, represent an activity from a different nasal muscle, such as the difator naris, compressor narium, alar nasalis or anomalous nasi muscles. The nasal muscle is typically a human nasal muscle, although applications to non-humans such as animals are not excluded.

The described methods and systems can be combined with additional input modalities for improving input quality (e.g., speed and accuracy) and input options (e.g., number of supported gestures). To this end, the HMD may additionally comprise one or more pressure sensors (not shown) arranged to be in contact with the user's nose when the HMD is worn. Preferably, the one or more pressure sensors are embedded in the nose bridge. A pressure sensor may reduce false positives (e.g., when eating and during facial expressions), for example by enabling EMG sampling only when significant pressure is exerted. Thus, the pressure sensor may also conserve energy.

Additionally, the described methods and systems can be utilized for complementing other HMD input methods such as eye-tracking or speech recognition. For example, the single flare gesture can be more reliable and safer than eye-blinks, when it comes to implementing the selection gesture during eye-tracking input on HMDs. By using a nose gesture to activate speech recognition, the HMD does not need to record and parse sound input continuously, but may do so selectively when activated. This way, energy may be conserved and a greater sense of privacy can be obtained.

Fig. 1B schematically depicts a head-mounted device 120 as depicted in Fig. 1A. One or more sensors 122 detect nasal activity and provide an output signal representative for the detected nasal activity. The output signal is preferably a nasal EMG signal representing activity of a nasal muscle. In a typical embodiment, the one or more sensors provide a continuous output signal. The output signal may be a representation of the voltage difference between two sensors, which can be positioned, e.g., on either side of the user's nose. In other embodiments, an additional sensor, placed further away from the nose, e.g., near a user's ear, may provide a reference signal.

The sensor output signal may be sampled using a predetermined sampling rate. The sampling rate should be sufficiently high to detect EMG signals. The sampling rate may be at least 150 Hz, preferably at least 180 Hz, more preferably at least 200 Hz.

The output signal may comprise a multitude of signal components from other sources than nasal activity, such as electro-encephalogram (EEG) signals (i.e., signals caused by brain activity); electro-oculogram (EOG) signals (i.e., signals caused by eye movement); and/or electromyogram (EMG) signals (i.e., signals caused by muscle contractions) from other muscles than the nasal muscle. The output signal may further comprise noise components caused by environmental electromagnetic radiation, e.g., a 50 Hz or 60 Hz signal component from the power grid.

Therefore, the device may comprise an optional filtering unit 124 to filter the sensor's output signal. EMG signals typically have a higher frequency than EEG signals.

Consequently, the filtering unit may comprise a high-pass filter. The high-pass filter may have a threshold of more than 60 Hz, preferably at least 70 Hz, more preferably about 80 Hz.

Alternatively, a band-pass filter may be used with a lower threshold of more than 60 Hz, preferably at least 70 Hz, more preferably about 80 Hz. The upper threshold of the band- pass filter may be at least 100 Hz. Signal filtering may reduce the contribution of these other signal components, resulting in a clearer and therefore easier and more reliably to classify signal, and thus in a more reliable control of the head-mounted device. Signal filtering may also lead to a simpler and/or easier to train classifier.

Additionally or alternatively, the filtering unit may comprise a noise filter, for example a wavelet-based noise filter. The noise filter may remove or reduce high-frequency noise from the detected signal and improve the signal-to-noise ratio. Noise reduction may result in a signal that is easier to classify and/or may be classified more accurately.

The filtered signal may still comprise signal components that are not related to voluntary nasal gestures. These signals components may, e.g., be associated with more or less common facial expressions, such as disgust, or with involuntary activity such as sneezing or yawning. Therefore, the filtered signal may be analysed by a classification unit 126 comprising a neural network, preferably a deep convolutional neural network, trained to detect and classify signals related to voluntary nasal gestures. The input signal may be classified (or categorised) into one of a plurality of predefined categories. Each category may correspond to a gesture associated with a control command. This association may be context-dependent, that is, a different control command may be associated with the same gesture, depending on the context, e.g., the available menu options. In some cases, a gesture may be associated with the same control command associated with a different gesture in a different context.

The input signal may be discretised, both with regards to time and amplitude, before the signal is provided to an input of the neural network. The discretisation may take place before or after the filtering.

In embodiments with a more than two nose sensors or with at least two nose sensors and a reference sensor, signals form the plurality of sensors may be combined, e.g. subtracted and amplified, before being provided to the neural network. Alternatively, the neural network may be trained to accept a plurality of input signals.

The neural network may be based on a one-dimensional convolutional network. The neural network can be calibrated to the user, increasing accuracy. The neural network will be described in more detail with reference to Fig. 4A and 4B.

The output of the neural network may comprise an indication of a category or label indicating which input signal, i.e., which nasal gesture, was detected. Based on the indicated category, an input signal may be provided to a user interface 128 of the operating system of the HMD. The input signal may be interpreted by the operating system as a control command. Thus, the HMD may be operated based on the control command determined based on the (filtered and classified) input signal obtained from the nasal muscle activity, and hence based on a nasal gesture.

The HMD may further comprise a processor 130 for executing one or more method steps of the nasal gesture input method, for example for running the classification unit. The HMD may also comprise a memory 132, coupled to the processor, for storing executable instructions. The same processor and memory may be used for classifying and, optionally, filtering the sensor data that are also used for running other software on the HMD.

Alternatively, dedicated hardware can be used.

In an embodiment, the sensors are included in a sensor device different from the head-mounted device. The sensor device may be communicatively coupled to the head- mounted device, e.g. using a wired or wireless connection and a suitable communication protocol. This way, a head-mounted device without (nasal) sensors can be adapted to implement the claimed method. The hardware and/or software for classifying the sensor input into gestures may be included in the sensor device, in the head-mounted device, or distributed between the two devices. In a different embodiment, the system is a single head- mounted device, preferably a head-mounted display device.

Fig. 2A-C schematically depict nose-operated input devices for a HMD according to embodiments. In these embodiments, at least the sensors are not integrated in the HMD, but are instead integrated in an input device which is external to the HMD and which can be communicatively coupled to the HMD, e.g. using the HMD's USB-C port. An advantage of the USB-C port is that it may be used both for data transfer and for power supply. The input device may be 3D-printed, allowing for highly customisable and comfortable designs, or manufactured in other suitable ways.

Fig. 2A depicts an embodiment in which the input device 202 comprises sensors 206 for obtaining input signals indicative for activity of the nasal muscle. The sensors may include a reference sensor for obtaining a reference signal. The input device is communicatively coupled to the HMD 204, e.g., via a communication interface. The HMD comprises a filtering unit 208 and a neural-network-based classifier 210 as described above with reference to Fig 1. The output of the classifier may be used to navigate or control the

HMD’s user interface 212. The HMD comprises a processor 214 and a memory 216 for executing the classifier. The coupling to the HMD may comprise a wire for transferring a sensor signal. When active sensors are used, the input device may be electrically coupled to the HMD to receive power from the HMD.

As off-the-shelf HMDs typically do not include dedicated signal filtering hardware suitable for filtering the nose-sensor input, in this embodiment, the filtering unit may be implemented as a software implementation.

Fig. 2B depicts an embodiment in which the input device 222 comprises sensors 226 for obtaining input signals indicative for activity of the nasal muscle and a filtering unit 228. The input device is communicatively coupled to the HMD 224, comprising a neural-network-based classifier 230. The output of the classifier may be used to navigate or control the HMD’s user interface 232. The HMD comprises a processor 234 and a memory 236 for executing the classifier. The input device is communicatively coupled to the HMD, for instance using a wire for transferring a filtered sensor signal. When active sensors and/or active filters are used, the input device may be electrically coupled to the HMD to receive power from the HMD.

An advantage of this embodiment is that the input device comprises the components required to change an ordinary head-mounted device into a nose-operated

HMD, while making maximum use of the HMD's capabilities. Thus, this embodiment is economical and flexible. Hardware signal filtering is typically faster than a software implementation.

Fig. 2C depicts an embodiment in which the input device 242 comprises sensors 246 for obtaining input signals indicative for activity of the nasal muscle, a filtering unit 248, and a neural-network-based classifier 250. The input device further comprises a processor 252 and a memory 254 for executing the classifier. The input device is communicatively coupled to the HMD 244, for instance using a wire for transferring a command signal. The output of the classifier may be used to navigate or control the HMD's user interface 252. The input device may further comprise a power supply and/or may be electrically coupled to the HMD to receive power from the HMD.

An advantage of this embodiment is that all components may be optimised for the nasal signal input. This may be particularly advantageous if the HMD does not comprise dedicated hardware for running a convolutional neural network.

Fig. 3 depicts a device for operating a further device, in this case a remote device, using nasal gestures, according to an embodiment. An input device 302 is used to detect nasal gestures. The input device may comprise sensors 3044; to obtain a signal representing nasal muscle activity. In the embodiment depicted in Fig. 3, the input device is a separate add-on device communicatively coupled 306 to a HMD 310. In other embodiments, the input device may be included in the HMD. The HMD, in turn, may be communicatively coupled to the further device, for instance a (so-called ‘smart’) thermostat 320. Typically, the HMD is coupled to the further device using a wireless protocol such as

RFID, Bluetooth, Zigbee, Wi-Fi or LoRaWan, etc. However, wired solutions are not excluded.

Many small devices such as thermostats already have an interface designed for two or three input buttons, making them easy to operate using a limited set of gestures.

Thus, the input to the HMD can be tethered to ‘smart’ devices, e.g., smart- home devices. That way, the user can perform the same interaction gestures used to operate the HMD to interact with smart-home devices, using the HMD as a proxy. For example, the user may perform a single flare to turn on the HMD and may use subsequent single flares to navigate (e.g., scroll through a menu) to a smart-home app, followed by a double flare, and launch the smart-home app. The smart-home app may enable the user to select which devices they would like to control (e.g., smart TV, smart thermostat, smart lights, etc.). In the example of Fig. 3, the user interacts with a smart thermostat.

As an example, the user may then use single flares to navigate to the desired device option by traversing a list of available smart-home devices. Here, the user may perform single flares, followed by a double flare gesture, to navigate to and select the smart thermostat device. With subsequent single flares, the user can switch between “=” and “+”

icons, whereas with double flares, the user can either decrease or increase the desired smart-home thermostat temperature, respectively. Finally, the user can either: — perform no action for ~3 seconds, allowing the display to turn off due to inactivity and exit to the main navigation menu; — perform a long flare exiting the smart home app and continue browsing through other apps in the main navigation menu; — perform 2 subsequent long flares: 1 long flare for exiting to the main navigation menu and 1 long flare for turning off the display.

Thus, the input device may be used to control the HMD, and, via the HMD, the input device may be communicatively coupled to a further device such as a smart device.

The HMD may be configured to operate the further device based on the input received from the input device, more in particular based on the determined category identifying a detected nasal input gesture. Evidently, other implementations can use different gestures, or map the same gestures on different actions than the ones mentioned above.

Fig. 4A and 4B depict a neural-network-based classifier to classify nasal input gestures. Fig. 4A schematically depicts an example of a 1D deep neural network architecture for use in the methods and systems for nasal operation of a user interface as described in this application. The input is a 1D array representing the input signal 400, possibly filtered in an earlier step. In this example, the input comprises 200 entries encoding a second’s worth of measurement data.

A 1D convolutional layer 402 may be applied to the input to obtain a number of 1D output frames, in this example 16 frames of 200 elements each. A further 1D convolutional layer 404 may take the output of the previous convolutional layer as input, in this example resulting in 16 frames of 204 elements each. A maxpool layer 406 may receive the output of the further convolutional layer as input and may reduce the dimensionality, in this case resulting in an output comprising 16 frames of 68 elements each.

In the depicted example, a similar group of layers, comprising a 1D convolutional layer 408, a further 1D convolutional layer 410, and a maxpool layer 412, was applied to the output of the first maxpool layer 406, resulting in 64 frames of 24 elements each. The parameters in this second group were chosen slightly differently from those in the first group, resulting in a four-fold multiplication of the number of frames in the first step instead of a sixteen-fold multiplication.

Two dense (fully-connected) layers 414,416 were applied to the output of the second group, resulting in four categories 418 to each of which a probability may be assigned.

Fig. 4B depicts the neural network depicted in Fig. 4A. The network may be implemented using 1D convolutional layers (1D CNN). The convolutions may use an activation function, for example a ReLU function, or a Leaky ReLU, tanh, sigmoid, or softmax function. A plurality of 1D convolutional layers, 424-430, may be used wherein minor variations in the number of layers and their defining parameters, e.g., differing activation functions, kernel amounts, use of subsampling and sizes, and additional functional layers such as dropout layers may be used in the implementation without losing the essence of the design of the deep neural network. One-dimensional convolutional neural networks such as described here have been shown to be particularly effective for classifying nasal EMG signals with a high accuracy.

In part to reduce the dimensionality of the internal representation of the data within the deep neural network and to reduce the risk of overfitting, one or more 1D max pooling layers 432,434 may be employed. At this point in the network, the internal representation may be passed to one or more densely-connected layers 436 aimed at being an intermediate for translating the representation in the sensor data to activations of potential labels, in particular gesture labels.

The final or output layer 438 may have the same dimensionality as the desired number of encoded labels and may be used to determine an activation value (analogous to a prediction) per potential label or gesture 440. The activation value may be determined by an activation function, e.g., a softmax function or one of the above-mentioned alternatives.

The network may be trained making use of a dataset with as input for the 1D

CNN layers a pre-processed, e.g., filtered, dataset of 1D data 422, i.e., a 1D digitised representation of sensor data as described with reference to Fig. 5. For each input sample (being a representation of a single gesture) a matching representation of the correct label 442 may be used to determine a loss between desired and actual output 440. This loss may be used during training as a measure to adjust parameters within the layers of the deep neural network. Optimizer functions may be used during training to aid in the efficiency of the training effort. The network may be trained for any number of iterations until the internal parameters lead to a desired accuracy of results. When appropriately trained, an unlabeled sample may be presented as input and the deep neural network may be used to derive a prediction for each potential label. For example, the network may be trained to achieve at least 90 % accuracy, preferably at least 95 %. As an example, a training method as described on the PyTorch website: https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial. html#train-the-network.

The training data set should preferably comprise various examples of each nasal gesture. To increase the amount of available training data, the training data set can be augmented by, e.g., shifting, possibly cyclically shifting obtained nose signals. Preferably, the training data set should also contain samples representing non-voluntary nasal activity which could lead to false positives, such as sneezing, nose blowing, nose rubbing, or an expression of disgust.

Hence, as the deep neural network is trained to classify a sensor data sample of a nasal gesture (or the absence of a gesture) into one of a plurality of gestures, e.g., four gestures as described in various examples below, the output of the neural network may be activation values and associated potential gesture labels. The potential gesture label with the highest activation value may indicate to the system that it is most likely that the sensor data sample represents a gesture as indicated by the label. In some cases, the gesture with the highest activation value may be selected as the input gesture regardless of the magnitude of the activation value, whereas in other cases, a gesture is only accepted if the activation value exceeds a predetermined threshold value, the input signal being classified as ‘no gesture’ if none of the labels exceeds the threshold. Which case is selected, and/or the value of the threshold, may be context-dependent; for example, when a user is navigating a menu, the threshold value may be lower than when a user is not yet navigating a menu. Similarly, when the user is about to activate a function that is marked by the user interface as ‘important’ (e.g., irretrievable deletion of information, confirmation of payment), a higher threshold may be used than for other tasks (e.g., selecting a next or previous item when browsing a list) in order to combine ease of use with security.

Before use, the trained neural network may be calibrated to the specific input of a user by having the user input gestures on demand.

The neural network may further be trained to identify the user, for example using the method described in Kurogi et al., ‘A study on a user identification method using dynamic time warping to realize an authentication system by s-EMG’, in: Barolli et al. (eds),

Advances in Internet, Data & Web Technologies. EIDWT 2018. Lecture Notes on Data

Engineering and Communications Technologies, vol 17 (Springer, Cham; 2018), which is hereby incorporated by reference. This way, the device may be configured to operate only when being worn by an authorised user. Alternatively, some operation may be available for all users while other operations are available only for authorised users.

Fig. 5 depicts a signal processing steps for operating a head-mounted device according to an embodiment. An HMD 500 comprises two EMG sensors 5024; which are in contact with a user's nose in a so-called bipolar configuration, and a third EMG sensor (not shown) on the mastoid process behind the user's ear, which is used as a reference sensor.

The reference sensor is preferably placed at least a few cm distant from the nasal muscle.

The reference sensor is preferably placed on a bony region, e.g. on the mastoid process and thus embedded in either the left or the right temple tip of the HMD. Other embodiments may use different sensor arrangements. In principle, the EMG sensors provide a continuous signal, representing a voltage with respect to the reference sensor. In the depicted embodiment, the sensor signals 5104, 51212 are sampled periodically with a frequency of 200 Hz, but other embodiments may use different sampling range. Preferably, a sampling rate of at least 150 Hz is used.

At a first time instance, the user may flare his nose 506 in order to operate the

HMD, e.g., to input a ‘select’ command in interaction with a user interface 504. This results in two first sensor signals 5104.2 corresponding to the two sensors. The shown signals correspond to two single (nasal) flares with a roughly 0.7 s pause in between. The signals from the left and right sensors may be provided to a filtering unit 514. In the depicted embodiment, the filtering unit comprises a differential amplifier and a high-pass filter. Other embodiments may comprise additional and/or different components, such as a wavelet noise reduction component. In this example, the signals from the left and right sensors are provided to the differential amplifier which amplifies the difference between the left and right signal. This may reduce noise that is common to both signals. The output of the differential amplifier is provided to a high-pass filter suppressing frequencies below 80 Hz, reducing low- frequency noise. The filtering unit provides a first filtered signal 516 at its output. In other embodiments, the filtering unit can be an analogue filtering unit, and digitizing of the signal may take place only after the filtering unit has filtered the signals. In yet other embodiments, the device may not comprise a filtering unit.

In a typical embodiment, the input is divided into sampling windows of a predefined width, in the depicted example, the sampling window 5221 has a width of 1 s. The division into sampling windows can take place before, after, or in the filtering unit. The sampling windows can be, e.g., consecutive sampling windows 5224; or overlapping sampling windows 5224.5. The width of the sampling windows should be compatible with the duration of the gestures that are to be classified. The width of the sampling window generally equals at least the duration of the fastest gesture.

In the depicted embodiment, three basic nasal gestures are used for interacting with the HMD: a single flare, a double flare, and a long flare; any input not classified into a category associated with one of these gestures is discarded. The minimum sampling window is selected to accommodate for simple (single flare), but also complex gestures {long and double flare). Thus, the duration of a single flare (typically about 0.2-0.3 s) may be taken as a lower bound of the sampling window. If a neural network with an internal stored state (a ‘memory’) is used, e.g., a recurrent neural network, the neural network may be trained to classify gestures spanning several sampling windows. Otherwise, a sampling window with a longer duration than a single flare (e.g., about 0.35-0.4 s) may be chosen to accommodate for a potential follow-up gesture, such as another single flare for producing a double flare gesture.

However, the durations of the gestures may vary among different users. To this end, the sampling window, as well as the method and system (i.e., the CNN model), may be customizable and adjusted to the preferences and capabilities of the individual user. To do so, a quick sampling session for all supported gestures may be required in the case of a new user.

The filtered signal is provided to the classification unit 520, comprising a convolutional neural network as described above with reference to Fig. 4. The classification unit classifies the input, in this case, a single flare. Consequently, a signal representing the ‘single flare’ gesture is sent to the HMD’s operating system, where it is translated into a context-specific action, in this case: ‘select’.

At a second time instance, the user may blink his eyes 508 without intending to provide an input signal to the HMD. This results in two second sensor signals 51242, corresponding to the two sensors. The shown signals correspond to two eye blinks with a roughly 0.9 s pause in between. The signals from the left and right sensors are again provided to a filtering unit, resulting in a second filtered signal 518. A sampling window 524 is applied to select a signal with a predefined duration, which is provided to the input of the classification unit. The classification unit recognises that the input signal does not correspond to one of the predefined input gestures, and consequently does not send an input signal to the HMD.

As was mentioned before, the number of predefined categories into which the neural network may classify the input signals may vary, depending on, e.g., use case.

However, adding new nasal gestures corresponding to new categories typically requires retraining the neural network. In an embodiment, the plurality of categories comprises at least three categories, preferably at least four categories. Each one of the plurality of categories may correspond to one of: no voluntary nasal motion, a single nasal flare, a double nasal flare, and a prolonged nasal flare. A nasal flare may be associated with a contraction of a nasal muscle, particularly the nasalis muscle, for a predetermined amount of time, for example between about 0.1 and 0.4 seconds, preferably between 0.2 and 0.3 seconds. This way, two, three, or four commands may be given by using an easy to detect and easy to perform voluntary nasal muscle contraction.

Other embodiments may use additional and/or different nasal gestures, such as triple flares, double prolonged flares, flares with more different duration, nose wrinkles instead of or in addition to nose flares, et cetera. A higher number of different gestures increases the flexibility of the system, while a low number of gestures is easier for the user to remember.

In the following, a number of practical examples of operating an HMD using nasal gestures are discussed.

In a first example, the HMD may be operated using nasal gestures to check a weather forecast. An HMD user flares once (single flare) to activate the HMD. With subsequent single flares, the user browses through the available apps presented on the

HMD’s User Interface (Ul) until the icon of the weather app is in focus. The focus may be associated with some sort of graphical differentiation for the selected app icon provided by the Ul navigation style. For example, the UI allows browsing apps in a rotation style and supports selection by augmenting the icon in focus. A double flare enables the user to select the weather icon in focus (e.g., a sun and a cloud) and launch the corresponding weather app. After the weather app is launched, the user can use single flares to browse through the hourly weather forecast presented in a list. After the user is informed about the weather forecast, they can either: - perform no action for ~3 seconds, allowing the display to turn off due to inactivity and exit to the main navigation menu; - perform a long flare exiting the weather app and continue browsing through other apps in the main navigation menu; or, - perform two subsequent long flares: 1 long flare for exiting to the main navigation menu and 1 long flare for turning off the display.

In a second example, a user operates an HMD using nasal gestures, wherein the HMD is equipped with or connected to a camera. The HMD user flares once (single flare) to activate the HMD. With subsequent single flares, the user browses through the available apps presented on the HMD User Interface until the icon of the camera app is in focus. The user performs a double flare to select the camera icon and launch the camera app. After the camera app is launched, the user can perform single flares to change the camera mode from picture taking to video recording. When the video icon is in focus, the user can perform another double flare to start the video recording. Another double flare may stop the video recording. Alternatively, a long flare stops and exits the camera app. After the user has finished recording a video, they can either: - perform a single flare to select picture taking and instantly capture a pictures by a subsequent double flares; - perform a long flare for exiting the camera app and continue browsing through other apps in the main navigation menu; or, - perform 2 subsequent long flares: 1 long flare for exiting to the main navigation menu and 1 long flare for turning off the display.

This example shows how the command executed in response to receiving a nasal input value may change depending on the context. Possible commands assaciated with nasal gesture are provided in Table 1 below, both when browsing files or icons, and when operating a camera included in or communicatively coupled to the HMD.

Table 1. Context-sensitive interpretation of nasal gestures to operate a HMD nasal gesture browsing filesficons camera app single flare ‘go to next’ ‘toggle photo/video mode’ double flare ‘select’ ‘start/stop picture/video’ long flare ‘exit selection’ ‘exit camera app’ nothing [no input] [no input]

In a third example, a user operates an HMD with nasal gestures in conjunction with a voice-operated personal assistant on the HMD. When the HMD is turned off, the user may perform a long flare to invoke the available personal assistant on the HMD (e.g., Alexa,

Siri, Google Assistant, etc.). After invocation, the HMD displays a “listening icon” for inviting the user to verbally express a query / command. The interaction may be terminated due to no input (e.g., ~3 seconds without user response), or by a long flare that terminates the personal assistant and turns off the HMD.

This way, the personal assistant can easily be turned on and off, saving energy and improving privacy, as the personal assistant does not need to continuously listen to everything that is said within hearing range of the HMD.

Fig. 6 depicts a method for hands-free operation of a head-mounted device using a nasal input signal. As was described in more detail above with reference to Fig. 2A—

C, the method steps may be performed by the head-mounted device, and/or by a device communicatively coupled to the head-mounted device. A first step 602 comprises receiving a signal from a sensor, the signal comprising a nasal electromyography (EMG) component representing an activity of a nasal muscle, the signal preferably representing a voltage over the nasal muscle. Optionally, additional signals from additional sensors may be received.

The additional sensors can be further EMG sensors or different types of sensors, e.qg., pressure Sensors.

An optional step 604 comprises filtering the received signal to reduce or eliminate signal components not related to the activity of the nasal muscle. The filtering may include a high-pass filter or a band-pass filter to suppress other physiological and/or environmental signal components. The filtering may further comprise a noise-suppressing filter such as a wavelet filter.

A step 606 comprises providing the signal to an input of a neural network, the neural network being trained to classify the input signal into one of a plurality of predefined categories, each category corresponding to an input gesture associated with a control command. The neural network can be a 1D deep convolutional neural network as described above with reference to Fig. 4A and 4B. Thus, the neural network may determine one of a set of predetermined nasal gestures, e.g., a single flare, a double flare, or a long flare, based on the received nasal EMG signal.

A step 608 comprises operating the head-mounted device based on the control command associated with the determined gesture. Operating the head-mounted device may comprise interaction with a user interface, possibly a graphical user interface, such selecting and activating menu items.

Fig. 7 is a block diagram illustrating an exemplary data processing system that may be used in embodiments as described in this disclosure. Data processing system 700 may include at least one processor 702 coupled to memory elements 704 through a system bus 706. As such, the data processing system may store program code within memory elements 704. Furthermore, processor 702 may execute the program code accessed from memory elements 704 via system bus 706. In one aspect, data processing system may be implemented as a computer that is suitable for storing and/or executing program code. It should be appreciated, however, that data processing system 700 may be implemented in the form of any system including a processor and memory that is capable of performing the functions described within this specification.

Memory elements 704 may include one or more physical memory devices such as, for example, local memory 708 and one or more bulk storage devices 710. Local memory may refer to random access memory or other non-persistent memory device(s) generally used during actual execution of the program code. A bulk storage device may be implemented as a hard drive or other persistent data storage device. The processing system 700 may also include one or more cache memories (not shown) that provide temporary storage of at least some program code in order to reduce the number of times program code must be retrieved from bulk storage device 710 during execution.

Input/output (I/O) devices depicted as input device 712 and output device 714 optionally can be coupled to the data processing system. Examples of input device may include, but are not limited to, for example, a keyboard, a pointing device such as a mouse, orthe like. Examples of output device may include, but are not limited to, for example, a monitor or display, speakers, or the like. Input device and/or output device may be coupled to data processing system either directly or through intervening I/O controllers. A network adapter 716 may also be coupled to data processing system to enable it to become coupled to other systems, computer systems, remote network devices, and/or remote storage devices through intervening private or public networks. The network adapter may comprise a data receiver for receiving data that is transmitted by said systems, devices and/or networks to said data and a data transmitter for transmitting data to said systems, devices and/or networks. Modems, cable modems, and Ethernet cards are examples of different types of network adapter that may be used with data processing system 700.

As pictured in Fig. 7, memory elements 704 may store an application 718. It should be appreciated that data processing system 700 may further execute an operating system (not shown) that can facilitate execution of the application. Application, being implemented in the form of executable program code, can be executed by data processing system 700, e.g., by processor 702. Responsive to executing application, data processing system may be configured to perform one or more operations to be described herein in further detail.

In one aspect, for example, data processing system 700 may represent a client data processing system. In that case, application 718 may represent a client application that, when executed, configures data processing system 700 to perform the various functions described herein with reference to a "client". Examples of a client can include, but are not limited to, a personal computer, a portable computer, a mobile phone, or the like. In another aspect, data processing system may represent a server. For example, data processing system may represent a server, a cloud server or a system of (cloud) servers.

Various embodiments of the invention may be implemented as a program product for use with a computer system, where the program(s) of the program product define functions of the embodiments (including the methods described herein). In one embodiment, the program(s) can be contained on a variety of non-transitory computer-readable storage media, where, as used herein, the expression “non-transitory computer readable storage media” comprises all computer-readable media, with the sole exception being a transitory, propagating signal. In another embodiment, the program(s) can be contained on a variety of transitory computer-readable storage media. lllustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media (e.g., read-only memory devices within a computer such as CD-ROM disks readable by a CD-ROM drive, ROM chips or any type of solid-state non-volatile semiconductor memory) on which information is permanently stored; and (ii) writable storage media (e.g., flash memory, floppy disks within a diskette drive or hard-disk drive or any type of solid-state random-access semiconductor memory) on which alterable information is stored. The computer program may be run on the processor 1502 described herein.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a" "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising" when used in this specification, specify the presence of stated features,

integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of embodiments of the present invention has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the implementations in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the present invention. The embodiments were chosen and described in order to best explain the principles and some practical applications of the present invention, and to enable others of ordinary skill in the art to understand the present invention for various embodiments with various modifications as are suited to the particular use contemplated.

Claims (17)

CONCLUSIONS A method for hands-free operation of a head wearable device using a nasal input signal, the method comprising: receiving a signal from a sensor, the signal comprising a nasal electromyography, EMG, component showing an activity of a nasal muscle, wherein the signal preferably represents an electrical voltage across the nasal muscle; providing the signal to an input of a neural network, preferably a deep convolutional neural network, the neural network being trained to classify the input signal into one of a plurality of predetermined categories, each category corresponding to a gesture that associated with a control command; and operating the head portable device based on the control command associated with the predetermined gesture. The method of claim 1, further comprising filtering the received signal to reduce or eliminate signal components unrelated to nasal muscle activity. The method of claim 2, wherein filtering the signal comprises filtering the signal using a high-pass filter, the high-pass filter preferably having a threshold greater than 60 Hz, more desirably at least 70 Hz, more desirably at least 80 Hz. The method according to claims 2 or 3, wherein filtering the signal comprises applying a noise reduction algorithm to the signal, preferably a wavelet noise reduction algorithm. The method according to the preceding claims, wherein the neural network is a one-dimensional convolutional neural network, preferably a deep one-dimensional convolutional neural network. The method according to the preceding claims, wherein the plurality of categories comprises at least three categories, preferably at least four categories. The method of the preceding claims, wherein one category of the plurality of categories corresponds to one of: no voluntary nasal movement, single nasal movement, double nasal movement, and sustained nasal movement. The method according to the preceding claims, further comprising operating a further device, preferably a remote device, the further device being communicatively coupled to the head-portable device. 9. A method of training a neural network, preferably a one-dimensional convolutional neural network, to classify an input signal into one of a plurality of predetermined categories, each category corresponding to a gesture from a plurality of predetermined gestures, preferably nasal gestures, the method comprising: receiving training data, the training data comprising sets of sensor data, the sensor data representing activity of a nasal muscle, the sensor data preferably comprising an electrical voltage across the nasal muscle, each set of sensor data associated with a gesture from the plurality of predetermined gestures; and using the training data to train the neural network to classify sensor data into one of the plurality of predetermined categories. 10. A system for operating a head wearable device using a nasal electromyography, EMG, signal, the system comprising: a sensor for acquiring a signal from a nasal muscle, the signal being an electrical voltage across the nasal muscle displays; and a computer-readable storage medium embodying computer-readable program code, the program code comprising at least one trained neural network, and at least one processor, preferably a microprocessor, coupled to the computer-readable storage medium, wherein in in response to executing the computer readable program code, the at least one processor is configured to execute executable instructions comprising: receiving a signal from a sensor, the signal comprising a nasal electromyography, EMG, component representing an activity of a nasal muscle, preferably the nasalis muscle, wherein the signal preferably represents an electrical voltage across the nasal muscle; providing the signal to an input of a neural network, the neural network being trained to classify the input signal into one of a plurality of predetermined categories, each category corresponding to a gesture associated with a control command; and operating the head portable device based on the control command associated with the determined gesture. The system according to claim 10, comprising at least two sensors, wherein the system preferably comprises a first and a second sensor arranged to each make contact with a side of a user's nose and a third sensor arranged is to make contact with the user's mastoid protrusion. The system according to claims 10 or 11, wherein the system is a head-portable device, preferably a head-portable display device. The system according to claims 10 to 12, further comprising a further device, preferably a remote device, communicatively coupled to the head-portable device, the system being arranged to operate the further device based on the particular category. 14. An input device adapted to be attached to the bridge of the nose of a head wearable device, the input device comprising: a first and a second sensor, the first and second sensors being adapted to each contact a side of a nose of a user, the first and second sensors being adapted to obtain an input signal from a nasal muscle, the input signal represents an electrical voltage across the nasal muscle; and a communication interface for providing the input signal, or a signal derived from the input signal, to the head portable device. The input device of claim 14, further comprising a third sensor, the third sensor being arranged to provide a reference signal, the third sensor preferably being arranged to contact the user's mastoid protrusion. A computer program product comprising software code portions adapted to perform, when executed by a computer, the method steps of claims 1 to 9. A computer readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method steps of claims 1 to 9.