HK1262869A1 - System and method for detecting physiological state - Google Patents

Description

System and method for detecting physiological state

Technical Field

The following generally relates to health diagnostics and, more particularly, to an image capture-based system and method for detecting a physiological state.

Background

Remote healthcare services utilize telecommunications and/or technology to provide healthcare-related services from a remote location. It not only expands the opportunity to obtain good quality patient care (especially to provide services to rural areas and people with scarce service resources), but also provides a method to reduce healthcare costs. It is pushing the healthcare delivery mode towards a better direction. According to HIS scale, the number of patients using remote healthcare services will increase in 2018 from about 35 million in 2013 to at least 700 million.

The most common form of remote healthcare service is for a doctor to visit a patient through a video chat platform. However, if the physician wants to collect more vital signs of the patient, such as heart rate, respiratory rate, and blood pressure, various additional equipment and training are required. These devices are invasive, often expensive, and require advance purchase prior to a visit.

Early diagnosis of various pathologies can improve the quality of life and longevity of many patients. One of the pathologies is stress, which has become one of the major health problems. Clinical researchers have found that stress is a major cause of a range of diseases including cardiovascular disease, depression and drug abuse. According to the american society for occupational stress research, operating pressure causes us losses of more than $ 3000 billion per year, including not only costs in healthcare, but also expenses in terms of malpractice, employee turnover, work damage compensation, and insurance.

Currently, there are two main methods to measure the stress level of a subject. The first method relies on self-reporting. Researchers developed various questionnaires to determine the stress level of a patient. A second, and more reliable and accurate, method is to measure physiological characteristics such as blood pressure, vagal tone or salivary cortisol. All of these measures require the use of advanced equipment and specialized training.

Disclosure of Invention

In one aspect, there is provided a system for detecting a physiological state from a sequence of captured images of a subject, the system comprising: a camera configured to capture a sequence of images of a subject, the sequence of images comprising a query set of images; a processing unit trained to determine a set of bit planes (bitplanes) of a plurality of images of the captured sequence of images representing a variation in Hemoglobin Concentration (HC) of the subject and maximizing a signal difference between different physiological states; a classifier trained using a training set comprising HC variations for subjects having known physiological states and configured to: detecting a physiological state of the subject based on the change in HC in the level set; and outputting the detected physiological state.

In another aspect, there is provided a method for detecting a physiological state from a sequence of captured images of a subject, the method comprising: a camera captures a sequence of images of a subject, the sequence of images including a query set of images; a trained processing unit processes the captured sequence of images to determine a set of bit planes for a plurality of images in the captured sequence of images that represent a change in Hemoglobin Concentration (HC) of the subject and maximize signal differences between different physiological states; the set of bit planes is processed using a classifier trained using a training set including HC variations of subjects having known physiological states to: detecting a physiological state of the subject based on the change in HC in the level set; and outputting the detected physiological state.

Drawings

The features of the present invention will become more apparent in the following detailed description with reference to the attached drawings, in which:

FIG. 1 is a block diagram of a transdermal optical imaging system for physiological condition detection;

FIG. 2 shows the re-emission of light from the epidermis and subcutaneous layers of the skin;

FIG. 3 is a set of surface and corresponding transdermal images showing the change in hemoglobin concentration associated with the physiological state of a particular human subject at a particular point in time;

fig. 4 is a plot showing the change in hemoglobin concentration of the forehead of a subject experiencing positive, negative, and neutral physiological states as a function of time (seconds);

FIG. 5 is a plot showing the change in hemoglobin concentration of the nose of a subject experiencing positive, negative, and neutral physiological states as a function of time (seconds);

fig. 6 is a plot showing changes in hemoglobin concentration of the cheek of a subject experiencing positive, negative, and neutral physiological states as a function of time (seconds);

FIG. 7 is a flow chart illustrating a fully automated transdermal optical imaging and invisible physiological state detection system;

FIG. 8 is an exemplary report generated by the system;

FIG. 9 is a diagram of a data driven machine learning system for optimized hemoglobin image combination;

FIG. 10 is a diagram of a data-driven machine learning system for multi-dimensional physiological data model building;

FIG. 11 is a diagrammatic view of an automatic invisible physiological state detection system; and

fig. 12 is a memory cell.

Detailed Description

Embodiments will now be described with reference to the accompanying drawings. For simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. Furthermore, numerous specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the embodiments described herein. Furthermore, this description is not to be taken as limiting the scope of the embodiments described herein.

Unless the context indicates otherwise, the various terms used throughout this specification may be read and understood as follows: as used throughout, "or" is inclusive, as written as "and/or"; as used throughout, singular articles and pronouns include their plural forms and vice versa; similarly, gender pronouns include their corresponding pronouns, such that the pronouns should not be construed as limiting any of the contents described herein to use, implementation, execution, etc. by a single gender; "exemplary" should be understood as "illustrative" or "exemplary" and not necessarily "preferred" relative to other embodiments. Other definitions of terms may be set forth herein; as will be understood by reading this specification, these other definitions apply to the preceding and subsequent examples of those terms.

Any module, unit, component, server, computer, terminal, engine, or device executing instructions exemplified herein can include or otherwise access a computer-readable medium, such as a storage medium, computer storage medium, or data storage device (removable and/or non-removable) such as a magnetic disk, optical disk, or magnetic tape. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, module, or both. Any such computer storage media may be part of or accessible to, or connectable to, a device. Further, any processor or controller set forth herein may be implemented as a single processor or as multiple processors, unless the context clearly dictates otherwise. The multiple processors may be arrayed or distributed, and any processing function mentioned herein may be performed by one or more processors, even though a single processor may be exemplified. Any of the methods, applications, or modules described herein can be implemented using computer-readable/executable instructions that can be stored or otherwise maintained by such computer-readable media and executed by one or more processors.

The following generally relates to physiological diagnostics and, more particularly, to an image capture-based system and method for detecting health-related information, particularly the physiological state of an individual captured in a series of images or videos. The system provides a remote and non-invasive method by which to detect physiological states with high confidence. Many people may access the digital camera for analysis purposes as disclosed herein and may therefore obtain a sequence of images of themselves or others (e.g., family members). Such a sequence of images may be captured by, for example, a webcam, a smartphone front or rear camera, a tablet camera, a traditional digital camera, and so forth. The image sequence may be transmitted to a computing device for analysis over a computer network, removable media, or the like.

The sympathetic and parasympathetic nervous systems respond to stress and pain. It has been found that the blood flow of an individual is controlled by the sympathetic and parasympathetic nervous systems, which is beyond the conscious control of most individuals. Thus, the pressure and pain inherently experienced by an individual can be readily detected by monitoring the blood flow of the individual. The intrinsic pressure and pain system prepares humans to deal with different conditions in the environment by modulating the activation of the Autonomic Nervous System (ANS); the sympathetic nervous system and the parasympathetic nervous system play different roles in the regulation of stress and pain, the former up-regulating the combat-escape response and the latter down-regulating the stress response. Basal pressure and pain have different ANS characteristics. Blood flow in most faces (e.g., eyelids, cheeks, and chin) is controlled primarily by sympathetic vasodilator neurons, while blood flow in nose and ears is controlled primarily by sympathetic vasoconstrictor neurons; in contrast, blood flow in the prefrontal area is innervated by both sympathetic and parasympathetic vasodilation. Thus, different intrinsic physiological states have different spatial and temporal activation patterns in different parts of the face. By capturing hemoglobin data from the system, facial Hemoglobin Concentration (HC) changes in each particular facial region can be extracted. These multidimensional and dynamic data arrays from the individual are then compared to a computational model based on normative data, which will be discussed in more detail below. From this comparison, reliable statistics-based inferences can be made regarding the intrinsic physiological state of the individual. Since the ANS-controlled facial hemoglobin activity is not amenable to conscious control, such activity provides a good window into the true deepest physiological state of an individual.

It has been found that it is possible to isolate the Hemoglobin Concentration (HC) from raw images taken by conventional digital cameras and to correlate the temporal and spatial variation of HC with the physiological state of humans. Referring now to fig. 2, a diagram showing the re-emission of light from the skin is shown. The light (201) travels under the skin (202) and is re-emitted (203) after passing through different skin tissue. The re-emitted light may then be captured by an optical camera (203). The main chromophores that affect the re-emitted light are melanin and hemoglobin. Since melanin and hemoglobin have different color characteristics, it has been found that an image reflecting mainly HC under the epidermis as shown in fig. 3 can be obtained.

The system implements a two-step method to generate a rule suitable for outputting an estimated statistical probability that a physiological state of a human subject belongs to one of a plurality of physiological states and a normalized intensity measure of the physiological state given a video sequence of an arbitrary subject. The physiological states that the system can detect correspond to those physiological states for which the system is trained.

Referring now to fig. 1, a system for physiological data detection is shown, according to an embodiment. The system includes interconnected elements including an image processing unit (104), an image filter (106), and an image classifier (105). The system may also include a camera (100) and a storage device (101), or may be communicatively linked to such a storage device (101): the storage device (101) is preloaded and/or periodically loaded with video imaging data captured from one or more cameras (100). An image classifier (105) is trained using a training set of images (102) and is operable to perform classification on a query set of images (103), the query set of images (103) being generated from images captured by the camera (100), processed by an image filter (106), and stored on the storage device (102).

Referring now to fig. 7, a flow chart illustrating a fully automated transdermal optical imaging and physiological data detection system is shown. The system performs image registration (registration)701 to record input about a video sequence captured by a subject with an unknown physiological state, hemoglobin image extraction 702, ROI selection 703, multi-ROI spatiotemporal hemoglobin data extraction 704, physiological state model 705 application, data mapping 706 (to map changing hemoglobin patterns), physiological state detection 707, and report generation 708. Fig. 11 depicts another such illustration of an automated physiological data detection system.

An image processing unit acquires each captured image or video stream and performs operations on the images to generate a corresponding optimized HC image of the subject. The image processing unit separates out the HC in the acquired video sequence. In an exemplary embodiment, an image of the face of the subject is taken at 30 frames per second using a digital camera. It will be appreciated that the process may be performed with alternative digital cameras and lighting conditions.

Separation of HC is achieved by: bit planes in a video sequence are analyzed to determine and isolate a set of bit planes that provide a high signal-to-noise ratio (SNR) and thus optimize signal discrimination between different physiological states beneath the facial epidermis (or any portion of the human epidermis). The high SNR bitplane is determined with reference to a first training set of images comprising the captured video sequence, the first training set of images being coupled with EKG, pneumatic respiration, blood pressure, laser doppler, oximetry data from a human subject from which the training set was obtained. EKG, pneumatic respiration data, blood pressure, blood oxygen data are first used to extract heart rate, respiration rate, blood pressure, and blood oxygen data from HC data. The second step includes training the machine to build a computational model for a particular physiological state using spatiotemporal signal patterns of transdermal HC changes in a region of interest ("ROI") extracted from optimized "bit-plane" images of a large number of samples of human subjects.

Heart rate, respiratory rate, blood pressure, blood oxygen index are acquired by analyzing the bit planes in the video sequence to determine and isolate the set of bit planes that best correlate with EKG, pneumatic respiration, blood pressure, and blood oxygen machine data.

The human brain innervates the heart by stimulating through the Autonomic Nervous System (ANS), which includes both the sympathetic and parasympathetic nervous systems. Activation of the sympathetic nervous system results in an increase in heart rate, while the parasympathetic nervous system, such as a decrease in heart rate. As a result of the saw battle between the two systems, the heart is constantly regulating between acceleration and deceleration. The change in time interval between Heartbeats (HRV) reflects the state of the autonomic nervous system.

Over a quarter century of clinical studies have shown that HRV can be a reliable indicator of the stress level in subjects. When a person is exposed to a pressure source, the parasympathetic nervous system is inhibited and the sympathetic nervous system is activated. Hormones (e.g., epinephrine and norepinephrine) are secreted into the bloodstream, which results in a series of physiological responses, such as vasoconstriction, increased blood pressure, and reduced heart rate variability. When the stressor is no longer present, the body ceases to produce cortisol, the balance between the sympathetic and parasympathetic nervous systems is reestablished, and the heart rate variability increases again.

After the empirical HC isolation procedure, the set of bit planes that provides the highest heartbeat signal-to-noise ratio is determined and the optimized heartbeat signal is extracted. By defining the distance between two consecutive heartbeat peaks, heartbeat interval time series data is calculated. Several digital signal conversions (e.g., fourier transforms) are performed and a pressure level index is obtained. By comparing the stress level index to a previously generated canonical stress index profile, a comparative stress level of the subject can be evaluated. A common heartbeat signal can be extracted from the HC in any ROI, and the system can utilize multiple ROIs to enhance and improve the extracted heartbeat signal because it is redundant information carried in all/any ROI. Once the pressure level (and optionally the heartbeat signal) is determined, it can be used as an input to a classifier to predict the overall physiological state of the subject. The stress index provides a valuable and distinct indication (independent of the heart beat signal or HC changes from what actually derived the stress index) of the subject's physiological state prediction/classification.

To perform training, video images of test subjects exposed to stimuli known to cause a particular physiological state are captured. The responses may be broadly grouped (medium, low, high), or more specifically grouped (highly stressed, slightly stressed, highly painful, slightly painful, etc.). In further embodiments, the level in each physiological state may be captured. Preferably, subjects are instructed not to express their physiological state on the face, so that the measured physiological response is an invisible physiological state and is expressed only as a change in HC. To ensure that subjects do not "leak" the physiological state in the facial expression, the sequence of surface images may be analyzed using a surface physiological expression detection program. EKG, pneumatic respiration, blood pressure, laser doppler, and blood oxygenation data may also be collected using EKG machines, pneumatic ventilators, continuous blood pressure machines, laser doppler machines, and oximeters, as described below, and provide additional information to reduce noise from the location analysis.

The ROI for physiological state detection (e.g., forehead, nose, and cheek) is defined for the video image either manually or automatically. These ROIs are preferred by the subject expert who is working on domain knowledge about how HC is relevant as an indication of physiological state. The local image of all bit planes comprising all three R, G, B channels is used to extract the signal that changes over a certain period of time (e.g., 10 seconds) on each ROI under a certain physiological state (e.g., compression). This process may be repeated for other physiological states (e.g., loose or neutral). The EKG and pneumatic respiration data can be used to prevent non-physiologic state system HC signals from masking real physiologically relevant HC signals. A Fast Fourier Transform (FFT) may be used on the EKG, respiration, and blood pressure data to obtain peak frequencies of EKG, respiration, and blood pressure and blood oxygen, and then a notch filter may be used to measure HC activity on the ROI with temporal frequencies centered around these frequencies. Independent Component Analysis (ICA) can be used to achieve the same goal.

Referring now to fig. 9, a diagram of data-driven machine learning for optimized hemoglobin image combination is shown. Using the filtered signals from the ROIs of the two or more example states 901 and 902, machine learning 903 is employed to systematically identify the bit planes 904 that will significantly increase the signal discrimination between different physiological states and bit planes that do not affect or decrease the signal discrimination between different physiological states. After discarding the latter, the remaining bit-plane image 905 that optimally distinguishes the physiological state of interest is obtained. More specifically, the bit plane selection includes: an RGB pixel bit pattern is selected that will maximize the signal-to-noise ratio (SNR) of the signal discrimination between different physiological states. To further improve the SNR, the results may be repeatedly fed back to the machine learning 903 process until the SNR reaches an optimal asymptotic value.

As the set of bit planes that are determined to maximize the SNR of the signal difference between different physiological states (e.g., maximize the SNR of the heartbeat signal) includes calibration, the determination may be performed once during the extraction process or may be performed periodically in order to continuously ensure the maximum SNR throughout the extraction process. This frequency provides a compromise between extraction time and the desired signal quality.

The machine learning process involves manipulating the bit-plane vectors (e.g., 8 × 8 × 8, 16 × 16 × 16) using image subtraction and addition to maximize the signal differences in all ROIs between different physiological states over a period of time for a portion (e.g., 70%, 80%, 90%) of the subject data, and validating the remaining subject data. The addition or subtraction is performed in a pixel manner. Existing machine learning algorithms (long short term storage (LSTM) neural networks, or suitable alternative algorithms) are used to efficiently obtain information about the discrimination between different physiological states with respect to an increase in accuracy, bit plane(s) contributing the best information, and bit planes that have no effect on feature selection. Long short term storage (LSTM) neural networks allow us to perform group feature selection and classification. The LSTM machine learning algorithm is discussed in more detail below. By this processing, a set of bit planes to be separated from the image sequence to reflect the temporal change in the HC is acquired. The image filter is configured to isolate the identified bit-planes in subsequent steps described below.

The image classifier 105 (which has been previously trained using a training set of images captured using the above method) classifies the captured images as corresponding to physiological states. In a second step, machine learning is again employed to build a computational model of the physiological state of interest (e.g., high risk versus low risk of heart attack) using a new training set of subject physiological data derived from the optimized bit-plane images provided above. Referring now to fig. 10, a diagram of data-driven machine learning for multi-dimensional invisible physiological state model building is shown. To create such a model, a second set of training subjects (preferably a new multi-family training subject set with different skin types) is recruited and an image sequence is acquired 1001 as they are exposed to stimuli that elicit known physiological responses. Exemplary stimulus sets are the International emotional Picture System (International emotion Picture System) and other well-established physiological state induction paradigms that have been commonly used to induce (intue) physiological states. An image filter is applied to the image sequence 1001 to generate a high HC SNR image sequence. The stimulus may also include non-visual aspects, such as auditory, taste, olfactory, tactile, or other sensory stimuli, or combinations thereof.

Using this new training set of subject physiological data 1003 derived from the bit-plane filtered images 1002, machine learning is again used to build a computational model 1003 of the physiological state of interest (e.g., high risk versus low risk of a heart attack). Note that the physiological state of interest used to identify the remaining level filtered images that optimally distinguish the physiological state of interest must be the same as the state used to build the computational model of the physiological state of interest. For different physiological states of interest, the former must be repeated before the latter begins.

The machine learning process also involves a portion of the subject data (e.g., 70%, 80%, 90% of the subject data) and uses the remaining subject data to validate the model. This second machine learning process thus produces a separate multidimensional (spatial and temporal) computational model 1004 of the trained physiological state.

To build different physiological models, facial HC variation data on each pixel of each subject's facial image is extracted (from step 1) as a function of time while the subject is observing a particular physiological state evoked stimulus. To improve SNR, the face of the subject is divided into multiple ROIs according to the different underlying ANS adjustment mechanisms of the aforementioned multiple ROIs, and the data in each ROI is averaged.

Referring now to fig. 4, a plot showing differences in hemoglobin distribution across a subject's forehead is shown. Although neither human nor computer-based facial expression detection systems can detect any facial expression differences, transdermal images show significant differences in hemoglobin distribution between the positive 401, negative 402, and neutral 403 conditions. The difference in hemoglobin distribution of the nose and cheek of the subject can be seen in fig. 5 and 6, respectively.

Long short term storage (LSTM) neural networks, or suitable alternatives such as non-linear support vector machines, and deep learning may also be used to assess the existence of a universal spatiotemporal pattern of hemoglobin variation across a subject. A long-short term storage (LSTM) neural network machine or surrogate is trained on transdermal data from a portion (e.g., 70%, 80%, 90%) of subject 1 to obtain a multi-dimensional computational model for each of the three invisible physiological classifications. These models were then tested on data from the remaining training subjects.

Following these steps, it is now possible to acquire a video sequence of an arbitrary subject and apply the HC extracted from the selected bit plane to a computational model of the physiological state of interest. The output will be: (1) an estimated statistical probability that the physiological state of the subject belongs to one of the trained physiological states, and (2) a normalized intensity measure of such physiological state. For long running video streams, the probability estimates and intensity scores over time, which depend on HC data based on a moving time window (e.g., 10 seconds), can be reported as the physiological state changes and the intensity fluctuates. It will be appreciated that the confidence level of the classification may be less than 100%.

In further embodiments, an optical sensor in the form of a wristwatch, wristband, hand strap, clothing, footwear, glasses, or steering wheel directed at or directly attached to the skin of any body part (e.g., wrist or forehead) may be used. The system can also extract dynamic hemoglobin changes associated with physiological states from these body regions while removing heartbeat artifacts and other artifacts such as motion and thermal interference.

In further embodiments, the system may be installed in robots and variants thereof (e.g., humanoid robots) that interact with humans to enable the robots to detect hemoglobin changes of the face or other body parts of the human being interacting with the robot. Thus, a robot equipped with transdermal optical imaging capability reads the non-visible physiological state of a human and other hemoglobin change related activities to enhance human-computer interaction.

Two example implementations for the following operations will now be described in more detail: (1) obtaining improved information in terms of accuracy regarding the differentiation between physiological states, (2) identifying the bit-planes that contribute the best information and the bit-planes that have no impact in feature selection, and (3) assessing the presence of a universal spatiotemporal pattern of hemoglobin variations across the subject. One example of such an implementation is a recurrent neural network.

One type of recurrent neural network is known as a long short term storage (LSTM) neural network, which is a type of neural network model designated for sequence data analysis and prediction. The LSTM neural network includes at least three layers of cells. The first layer is an input layer, which accepts input data. The second layer (and possible additional layers) is a hidden layer, comprising memory cells (see fig. 12). The last layer is an output layer that uses logistic regression to generate output values based on the hidden layer.

As shown, each memory cell includes four main elements: an input gate, a neuron with a self-recursive connection (a connection to itself), a forgetting gate, and an output gate. The self-recursive connections have a weight of 1.0 and ensure that (apart from any external disturbances) the state of the memory cell can remain unchanged from one time step to another. These gates are used to modulate the interaction between the memory cell itself and its environment. The input gate permits or prevents an incoming signal from altering the state of the memory cell. On the other hand, the output gate may permit or prevent the state of the memory cell from affecting other neurons. Finally, the forgetting gate can modulate the self-recursive connection of the memory cell, permitting the cell to remember or forget its previous state as desired.

The following equation describes how the memory cell layer is updated at each time step t. In these equations: x is the number of_tThe input array to the memory cell layer at time t. In the present application, this is the blood flow signal at all ROIs:

W_i、W_f、W_c、W_o、U_i、U_f、U_c、U_oand V_oIs a weight matrix; and b is_i、b_f、b_cAnd b_oIs a deviation vector.

First, we calculate the input gate i at time t_tAnd candidate value of state of memory cellThe value of (c):

i_t＝σ(W_ix_t+U_ih_t-1+b_i)

then, we calculate the activation f of the forgetting gate of the memory cell at the time t_tThe value of (c):

f_t＝σ(W_fx_t+U_fh_t-1+b_f)

given input gate activation i_tForget gate activation f_tAnd candidate state valuesCan calculate the new state C of the memory cell at time t_t：

With the new states of the memory cells, we can compute the values of their output gates and then compute their outputs:

o_t＝σ(W_ox_t+U_oh_t-1+V_oC_t+b_o)

h_t＝o_t*tanh(C_t)

based on the model of the memory unit, we can compute the output from the memory unit for the blood flow distribution at each time step. Thus, according to the input sequence x₀、x₁、x₂、……、x_nThe memory cells in the LSTM layer will generate a token sequence h₀、h₁、h₂、……、h_n。

The goal is to classify sequences into different conditions. The logistic regression output layer generates a probability for each condition based on the characterization sequence from the LSTM hidden layer. The probability vector at time step t can be calculated as follows:

p_t＝softmax(W_outputh_t+b_output)

wherein, W_outputIs a weight matrix from the hidden layer to the output layer, and b_outputIs the deviation vector of the output layer. The condition with the greatest cumulative probability will be the predicted condition for the sequence.

Other machine training methods, such as deep learning, may also be used.

Referring now to fig. 8, an exemplary report showing the output of a system for detecting a physiological state of a human is shown. The system may attribute a unique customer number 801 to the name 802 and gender 803 of a given principal. The physiological state 804 is identified with a given probability 805. Physiological state intensity levels 806 are identified, as well as physiological state intensity index scores 807. In an embodiment, the report may include a graph that over time 811 is shown as the physiological state experienced by the subject 808 based on a given ROI 809 compared to the model data 810.

Although the above embodiments refer to detecting pressure, one skilled in the art will appreciate that other physiological states may be detected using the same method. For example, the method can be used to detect the presence or absence of pain in a subject. Since pain states and no-pain states primarily activate the sympathetic and parasympathetic nervous systems, respectively, they can be distinguished by analyzing spatial and temporal HC changes in the subject's face. An optimal plane is determined for pain/no pain discrimination, and a pain/no pain computational model is built using machine learning methods and used to estimate the statistical probability that a subject experiences/does not experience pain.

The above-described systems and methods may be applied in a variety of fields, including personal physiological data capture. In one embodiment, a person may capture one or more sets of images of themselves using a conventional digital camera (e.g., a webcam, a camera built into a smartphone, etc.). The image set may then be analyzed using a computing device having a physiological data model constructed from training. This may be done locally or remotely by sending the captured image set to another computing device (e.g., during a video-based remote healthcare session).

The method can also be used to detect skin lesions that are often difficult to detect visually. Various skin lesions ranging from acne and pimples to basal cell and squamous cell carcinomas can lead to abnormal local hemoglobin/melanin concentrations and can be detected at a very early stage from images of the transdermal structures.

In addition, some diseases can be detected early by the above-mentioned methods. This can be used to screen for infectious symptoms at the margin and other checkpoints.

In an embodiment, the system may be used to determine the stress or pain state of a subject who is unable to speak and/or suffers from a muscle disability.

In other embodiments, the system may be used to quantify the stress level of a subject during a stress event to determine how well a particular subject is adapted for a particular location, role, etc.

The system may be used to identify stress, pain, and fatigue experienced by employees in transportation or military environments. For example, a tired driver, pilot, captain, soldier, etc. may be identified as too tired to effectively continue shift work. In addition to security improvements that may be enacted by the transportation industry, an analysis of the notification schedule may also be derived.

In yet another aspect, the system may be used by financial institutions seeking to reduce risks associated with trade practices or loans. The system may provide insight into the level of stress experienced by the trader, providing a counter balance for risk trading.

The system may be used by telemarketers who attempt to assess user responses to particular words, phrases, marketing strategies, etc. that may inform the best marketing method to stimulate brand loyalty or complete a sale.

In further embodiments, the system may be used as a tool in emotional neuroscience. For example, the system may be coupled with an MRI or NIRS or EEG system to measure not only neural activity associated with pressure and/or pain in a subject, but also changes in transdermal blood flow. The collected blood flow data may be used to provide additional and validated information about the subject's stress and/or pain state, or to isolate physiological signals generated by the cortical central nervous system from those generated by the autonomic nervous system. For example, the blush and brain problems in functional near infrared spectroscopy (fNIRS) studies where skin hemoglobin changes are often mixed with scalp hemoglobin changes can be addressed.

In further embodiments, the system may detect physiological conditions caused by sounds other than vision (e.g., music, crying, etc.). Physiological conditions caused by other senses, including smell, taste, and vestibular sensations may also be detected.

Other applications may become apparent.

While the present invention has been described with reference to certain specific embodiments, various modifications thereof will be apparent to those skilled in the art without departing from the spirit and scope of the invention as outlined in the claims appended hereto. The entire disclosures of all of the above references are incorporated herein by reference.

Claims (18)

1. A system for detecting a physiological state from a sequence of captured images of a subject, the system comprising:

a camera configured to capture a sequence of images of the subject, the sequence of images comprising a query set of images;

a processing unit, in communication with the data storage device, trained to determine a set of bitplanes of a plurality of images of the captured sequence of images representing variations in hemoglobin concentration, HC, of the subject and maximizing signal differences between different physiological states, wherein the set of bitplanes representing variations in HC is determined for a selected plurality of regions of interest, ROIs, of the subject that are relevant as indications of physiological states;

a classifier trained using a training set comprising HC variations for subjects having known physiological states and configured to:

detecting a physiological state of the subject based on HC changes in the set of levels; and

outputting the detected physiological state of the subject,

wherein the processing unit is further configured to: extracting a heartbeat signal of the subject from HC variations of one or more ROIs, processing the heartbeat signal to determine heartbeat interval time series data of the subject, performing a fourier transform on the heartbeat interval time series data to obtain a stress level index, comparing the stress level index to a predetermined standard stress index profile to determine a comparative stress level of the subject, and providing the comparative stress level of the subject to the classifier for detecting a physiological state of the subject.

2. The system of claim 1, wherein determining the set of bitplanes that maximizes the difference between different physiological states comprises: machine learning techniques are used to select the RGB pixel bit combination that maximizes the heartbeat signal-to-noise ratio.

3. The system of claim 1, wherein detecting the physiological state of the subject based on HC changes comprises: calculating an estimated statistical probability that the physiological state of the subject corresponds to a known physiological state from the training set, and calculating a normalized intensity measure of the physiological state so determined.

4. The system of claim 3, wherein outputting the physiological state of the subject comprises: each detected physiological state is grouped into one of a plurality of groups based on the calculated estimated statistical probability and the normalized intensity metric.

5. The system of claim 1, wherein the physiological state comprises a stress level, a pain level, or a fatigue level.

6. The system of claim 1, wherein the camera comprises an optical sensor directly attached to the subject's skin, and the processing unit is further configured to remove artifacts due to motion and heat from the captured image sequence.

7. The system of claim 1, wherein the processing unit is further configured to process the captured sequence of images to detect skin lesions that are difficult to visually detect.

8. The system of claim 1, further comprising one of a magnetic resonance imaging unit, a near infrared spectroscopy imaging unit, or an electroencephalography imaging unit for capturing a second sequence of images, and the processing unit is further configured to determine a change in transdermal blood flow of the subject from the second sequence of images.

9. A method for detecting a physiological state from a sequence of captured images of a subject, the method comprising:

a camera captures a sequence of images of the subject, the sequence of images comprising a query set of images;

a trained processing unit processes the captured sequence of images to determine a set of bit planes for a plurality of images of the captured sequence of images representing variations in hemoglobin concentration, HC, of the subject and maximizing signal differences between different physiological states, wherein the set of bit planes representing variations in HC are determined for a selected plurality of regions of interest, ROIs, of the subject in relation to which an indication of a physiological state relates;

extracting, by the trained processing unit, heartbeat signals of the subject from HC variations of one or more ROIs;

processing, by the trained processing unit, the heartbeat signals to determine heartbeat interval time series data of the subject;

performing, by the trained processing unit, a Fourier transform on the heartbeat interval time series data to obtain a stress level index;

comparing, by the trained processing unit, the stress level index to a predetermined standard stress index profile to further determine a comparative stress level of the subject;

processing the set of bit-planes using a classifier trained using a training set including HC variations for subjects having known physiological states to:

detecting a physiological state of the subject based on HC changes in the level set and a comparative pressure level of the subject; and

outputting the detected physiological state.

10. The method of claim 9, wherein determining the set of bitplanes that maximize the difference between different physiological states comprises using a machine learning technique to select an RGB pixel bitpattern that maximizes a heartbeat signal-to-noise ratio.

11. The method of claim 9, wherein detecting the physiological state of the subject based on HC changes comprises: calculating an estimated statistical probability that the physiological state of the subject corresponds to a known physiological state from the training set, and calculating a normalized intensity measure of the physiological state so determined.

12. The method of claim 11, wherein outputting the physiological state of the subject comprises grouping each detected physiological state into one of a plurality of groupings based on the calculated estimated statistical probability and a normalized intensity metric.

13. The method of claim 9, wherein the physiological state comprises a stress level, a pain level, or a fatigue level.

14. The method of claim 9, wherein the camera comprises an optical sensor directly attached to the subject's skin, and further comprising removing artifacts due to motion and heat from the captured image sequence.

15. The method of claim 9, further comprising processing the captured sequence of images to detect skin lesions that are difficult to visually detect.

16. The method of claim 9, further comprising one of a magnetic resonance imaging unit, a near infrared spectroscopy imaging unit, or an electroencephalography imaging unit capturing a second sequence of images, and processing the second sequence of images to determine a change in transdermal blood flow of the subject from the second sequence of images.

17. A system for detecting a physiological state from a sequence of captured images of a subject, the system comprising:

a camera configured to capture a sequence of images of the subject, the sequence of images comprising a query set of images;

a processing unit, in communication with the data storage device, trained to determine a set of bit planes for a plurality of images of the captured sequence of images representing a change in hemoglobin concentration HC of the subject and maximizing signal differences between different physiological states, the determination comprising:

extracting a heartbeat signal of the subject from the HC variation, an

Selecting an RGB pixel bit combination that maximizes the heartbeat signal-to-noise ratio using machine learning techniques; a classifier trained using a training set comprising HC variations for subjects having known physiological states and configured to:

detecting the physiological state of the subject based on the HC changes in the set of bit planes, the detecting comprising calculating an estimated statistical probability that the physiological state of the subject conforms to known physiological states from the training set and calculating a normalized intensity measure of the physiological state so determined; and

outputting the detected physiological state.

18. A method for detecting a physiological state from a sequence of captured images of a subject, the method comprising:

capturing, by a camera, a sequence of images of the subject, the sequence of images comprising a query set of images;

determining, by a trained processing unit, a set of bitplanes of a plurality of images of the captured sequence of images representing a change in hemoglobin concentration, HC, of the subject and maximizing a difference in signal between different physiological states, the determining comprising:

extracting a heartbeat signal of the subject from the HC variation, an

Selecting an RGB pixel bit combination that maximizes the heartbeat signal-to-noise ratio using machine learning techniques;

processing the set of bit-planes by a classifier trained using a training set including HC variations for subjects having known physiological states to:

detecting the physiological state of the subject based on the HC changes in the set of bit planes, the detecting comprising calculating an estimated statistical probability that the physiological state of the subject matches a known physiological state from the training set and calculating a normalized intensity measure of the physiological state so determined; and

outputting the detected physiological state.