CN116602636A - Eye health monitoring system based on intelligent watch - Google Patents

Abstract

The invention discloses an eye health monitoring system based on an intelligent watch, which comprises the intelligent watch, an external positioning module, a nose bridge positioning verification module and an auxiliary gesture confirmation module, wherein the intelligent watch comprises a heart rate sensor module, a gesture motion tracking module, a risk assessment module, an eye temperature measurement module, a camera module, a health management algorithm module, a reminding output module and a display module, and when the algorithm detects that eye data is abnormal or health risk is increased, health reminding is carried out and corresponding eye health care suggestions are given. The invention realizes real-time eye health monitoring, reduces false alarm and automatic risk assessment, provides personalized health advice for users, and provides a convenient and accurate eye health monitoring scheme for special crowds such as children.

Description

A smart watch-based eye health monitoring system

technical field

The invention belongs to the technical field of health monitoring, and in particular relates to a smart watch system for monitoring eye health.

Background technique

With the development of technology and the popularization of smart devices, smart watches have become a common wearable device in people's daily life. Smart watches have the advantages of easy portability, real-time data collection, and diverse functions, providing users with a wealth of health management and life assistance functions. However, the smart watches currently on the market focus on the monitoring of physiological indicators such as heart rate and exercise, and have not yet fully utilized their potential to solve eye health problems, especially for special populations such as children.

Eye disease has a major impact on children's quality of life and learning ability. In recent years, the incidence of high myopia in children has been increasing year by year, and eye complications caused by type 1 diabetes in children are also becoming more and more serious. These eye problems require prompt diagnosis and monitoring for effective treatment. In addition, eye health monitoring during the recovery period after eye surgery is also crucial to the patient's recovery process.

The purpose of the present invention is to detect the eye conditions of people with eye diseases, especially children and the like. By combining the portability, real-time and functional diversity of smart watches, the present invention is dedicated to solving the problems of children's high myopia, eye health detection caused by children's type 1 diabetes, and eye postoperative recovery. The invention adopts various sensors and modules to realize real-time monitoring of the user's eye health status, automatic risk assessment and personalized health advice, and is expected to provide a more convenient, accurate and intelligent eye health monitoring solution for people with eye diseases.

Contents of the invention

Aiming at the above-mentioned defects in the prior art, the present invention provides an eye health monitoring system based on a smart watch. The system includes a smart watch, an external positioning module, a nose bridge positioning verification module, and an auxiliary gesture confirmation module. The smart watch includes a heart rate sensor module, a gesture movement tracking module, a risk assessment module, an eye temperature measurement module, a camera module, a health management algorithm module, a reminder output module and a display module, and is characterized in that:

The heart rate sensor module is used to regularly monitor the wearer's heart rate to confirm that the wearer is in a non-sleeping state;

The gesture motion tracking module is used to start the wearer's gesture tracking and detection function after confirming that the wearer is in a non-sleep state, and the wearer's gestures include eye rubbing gestures;

Among them, the gesture motion tracking module performs gesture tracking and detection of the wearer, including jointly performing gesture tracking with an external positioning module, a positioning verification module and an auxiliary gesture confirmation module;

The risk assessment module is used to determine whether the eye-rubbing gesture triggers the risk assessment mechanism within a preset time period based on the intensity and frequency of the eye-rubbing gesture, and prompts the wearer to follow the instructions on the display module through the display module after the risk assessment mechanism is triggered. Prompts for eye temperature measurement and eye image capture;

The eye temperature measurement module is used to measure the eye temperature through the temperature sensor on the edge of the watch close to the eyelid;

The camera module is used to take eye images, automatically analyze the eye images based on computer vision algorithms, including preprocessing the images, and extracting eigenvalue vectors related to abnormal eye symptoms based on the preprocessed images;

The health management algorithm module is used to analyze the risk degree of the wearer's eye blood flow and optic nerve health according to the monitoring parameters and the eigenvalue vector of the eye image;

The reminder output module is used to provide health reminders and give corresponding eye care suggestions when the algorithm detects abnormal eye data or increased health risks.

Among them, for smart watches, external positioning modules, nose bridge positioning verification modules, and auxiliary gestures to confirm the wearing position of the module, including:

The smart watch is worn on the wrist of the wearer's left or right hand;

The external positioning module is fixed on the headband, and the external device is located in the center of the wearer's forehead;

The nose bridge positioning verification module can be installed on the nose bridge part of the glasses worn by the wearer; the auxiliary gesture confirmation module is a wrist strap, which is worn on the other wrist where the smart watch is not worn.

Among them, the gesture motion tracking module performs gesture tracking and detection of the wearer, including joint gesture tracking with an external positioning module and a positioning verification module, including:

The wearer wears an external positioning module including a Bluetooth module on the head, and a positioning verification module is set on the nose bridge of the glasses, and the positioning verification module includes a gyroscope and an accelerometer;

After the wearer wears the glasses, turn on the Bluetooth tracking function, and the external positioning module records the signal strength RSSI and the angle of arrival AOA between the positioning verification module as a reference, and then turns off the Bluetooth tracking function of the external positioning module and the positioning verification module.

Wherein, when the gyroscope and the accelerometer of the positioning verification module detect a position change, an activation beacon is sent to the external positioning module, and the activation beacon is used to notify the external positioning module to start the Bluetooth tracking function;

In the external positioning module, the positioning verification module is continuously tracked by bluetooth. Check the signal strength RSSI and the angle of arrival AOA between the modules, and when it is judged that the change of the signal strength RSSI or the angle of arrival AOA exceeds the threshold, the external positioning module sends a Bluetooth tracking establishment request to the smart watch and the auxiliary gesture confirmation module, and the smart watch and After receiving the establishment request, the auxiliary gesture confirmation module turns on Bluetooth to establish a Bluetooth connection with the external positioning device.

Among them, the external positioning module performs Bluetooth tracking on the smart watch and the auxiliary gesture confirmation module. By comparing the RSSI and AOA between the smart watch and the auxiliary gesture confirmation module with the recorded signal strength and AOA as a reference, when the tracked smart When at least one positioning data in the watch and auxiliary gesture confirmation module meets the condition, start the timer;

And, when the timer reaches the set timer threshold, it is judged as an eye rubbing action.

Wherein, the angle of arrival AOA is the angle between the signal propagation path and the reference direction;

When the relative position between the external positioning module and the positioning verification module remains constant, the angle of arrival AOA also remains constant;

determining a reference direction according to the direction of the external positioning device itself, and the reference direction will change with the movement and rotation of the device;

And, select the horizontal direction of the external positioning device after the head is fixed as the reference direction for AOA measurement.

Wherein, at least one positioning data in the tracked smart watch and the auxiliary gesture confirmation module meets the conditions, including:

The absolute value ΔRSSI of the difference between the signal strength RSSI of the detected smart watch or auxiliary gesture confirmation module and the recorded signal strength RSSI between the reference and the positioning verification module is less than the threshold value, and the detected smart watch or auxiliary gesture Confirm that the AOA of the module is within the preset angle range.

Wherein, when the external positioning module detects the positioning data that meets the conditions, it starts counting, and the starting time is the starting time;

Continuously monitor the hand movement data during the timing process, the timer will stop under the second condition, and the stop time is the end time; the second condition is: when it is detected that the hand movement data no longer meets the characteristics of the eye rubbing gesture, it means The eye-rubbing action has stopped. At this time, at least one of ΔRSSI and AOA does not meet the requirements of the eye-rubbing gesture. Specifically, the ΔRSSI of the smart watch is greater than the threshold or the AOA is not within the preset angle range, and the ΔRSSI of the auxiliary gesture confirmation module is greater than the threshold or AOA is not within the preset angle range;

After the timer stops, the strength of the gesture is determined based on the accumulated eye rubbing duration.

Among them, when the risk assessment mechanism is triggered, the display module can remind the wearer to take eye temperature measurement and eye image shooting by popping up prompt information or sending out sound/vibration;

Prompt information includes text, image, voice or animation, etc., instructing the wearer how to correctly measure eye temperature and take eye images;

After the wearer completes eye temperature measurement and eye image shooting, the display module displays the measurement results.

Among them, in the camera module, the eye image is automatically analyzed based on the computer vision algorithm, including image preprocessing;

The preprocessing includes performing, color space conversion, histogram equalization, noise reduction processing, identification of eye regions, image cropping and scaling and rotation correction;

In the camera module, based on the preprocessed image, the eigenvalue vector of the abnormal eye symptom is extracted, and the eigenvalue vector is a combination vector, including combining the output of the color classifier and the LBP texture feature after PCA dimensionality reduction into a feature Vector, combined in the form of splicing two features;

Use the deep learning model to learn the monitoring parameters and the eigenvalue vector of the eye image, and analyze the relationship and fusion analysis between different features to obtain the risk level of the user's eye blood flow status and optic nerve health.

The smart watch-based eye health monitoring system provided by the present invention solves some problems in the prior art and realizes real-time monitoring of the user's eye health status, reduction of false alarms, automatic risk assessment and personalized health advice, etc. technical effect. This will provide a more convenient, accurate and intelligent eye health monitoring solution for people with eye diseases, especially children.

Description of drawings

The above and other objects, features and advantages of exemplary embodiments of the present disclosure will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the present disclosure are shown by way of illustration and not limitation, and the same or corresponding reference numerals indicate the same or corresponding parts, wherein:

Fig. 1 is a schematic diagram showing a smart watch in a smart watch-based eye health monitoring system according to an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise, "multiple" Generally contain at least two.

It should be understood that although the terms first, second, third, etc. may be used to describe ... in the embodiments of the present invention, these ... should not be limited to these terms. These terms are only used to distinguish from. For example, without departing from the scope of the embodiments of the present invention, first... may also be referred to as second..., and similarly, second... may also be referred to as first....

It should be understood that the term "and/or" used herein is only an association relationship describing associated objects, which means that there may be three relationships, for example, A and/or B, which may mean that A exists alone, and A and B exist simultaneously. B, there are three situations of B alone. In addition, the character "/" in this article generally indicates that the contextual objects are an "or" relationship.

Depending on the context, the words "if", "if" as used herein may be interpreted as "at" or "when" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrases "if determined" or "if detected (the stated condition or event)" could be interpreted as "when determined" or "in response to the determination" or "when detected (the stated condition or event) )" or "in response to detection of (a stated condition or event)".

It should also be noted that the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that an article or arrangement comprising a list of elements includes not only those elements but also includes items not expressly listed. other elements of the product, or elements inherent in the product or device. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in an article or device comprising said element.

Existing smart watches mainly focus on the monitoring of physiological indicators such as heart rate and exercise, but lack attention and solutions to eye health problems. Especially for special populations such as children, eye diseases have an important impact on their quality of life and learning ability, such as high myopia in children, eye complications caused by type 1 diabetes in children, and eye surgery recovery.

Aiming at the deficiencies of the above background technologies, the present invention proposes an eye health monitoring system based on smart watches, aiming to solve the following technical problems and achieve corresponding technical effects: by integrating various sensors and modules, the user’s eye health can be monitored. Real-time monitoring of the condition, so that users can keep abreast of their own eye conditions and prevent the occurrence and development of eye diseases. By combining the external positioning module, nose bridge positioning verification module and auxiliary gesture confirmation module, false alarms are effectively reduced, the accuracy of monitoring data is improved, and users can use the eye monitoring system with more confidence. The health management algorithm module is used to automatically analyze the eigenvalue vector of the eye image, and evaluate the risk of the user's eye blood flow and optic nerve health based on the monitoring parameters, reducing the dependence on professionals and improving the efficiency of risk assessment. According to abnormal eye data or increased health risks detected by the algorithm, it provides users with personalized health reminders and eye care suggestions to help users better pay attention to and improve eye health, especially for special groups such as children.

The invention discloses an eye health monitoring system based on a smart watch. The system includes a smart watch, an external positioning module, a nose bridge positioning verification module, and an auxiliary gesture confirmation module. The smart watch includes a heart rate sensor module, gesture motion The tracking module, the risk assessment module, the eye temperature measurement module, the camera module, the health management algorithm module, the reminder output module and the display module are characterized in that:

The heart rate sensor module is used to regularly monitor the user's heart rate to confirm that the user is in a non-sleeping state;

The gesture motion tracking module is used to start after confirming that the user is in a non-sleep state, and start the user gesture tracking detection function, and the user gesture includes an eye-rubbing gesture;

Among them, the gesture motion tracking module performs tracking and detection of user gestures, including jointly performing gesture tracking with an external positioning module, a positioning verification module and an auxiliary gesture confirmation module.

The risk assessment module is used to determine whether the eye-rubbing gesture triggers the risk assessment mechanism within a preset time period based on the intensity and frequency of the eye-rubbing gesture, and prompts the wearer to follow the instructions on the display module through the display module after the risk assessment mechanism is triggered. Prompts for eye temperature measurement and eye image capture;

The eye temperature measurement module is used to measure the eye temperature through the temperature sensor on the edge of the watch close to the eyelid;

The camera module is used to take eye images, automatically analyze the eye images based on computer vision algorithms, including preprocessing the images, and extracting eigenvalue vectors related to abnormal eye symptoms based on the preprocessed images, the The eigenvalue vector is a combined vector;

The health management algorithm module is used to analyze the risk degree of the user's eye blood flow and optic nerve health according to the monitoring parameters and the eigenvalue vector of the eye image;

The reminder output module is used to provide health reminders and give corresponding eye care suggestions when the algorithm detects abnormal eye data or increased health risks.

Among them, for the wearing position of the smart watch, the external positioning module, the nose bridge positioning verification module, and the auxiliary gesture confirmation module, the smart watch is usually worn on the user's wrist, which can be left or right. The external positioning module is fixed on the headband, and the external device is positioned at the center of the user's forehead. The nose bridge positioning verification module can be installed on the nose bridge part of the glasses worn by the user. The auxiliary gesture confirmation module can be a wrist strap, which is worn on the other wrist where the smart watch is not worn.

The smart watch system of the present invention assesses the risk of eye blood flow and optic nerve health by tracking user gestures, measuring eye temperature, and taking eye images. etc. to remind.

The smart watch system of the present invention can monitor the user's gestures, eye temperature and eye image in real time to assess the risk of eye blood flow and optic nerve health, and remind the user to pay attention to eye health in time. Compared with traditional eye health detection equipment, the present invention has higher practicability and convenience.

In a certain embodiment, the heart rate sensor mainly includes two types of electrical signal sensor and optical sensor, wherein the optical sensor indirectly measures the heart rate by detecting changes in blood flow through optical technology. In the present invention an optical sensor is used to measure the user's heart rate. The working principle of the optical sensor is based on photoplethysmography (PPG) signal detection of optical sensing technology. The PPG signal is transmitted by the transmitter (usually a green LED) to the skin, and then the light reflected by the skin, blood vessels and other tissues is received by the photodiode receiver. When the heart beats, the blood flow in the vessels changes, and the intensity of the reflected light changes accordingly. Heart rate can be measured indirectly by detecting changes in reflected light intensity.

In the non-sleep state, the user's heart rate is usually increased, while in the sleep state, the heart rate is relatively low. Therefore, it is possible to determine whether the user is in a non-sleeping state by analyzing the heart rate data, including setting an appropriate heart rate threshold based on the user's age, gender, and other information to distinguish between a non-sleeping state and a sleeping state. For adults, the heart rate threshold is usually set within the range of 60-100 beats/minute, which can be based on the user's daily data, such as determining the average heart rate at night (22:00-8:00 am) and daytime (8:00 am-8:00 am) 22 points) the mean value of the heart rate, and the mean value of the two mean values is determined as the heart rate threshold value.

The heart rate data is continuously analyzed using a sliding window method. For example, set a sliding window of 5 minutes, move for 1 minute each time, and calculate the average heart rate within the window. When the average heart rate within the sliding window is higher than the set heart rate threshold, the user is considered to be in a non-sleep state; otherwise, the user is considered to be in a sleep state.

In a certain embodiment, the gesture motion tracking module performs tracking and detection of user gestures, including jointly performing gesture tracking with an external positioning module and a positioning verification module. Users of certain groups of people wear an external positioning module including a Bluetooth module on their heads, and set a positioning verification module on the nose bridge of the glasses, which includes a gyroscope and an accelerometer. After the user wears the glasses, turn on the Bluetooth tracking function, and the external positioning module records the signal strength RSSI and the angle of arrival AOA between the positioning verification module as a reference, and then turns off the Bluetooth tracking function of the external positioning module and the positioning verification module.

When the gyroscope and accelerometer of the positioning verification module detect that the position change is relatively obvious, such as when a sudden change in the accelerometer is detected (indicating that the user may have taken off the glasses), an activation beacon is sent to the external positioning module, and the activation signal The mark is used to notify the external positioning module to start the Bluetooth tracking function.

In the external positioning module, the Bluetooth tracking is continuously performed on the positioning verification module. At the same time, the external positioning module continuously monitors the signal strength RSSI and the angle of arrival AOA of the positioning verification module, and judges whether the signal strength RSSI or When the change of the angle of arrival AOA exceeds the threshold (indicating that an obvious movement of the wrist is detected, and the wrist movement needs to be tracked and monitored), the external positioning module sends a Bluetooth tracking establishment request to the smart watch and the auxiliary gesture confirmation module, and the smart watch and the auxiliary After receiving the establishment request, the gesture confirmation module turns on Bluetooth and establishes a Bluetooth connection with an external positioning device;

The external positioning module performs Bluetooth tracking on the smart watch and the auxiliary gesture confirmation module. By comparing the RSSI and AOA of the smart watch and the auxiliary gesture confirmation module with the recorded signal strength and AOA, when the tracked smart watch and the auxiliary gesture confirmation module When at least one of the positioning data meets the condition, the timer is started, and when the timer reaches the set timer threshold (such as 1 second), it is judged as an eye rubbing action.

Among them, the angle of arrival refers to the angle when the signal reaches the signal receiving end after the signal sending end sends out the Bluetooth signal, and the arrival angle is the angle between the signal propagation path and the reference direction. Assuming that the user's head moves, but the relative position between the external device on the top of the head and the positioning calibration module on the nose bridge on the glasses does not change, the angle of arrival (AoA) will basically not change, because the angle of arrival (AoA) is based on The relative positional relationship between the signal sending end and the receiving end is determined. Here the relative position between the external positioning module and the positioning verification module is kept constant, so the angle of arrival should also be kept constant. In the present invention, it is necessary to determine the reference direction according to the direction of the external positioning device itself. This reference direction will change with the movement and rotation of the device. For example, the direction horizontal and vertical to the external positioning device after the head is fixed can be selected as the reference for AOA measurement direction.

Wherein, at least one positioning data in the tracked smart watch and the auxiliary gesture confirmation module meets the conditions, including:

The detected signal strength RSSI of the smart watch or the auxiliary gesture confirmation module and the recorded signal strength RSSI between the positioning verification module are less than the threshold, and the detected AOA of the smart watch or the auxiliary gesture confirmation module is within the preset angle range.

In one embodiment, take the Bluetooth 4.0 low-power BLE module CC2541 as an example. CC2541 is a commonly used BLE module. The RSSI accuracy of CC2541 may be affected by various factors, including device differences, environmental interference, and signal occlusion. . In order to ensure sufficient accuracy, the RSSI error threshold can be set to 5dBm. ΔRSSI=|RSSI_measured-RSSI_standard|, wherein, RSSI_measured is the RSSI between the smart watch and the external positioning module or the RSSI between the auxiliary gesture confirmation module and the external positioning module, and RSSI_standard is the record of the external positioning module to the positioning verification module RSSI, ΔRSSI is the gap between the RSSI between the two, that is, the RSSI between the smart watch and the external positioning module or the RSSI between the auxiliary gesture confirmation module and the external positioning module and the record of the positioning verification module by the external positioning module The gap between the RSSI.

This means that when the two ΔRSSI are less than 5dBm, it is considered that the distance d1 between the smart watch and the external positioning module and/or the distance d2 between the auxiliary gesture confirmation module and the external positioning module is the same as that of the external positioning module to the positioning verification module. The initial distance d3 is close, ie d1 ≈ d3 and/or d2 ≈ d3. In the process of rubbing the eyes, the hand may block the signal, resulting in the weakening of RSSI. In order to reduce the influence of occlusion on the RSSI measurement, the external positioning module is installed on the head-mounted device, for example, it is fixed on the headband , the external device can be located in the center of the forehead, which can reduce the possibility of hand blocking the signal.

In a certain embodiment, for the AOA within a preset angle range, the preset range is ±135°.

In a certain embodiment, in order to calculate the angle between the wrist and the top of the head and the top of the head and the bridge of the nose, the following distances and dimensions need to be obtained: the distance from the top of the head to the bridge of the nose (a), the distance from the top of the head to the wrist (b), and the distance between the bridge of the nose and the bridge of the nose. The distance from the wrist (c), and then calculate the outer angle θ according to the formula of the law of cosines: cosθ=(a2+b2-c2)/(2ab). α is the angle of the AOA recorded as a reference.

According to the reference direction and the three-dimensional geometric characteristics and left-right symmetry of several devices, the preset angle range β of AOA needs to satisfy: |β|<(θ+α) and |β|<(360-(θ+α))

In the present invention, after determining the outer angle θ through the set user age, it can be determined that the AOA is in the preset angle range. The preset angle range is greater than the outer angle θ. As the height increases, the outer angle θ is more come bigger.

In a certain embodiment, for example, when the user's age is set to 7 and the nationality is Chinese, the AOA is set in the preset angle range according to the average value of 7-year-old children in China.

According to the general body size data of 7-year-old children in China, the following distances are estimated:

The distance from the top of the head to the bridge of the nose a: The height of the head of a 7-year-old child is about 18 cm (from the top of the head to the chin). Assume that the distance from the top of the head to the bridge of the nose is about half the height of the head, which is 9 cm.

Distance b from the top of the head to the wrist: The length of the arm of a 7-year-old child is about 40 cm (from the shoulder to the wrist). When the arm is bent, the distance from the top of the head to the wrist may be about 20 cm.

The distance c from the bridge of the nose to the bridge of the nose: during the eye rubbing action, the horizontal distance between the wrist and the bridge of the nose is very small, maybe about 3 cm, and the vertical distance between the wrist and the bridge of the nose also needs to be considered, assuming that this distance is about 7 cm. Therefore, use the Pythagorean theorem to calculate the distance from the bridge of the nose to the wrist sqrt(32+72)≈7.6 cm.

Based on the above distances a, b, and c, substitute into the formula of the law of cosines: cosθ=(92+202-7.62)/(2×9×20)≈0.6146, θ=arccos(0.6146)≈52.2°. Therefore, when using the data of 7-year-old children in China, the external angle between the wrist, the top of the head and the bridge of the nose is about 52.2°.

The angle α of the AOA recorded as a reference is 35 degrees.

Then, according to the geometric characteristics, for Chinese 7-year-old children, the preset angle range β of AOA needs to satisfy: |β|<131.2 and |β|<228, that is, |β|<131.2. In order to avoid errors, the AOA’s The preset angle range β is set to |β|<135.

In a certain embodiment, after starting the gesture motion tracking module, in the gesture recognition process, when the timer reaches the set timer threshold (such as 1 second), it is judged to be an eye rubbing action, if it is determined to be an eye rubbing action record The start and end time of each eye-rubbing action, and the variable of the eye-rubbing counter is increased by 1, and the sliding window is used to count the number of eye-rubbing actions that occur within a specific period of time (such as 1 hour) as the frequency of eye-rubbing actions.

When the external positioning module detects the positioning data that meets the conditions, it will start timing (this moment is the starting time). During the timing process, the system will continue to monitor the hand movement data, and the timer will stop under the following conditions (this moment is end time), including when it is detected that the hand motion data no longer conforms to the characteristics of the eye-rubbing gesture, it means that the eye-rubbing action has stopped. At this time, at least one of ΔRSSI and AOA does not meet the requirements of the eye-rubbing gesture, specifically the ΔRSSI of the smart watch greater than the threshold or the AOA is not within the preset angle range, and the ΔRSSI of the auxiliary gesture confirmation module is greater than the threshold or the AOA is not within the preset angle range.

After the timer stops, the strength of the gesture can be judged according to the accumulated eye rubbing duration.

In a certain embodiment, the duration of the eye rubbing action is divided into different levels, such as 1 degree (1-3 seconds), 2 degrees (3-6 seconds), 3 degrees (6-10 seconds), 4 degrees (10 seconds) -15 seconds), 5 degrees (more than 15 seconds).

In a certain embodiment, the strength and frequency of the analyzed gestures are sent to the risk assessment module after calculation and processing in the external positioning module. Or the external positioning module sends the timer start data conforming to the eye rubbing action to the risk assessment module, and the risk assessment module analyzes the strength and frequency of the gesture.

In one embodiment, by sliding the window, when new eye rubbing gestures appear, they are added to the window, and old gestures beyond the time period are removed. Every time the window is updated, recalculate the number and strength of the blink gestures within the window.

The condition for triggering the risk assessment mechanism can be expressed as: Risk_Assessment=(N>N_threshold)OR(I>I_threshold), wherein, T is the length of the preset time period, which can be adjusted according to the user's settings; The number of eye gestures; I is the average intensity of eye rubbing gestures in a preset time period; N_threshold is the preset threshold for the number of eye rubbing gestures; I_threshold is the preset threshold for the intensity of eye rubbing gestures. That is, when the number N of eye rubbing gestures in the window is greater than the preset threshold N_threshold, or the average intensity I is greater than the preset threshold I_threshold, the risk assessment mechanism is triggered.

In a certain embodiment, when the risk assessment mechanism is triggered, the display module can remind the wearer to take eye temperature measurement and eye image shooting by popping up prompt information or making sound/vibration. The prompt information may include text, image, voice or animation, etc., instructing the wearer how to correctly measure the eye temperature and photograph the eye image. For example, it is possible to show the correct posture of placing the temperature sensor on the edge of the smart watch close to the eyelids, and use the camera module to capture clear eye images through the line of sight indicator map that prompts the user to look directly at the position of the line of sight.

After the wearer completes eye temperature measurement and eye image shooting, the display module can display the measurement results.

In a certain embodiment, in the eye temperature measurement module, a non-contact infrared temperature sensor (Infrared Thermopile Sensor) is used to measure the eye temperature.

The non-contact infrared temperature sensor can measure the surface temperature of the target object without directly touching it. This sensor is especially suitable for eye temperature measurement, as it avoids discomfort or damage to the eye.

The working principle of infrared temperature sensors is based on the relationship between the infrared radiation emitted by the surface of an object and its temperature. As the temperature of an object increases, the infrared radiation emitted by its surface also increases. The thermopile (Thermopile) inside the temperature sensor converts the received infrared radiation into an electrical signal. Then, by amplifying and processing the electrical signal, the surface temperature of the target object can be calculated.

Hold the edge of the smartwatch close to your eye so that the non-contact infrared temperature sensor faces the eyelid. Keep a certain distance (usually 1-5 cm) to avoid discomfort to the eyes. After the sensor receives enough infrared radiation and generates a stable electrical signal, the eye surface temperature is calculated.

Show the user the correct measurement posture and method through the display module to reduce operating errors, ensure that the sensor's field of view (Field of View, FOV) is appropriate, and accurately align the target measurement area (such as the eyelid) to obtain accurate measurement results.

In a certain embodiment, the left and right eyes of the camera are determined according to whether the eye-rubbing action is based on the detection result of the smart watch or the positioning verification module. For example, if it is judged to be an eye rubbing action based on the RSSI and AOA of the smart watch, and the smart watch is worn on the left hand, then according to common eye rubbing gestures, it is prompted that the glasses corresponding to the captured image are images of the left eye.

In a certain embodiment, the camera module automatically analyzes the eye image based on a computer vision algorithm, including preprocessing the image.

In a certain embodiment, before analyzing the eye image, the image needs to be pre-processed, and the pre-processing includes color space conversion, histogram equalization, noise reduction processing, eye region identification, image cropping and scaling and rotation correction.

Convert the image from the RGB color space to a color space more suitable for eye analysis, such as HSV or Lab, enhance the image contrast through histogram equalization, and adjust the brightness distribution of the image to make the brightness of the image evenly distributed in the entire range, thereby improving The contrast of the image makes the eye structure more clearly visible. Noise in the image may interfere with feature extraction and eye anomaly detection. Replace each pixel value in the image pixel-by-pixel with the median of the pixel values in its neighborhood to eliminate salt and pepper noise, and perform convolution operations on the image, using Gaussian The kernel is smoothed.

In the converted color space, a threshold is set to differentiate the eyes from the surrounding skin. For example, set a suitable threshold in HSV color space to detect eye regions. The hue and saturation of the eye area are quite different from those of the skin. This feature can be used to distinguish the two, and the upper and lower threshold ranges are determined in the hue, saturation, and brightness, and the eyeball area is determined to fall within the threshold range of the three.

After the eye region is identified, the image is cropped to the region of interest (such as the eye) and scaled to an appropriate resolution for subsequent feature extraction and analysis. By detecting the position of the eyes, the tilt angle of the eye area is calculated, and the image is rotated and corrected.

In a certain embodiment, an eigenvalue vector related to abnormal eye symptoms is extracted in the camera module based on the preprocessed image, and the eigenvalue vector is a combination vector.

A support vector machine (SVM) classifier was trained as a color classifier using color features (pixel proportions of redness, vessel rupture, and conjunctival hyperemia). Divide the feature vectors and labels into training and testing sets, then use the training set to train an SVM classifier. Computes the output of the classifier using the color features of the test set, which can be class probabilities or class prediction scores.

Using LBP (Local Binary Patterns) descriptor to extract the texture features of the eye image. First convert the image to a grayscale image, then calculate the LBP features, and generate a histogram, calculate the principal components of the LBP texture feature histogram, and select the number of retained principal components (for example, the first 10 principal components), so that while reducing the dimension Retain a high amount of information, and then reduce the dimensionality of the LBP texture feature histogram.

Combining the output of the color classifier and the LBP texture feature after PCA dimensionality reduction into a feature vector, the combined feature vector includes: the output of the color classifier (category probability or category prediction score), and the LBP texture feature after PCA dimensionality reduction, which can be used Combine the two forms of feature splicing.

In a certain embodiment, for the training of the color classifier, the following process is required:

Prepare a dataset of eye images with labels (normal eyes, redness, ruptured blood vessels, and conjunctival congestion), and divide the dataset into training and testing sets. Convert all images to HSV color space and resize them to the same dimensions (64x64 pixels). For each abnormal symptom (redness, vascular rupture, and conjunctival hyperemia), the proportion of pixels of the corresponding abnormal color (such as red, dark red, and bright red) in the eye image was calculated. By setting different threshold ranges, a mask is created for each abnormal symptom, and then the proportion of corresponding color pixels in the eye image is calculated.

Construct the feature vector of the color classifier, and the feature vector can include the following elements: redness color feature (red pixel ratio), blood vessel rupture color feature (dark red pixel ratio), conjunctival hyperemia color feature (bright red pixel ratio). Create a vector of eigenvalues by stacking these features horizontally.

Train a Support Vector Machine (SVM) classifier using the extracted feature vectors of the color assignor and the corresponding labels. Divide the feature vectors and labels into training and testing sets, then use the training set to train an SVM classifier.

The output of the color classifier is the class probability or class prediction score.

In one embodiment, it is assumed that the output of the color classifier has four classes: normal eye (0), redness (1), ruptured blood vessel (2) and conjunctival hyperemia (3).

For a new eye image, the probabilities output by the classifier might look like this:

Probability: [[0.1,0.6,0.2,0.1]]

The physical meaning of the output probability is: the probability of normal eye (category 0) is 10%, the probability of redness (category 1) is 60%, the probability of blood vessel rupture (category 2) is 20%, and the probability of conjunctival hyperemia (category 3) The probability is 10%.

In a certain embodiment, a deep learning model (such as a convolutional neural network, a recurrent neural network, etc.) is used to automatically learn the monitoring parameters and the eigenvalue vector of the eye image, and the user's eye image is obtained after fusion and analysis of the relationship between different features. The degree of risk to local blood flow and optic nerve health.

In a certain embodiment, for each input feature (strength and frequency of rubbing gestures, eye temperature, eye pattern feature vector (combination of the output of the color classifier and the LBP texture feature after PCA dimensionality reduction)), respectively Create a subnetwork that can be a choice of Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), or Multilayer Perceptrons (MLP). For example, RNN can be used for time series data such as the intensity and frequency of eye rubbing gestures; CNN can be used for image feature data such as LBP texture features after PCA dimensionality reduction.

Each sub-network is trained separately using data labeled with known ocular blood flow conditions and optic nerve health risks, and each sub-network learns the relationship between specific input features and output targets.

Create a fusion layer that takes the output of all sub-networks as input. The purpose of the fusion layer is to learn how to combine information from different sub-networks to get the best prediction results. After the fusion layer, an output layer is added to predict the degree of risk for ocular blood flow conditions and optic nerve health.

In the present invention, the entire multimodal deep learning model is regarded as a whole, and a labeled data set is used for end-to-end training. The model will learn how to automatically fuse different types of features to achieve the best predictive performance.

In a certain embodiment, the fusion layer is a fully connected layer. The fully connected layer FC layer is a common layer structure in neural networks. In the fully connected layer, each neuron is connected to all neurons in the previous layer, and the input of each neuron is all neurons in the previous layer. Output.

The fully connected layer can realize the fusion of features and the mapping of output results. In multimodal deep learning, fully connected fusion layers take as input the outputs of sub-networks and learn how to combine these inputs to achieve the best prediction results.

In a certain embodiment, a sub-output layer is assigned to the eye blood flow condition and the optic nerve health, and each sub-output layer adopts a fully connected layer structure, wherein the number of neurons is equal to the number of categories of the corresponding target. If the risk The degree is divided into three categories: low, medium and high, and each sub-output layer contains 3 neurons. The activation function of the sub-output layer selects the Softmax function to convert the output into a probability distribution. Each sub-output layer will predict the category probability of a target, and classify the risk level with the highest category probability as the corresponding target (eye blood flow status and optic nerve health degree of risk to both objectives).

In one embodiment, the multi-label classification model predicts 3 classes for ocular blood flow status and optic nerve health: low risk (1), medium risk (2) and high risk (3). When the result of the health management algorithm module is that the eye data is abnormal or the health risk is increased, the reminder output module generates corresponding health reminders and eye care suggestions according to the prediction results.

In one embodiment, assume that the model predicts the output as follows: risk of ocular blood flow condition: low risk (1); risk of optic nerve health: high risk (3).

Based on this prediction result, the following health reminders and recommendations are generated:

1. Eye exercises and eye movement reminders, through smart watch vibration and/or display module reminders. Eye exercises include vision restoration exercises such as distance vision.

2. Carry out eye examination, especially the examination related to optic nerve.

The health reminders may be generated according to corresponding templates of the output groups of the two goals.

The smart watch-based eye health monitoring system provided by the present invention solves some problems in the prior art and realizes real-time monitoring of the user's eye health status, reduction of false alarms, automatic risk assessment and personalized health advice, etc. technical effect. This will provide a more convenient, accurate and intelligent eye health monitoring solution for people with eye diseases, especially children.

It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device, or may exist independently without being incorporated into the electronic device.

Computer program code for carrying out the operations of the present disclosure can be written in one or more programming languages, or combinations thereof, including object-oriented programming languages—such as Java, Smalltalk, C++, and conventional Procedural Programming Language - such as "C" or a similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as through an Internet service provider). Internet connection).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. Wherein, the name of a unit does not constitute a limitation of the unit itself under certain circumstances.

The preferred embodiments of the present invention have been introduced above, aiming to make the spirit of the present invention clearer and easier to understand, not to limit the present invention, all modifications, replacements and improvements made within the spirit and principles of the present invention , should be included within the scope of protection outlined in the appended claims of the present invention.

Claims (10)

1. Eye health monitoring system based on intelligent wrist-watch, the system includes intelligent wrist-watch, outside location module, bridge of nose bridge location check-up module, supplementary gesture confirms the module, intelligent wrist-watch includes heart rate sensor module, gesture motion tracking module, risk evaluation module, eye temperature measurement module, camera module, health management algorithm module, reminds output module and display module, its characterized in that:

the heart rate sensor module is used for periodically monitoring the heart rate of the wearer and confirming that the wearer is in a non-sleep state;

the gesture motion tracking module is used for starting a gesture tracking detection function of a wearer after confirming that the wearer is in a non-sleep state, and the gesture of the wearer comprises an eye rubbing gesture;

The gesture motion tracking module performs gesture tracking detection on a wearer, and comprises an external positioning module, a positioning verification module and an auxiliary gesture confirmation module;

the risk assessment module is used for determining whether the eye rubbing gesture triggers a risk assessment mechanism within the length of a preset time period or not based on the intensity and the frequency of the eye rubbing gesture, and prompting a wearer to conduct eye temperature measurement and eye image shooting according to prompts on the display module through the display module after the risk assessment mechanism is triggered;

the eye temperature measuring module is used for measuring the eye temperature in a way that a temperature sensor at the edge of the watch is close to eyelid;

the camera module is used for shooting an eye image, and automatically analyzing the eye image based on a computer vision algorithm, and comprises the steps of preprocessing the image and extracting a characteristic value vector related to abnormal symptoms of the eye based on the preprocessed image;

the health management algorithm module is used for analyzing the eye blood flow condition of the wearer and the risk degree of the optic nerve health degree according to the monitoring parameters and the eigenvalue vector of the eye image;

and the reminding output module is used for carrying out health reminding and giving corresponding eye health care suggestions when the algorithm detects that the eye data is abnormal or the health risk is increased.

2. The eye health monitoring system based on a smart watch according to claim 1,

to intelligent wrist-watch, external positioning module, bridge of nose bridge location verification module, supplementary gesture confirms the wearing position of module, include:

the intelligent watch is worn on the wrist part of the left hand or the right hand of the wearer;

the external positioning module is fixed on the head band, and the external equipment is positioned in the central position of the forehead of the wearer;

the bridge positioning verification module can be arranged on a bridge part of the bridge of the nose on the glasses worn by the wearer;

the auxiliary gesture confirmation module is a wrist strap and is worn on the other wrist without the intelligent watch.

3. The eye health monitoring system of claim 1, wherein the gesture motion tracking module performs gesture tracking detection of the wearer, comprising performing gesture tracking in combination with the external positioning module and the positioning verification module, comprising:

the wearer wears an external positioning module comprising a Bluetooth module on the head, and a positioning verification module is arranged at the nose bridge part of the glasses, wherein the positioning verification module comprises a gyroscope and an accelerometer;

after wearing glasses, a wearer starts a Bluetooth tracking function, an external positioning module records signal strength RSSI and an arrival angle AOA between the external positioning module and a positioning verification module as references, and then the Bluetooth tracking functions of the external positioning module and the positioning verification module are closed.

4. An eye health monitoring system based on a smart watch as claimed in claim 3, wherein,

when the gyroscope and the accelerometer of the positioning verification module detect position changes, an activation beacon is sent to the external positioning module, and the activation beacon is used for informing the external positioning module to start a Bluetooth tracking function;

and continuously carrying out Bluetooth tracking on the positioning verification module in the external positioning module, continuously monitoring the signal intensity RSSI and the arrival angle AOA of the positioning verification module by the external positioning module, and starting Bluetooth connection with external positioning equipment after the intelligent watch and the auxiliary gesture verification module receive the establishment request when judging that the change of the signal intensity RSSI or the arrival angle AOA exceeds a threshold value by comparing the recorded signal intensity RSSI and the arrival angle AOA between the positioning verification module.

5. The eye health monitoring system based on a smart watch as claimed in claim 4, wherein,

the external positioning module performs Bluetooth tracking on the intelligent watch and the auxiliary gesture confirmation module, compares RSSI and AOA between the intelligent watch and the auxiliary gesture confirmation module with recorded signal strength and AOA serving as references, and starts a timer when at least one positioning data in the tracked intelligent watch and auxiliary gesture confirmation module meets the conditions;

And judging that the eye-rubbing action is performed when the timer reaches the set timer threshold.

6. An eye health monitoring system based on a smart watch as claimed in claim 4 or 5, wherein,

the arrival angle AOA is an included angle between a signal propagation path and a reference direction;

when the relative position between the external positioning module and the positioning verification module is kept constant, the arrival angle AOA is also kept constant;

determining a reference direction according to the direction of the external positioning device, wherein the reference direction can be changed along with the movement and rotation of the device;

and selecting the horizontal direction of the external positioning device after the head is fixed as the reference direction of AOA measurement.

7. The eye health monitoring system of claim 5, wherein the at least one of the tracked smart watch and the auxiliary gesture confirmation module positioning data satisfies a condition, comprising:

the absolute value delta RSSI of the gap between the detected signal intensity RSSI of the intelligent watch or the auxiliary gesture confirmation module and the recorded signal intensity RSSI serving as a reference and the positioning verification module is smaller than a threshold value, and the AOA of the detected intelligent watch or the auxiliary gesture confirmation module is in a preset angle range.

8. The eye health monitoring system based on a smart watch according to claim 5,

when the external positioning module detects positioning data meeting the conditions, starting timing, wherein the starting time is the starting time;

continuously monitoring hand motion data in the timing process, stopping the timer under a second condition, wherein the stopping time is the ending time; the second condition is: when the hand motion data is detected to be no longer in accordance with the characteristics of the eye rubbing gesture, the eye rubbing action is stopped, at least one of the delta RSSI and the AOA is not in accordance with the requirements of the eye rubbing gesture, specifically, the delta RSSI of the intelligent watch is larger than a threshold value or the AOA is not in a preset angle range, and the delta RSSI of the auxiliary gesture confirmation module is larger than the threshold value or the AOA is not in the preset angle range;

after the timer is stopped, the gesture intensity is determined according to the accumulated eye-rubbing action duration.

9. The eye health monitoring system based on a smart watch according to claim 1,

when the risk assessment mechanism is triggered, the display module can prompt a wearer to measure the eye temperature and shoot an eye image by popping up prompt information or making sound/vibration;

the prompt information comprises the forms of characters, images, voices or animations, and the like, and guides the wearer how to correctly measure the eye temperature and shoot the eye images;

After the wearer completes eye temperature measurement and eye image shooting, the display module displays the measurement result.

10. The eye health monitoring system based on a smart watch according to claim 1,

automatically analyzing the eye image based on a computer vision algorithm in the camera module, wherein the automatic analysis comprises preprocessing the image;

the preprocessing comprises performing, color space conversion, histogram equalization, noise reduction processing, eye region identification, image clipping, scaling and rotation correction;

extracting a characteristic value vector of an eye abnormal symptom from the preprocessed image in the camera module, wherein the characteristic value vector is a combined vector, and the method comprises the steps of combining the output of a color classifier and LBP texture characteristics subjected to PCA dimension reduction into a characteristic vector, and combining the two characteristics in a spliced mode;

and learning the characteristic value vector of the monitoring parameter and the eye image by using a deep learning model, and analyzing and fusing the relation among different characteristics to obtain the risk degree of the blood flow condition of the eyes of the user and the optic nerve health degree.