WO2023002503A1

WO2023002503A1 - A system and a method for synthesization and classification of a micro-motion

Info

Publication number: WO2023002503A1
Application number: PCT/IN2022/050647
Authority: WO
Inventors: Ranjani Ramesh; Ramesh Venkatesan
Original assignee: Individual
Current assignee: Individual
Priority date: 2021-07-19
Filing date: 2022-07-17
Publication date: 2023-01-26
Anticipated expiration: 2024-01-19

Abstract

The present invention relates to a system and a method for synthesization and classification of a micro-motion, comprising: a communication device for collecting plurality of video samples, a processing unit to simultaneously process the said samples by using: an image processing technique, a region-based shape extracting means, a labelling means for labelling of the micro-motion videos observed in an extracted result from the said extracting means, an overlapping means for composing video samples resulted from magnification means and labelling means, a server to upload the said plurality of video samples collected by means of the said communication device, and a classification means for classifying the resulted video samples. Further, in accordance with present invention, an image processing operator is used between the original video and the magnified video which yields a suitable heatmap that can pick up on the subtle changes in position.

Description

FIELD OF INVENTION:

[001] The present invention relates to a system and a method for synthesization and classification of a micro-motion. More particularly, the present invention relates to the system and the method for synthesization and classification of micro-motions in the hand by using a communication device and various image processing techniques.

BACKGROUND OF INVENTION:

[002] Digital media, particularly videos, play an essential role in the documentation of information, in today’s world. With the help of computer vision and artificial intelligence, important data stored in videos can be extracted and used in multiple domains with the help of Computer Vision and AI. The videos of the hands of human beings reveal essential information in the form of micro - motions such as the throbbing of the veins due to the pulse or tremor components. The remote monitoring of vitals such as pulse and respiration rates of a person and remote detection of neurodegenerative diseases such as Parkinson’s disease are essential in times such as the present COVID - 19 pandemic.

[003] Non - patent literature such as Williams et al.’s proposition details a method to aid in clinical detection of Parkinson’s Disease (PD) using a computerized magnification of tremors in atremulous PD patients. Hilledondsberg et al.’s approach aims to enhance the fasciculations in Amyotrophic Lateral Sclerosis (ALS) patients using Eulerian Video Magnification. These non-patent literatures have shown promising results in improving the accuracy of detection of diseases by specialists.

[004] Patent document such as US 20180263703 discloses a telepresence device which may autonomously check patients. The telepresence device may determine the frequency of checking based on whether the patient has a risk factor. The telepresence device may include an image sensor, a thermal camera, a depth sensor, one or more systems for interacting with patients, or the like. The telepresence device may be configured to evaluate the patient's condition using the one or more sensors. The telepresence device may measure physiological characteristics using Eulerian video magnification, may detect pallor, fluid level, or fluid color, may detect thermal asymmetry, may determine a psychological state from body position or movement, or the like. The telepresence device may determine whether the patient is experiencing a potentially harmful condition, such as sepsis or stroke, and may trigger an alarm if so. To overcome alarm fatigue, the telepresence device may annoy a care provider until the care provider responds to an alarm.

[005] Patent document such as US 20210145359 discloses such embodiments of tissue monitoring and therapy systems and methods. In some embodiments, a monitoring and therapy system comprises collecting video images of a tissue site, amplifying said video images via Eulerian Video Magnification, and determining a treatment parameter from the amplified video images detectable by Eulerian Video Magnification. If the treatment parameter differs from a threshold, an alert may be generated.

[006] However, as it can be seen from the above patent and non-patent documents that a basic framework that can be used for easy detection of micro-motions in the hand by using various image processing techniques is lacking.

OBJECTIVE OF INVENTION:

[007] The primary objective of the present invention is to synthesize and automatically classify micro-motions of a hand by using a communication device.

[008] Another objective of the present invention is to synthesize and automatically classify micro-motions of the hand by using machine learning.

[009] Yet another objective of the present invention is to process a plurality of video samples to identify micro-motions by using various techniques which are including but not limited to an image processing technique, a region-based shape extracting means, a labelling means and an overlapping means etc.

[010] Other objects and advantages of the present invention will become apparent from the following description taken in connection with the accompanying drawings.

SUMMARY OF INVENTION:

[011] In accordance with the present invention, a system for synthesization and classification of a micro-motion is disclosed, comprising: a communication device for collecting plurality of video samples, a processing unit to simultaneously process the said samples by using: an image processing technique, a region-based shape extracting means, a labelling means for labelling of the micro-motion videos observed in an extracted result from the said extracting means, an overlapping means for composing video samples resulted from magnification means and labelling means, a server to upload the said plurality of video samples collected by means of the said communication device, and a classification means for classifying the resulted video samples.

[012] Further, in accordance with the present invention, a method for synthesization and classification of a micro-motion is disclosed, comprising the steps of: collecting, a plurality of video samples by means of a communication device; processing the said video samples by using: an image processing technique, a region-based shape extracting process and manually labeling of the micro-motion observed in an extracted result from the said extracting process; a magnification process and manually labelling of the micro-motion observed in the said extracted result; a server to upload the said plurality of video samples collected by means of the said communication device; and classifying the said video samples.

[013] An image processing operator is used between the original video and the magnified video which yields a suitable heatmap that can pick up on the subtle changes in position, i.e., the small motions of the various parts.

BRIEF DESCRIPTION OF DRAWINGS: [014] The present invention will be better understood after reading the following detailed description of the presently preferred aspects thereof with reference to the appended drawings, in which the features, other aspects and advantages of certain exemplary embodiments of the invention will be more apparent from the accompanying drawings in which: [015] Figure 1 depicts the still image from ideal video captured through a communication device.

[016] Figure 2 depicts a particular way of capturing video samples to acquire best possible results to process the said video samples.

[017] Figure 3 depicts the processing of the said video using an image processing technique. [018] Figure 4 depicts the processing of the said video by using a region-based shape extracting means.

[019] Figure 5 depicts the plurality of images from the processed videos depicting Vein throbbing, Finger movement, movement of the stomach. [020] Figure 6 is a plurality of waveforms depicting the movement of the stomach due to respiration.

[021] Figure 7 depicts the waveforms of all pixels of a particular video.

[022] Figure 8 depicts a heat mapped results of a hand.

[023] Figure 9 depicts an incorrect depiction of key points.

[024] Figure 10 depicts correct detection of all key points.

[025] Figure 11 describes a method used to classify motions in the hand.

[026] Figure 12 describes user-method interaction and automated detection of normal / abnormal hand motion

DETAILED DESCRIPTION OF THE INVENTION

[027] The following description describes various features and functions of the disclosed device and methods with reference to the accompanying figures. In the figures, similar symbols identify similar components, unless context dictates otherwise. The illustrative aspects described herein are not meant to be limiting. It may be readily understood that certain aspects of the disclosed system, method and apparatus can be arranged and combined in a wide variety of different configurations, all of which are contemplated herein.

[028] These and other features and advantages of the present invention may be incorporated into certain embodiments of the invention and will become more fully apparent from the following description and claims or may be learned by the practice of the invention as set forth hereinafter.

[029] Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the invention. In addition, descriptions of well-known functions and constructions are omitted for clarity and conciseness.

[030] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but, are merely used to enable a clear and consistent understanding of the invention. Accordingly, it should be apparent to those skilled in the art that the following description of exemplary embodiments of the present invention are provided for illustration purpose only and not for the purpose of limiting the invention.

[031] It is to be understood that the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise.

[032] It should be emphasized that the term “comprises/comprising” when used in this specification is taken to specify the presence of stated features, integers, steps or components but does not preclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

[033] In accordance with the present invention, a plurality of video samples was acquired by using a communication device. These plurality of video samples were taken by keeping the hand steadily on the arm of a furniture unit, facing down, and keeping the camera of a communication device on a stand at a particular distance from the subject as depicted in Figure 1 and Figure 2.

[034] More specifically, the furniture unit, in accordance with the Figure 2 of the present invention, is including but not limited to a chair. Particular distance between the stand and the subject is one meter to acquire best possible result to process the said plurality of video samples.

[035] These communication devices, in accordance with the present invention, are including but not limited to a mobile phone, laptop, tablet etc.

Pre-processing for classification:

[036] The plurality of video samples is then subjected to various pre-processing techniques for classification of micro-motions, which are including but not limited to the following:

[037] A region-based shape extracting means - The plurality of videos is firstly subjected to the region-based shape extracting means. Through the region-based shape extracting means, a plurality of key points is detected and these key points are essential to mark location of the micro-motions.

[038] The said region-based shape extracting means also join these points to form a skeleton like structure. Figure 3 and Figure 4 of the present invention depicts the illustration of identifying of key points and forming a skeleton like structure through region-based shape extracting means.

[039] The region-based shape extracting means is a skeletonization of the said plurality of video samples.

[040] The said plurality of videos is subjected to a magnification means parallelly. By magnifying the said plurality of videos, various micro-motions in the hand such as the pulse, tremor components, random movements in the hand, etc., become visible to the human eye.

[041] Further, in accordance with Figure 3 and 4, the said magnified results are then processed by an image processing technique to magnify larger micro-motions. Further, an image processing operator is used between the original video and the magnified video which yields a suitable heatmap that can pick up on the subtle changes in position, i.e., the small motions of the various parts of the hand. Particularly, the image processing technique is a heat mapping process.

[042] Finally, the results from the heat-mapped and skeletonized results are averaged to overlap the skeletonization and heat- mapped results. This overlapped video is used to label motion. The intensity of colour and key points detected in the hand are essential features for classifying the motions in different parts of the hand.

Labelling:

Stage 1 - Labelling motions occurring in original video submitted:

[043] In this first stage, only the original video was considered. Extra background motions such as the movement of people walking in the background, the movement of a swing, etc., and the movement of the entire frame was labelled as background motion.

Stage 2 - Labelling motions from Magnified Results:

[044] Figure 5 - In this stage, the magnified result is used to label certain other motions. These are finger, hand movements or tremors and throbbing of blood vessels in hand. In specific videos where subjects recorded their entire middle bodies and hands, movement of the stomach during breathing was also labelled. Uploading of Video Samples and Waveform Creation:

[045] The said plurality of video samples collected by the communication device are uploaded to a server. Particularly, a cloud - server for further extraction of a waveform.

[046] Figure 6 - The waveforms are created using the RGB values of select pixels in the right stomach. These pixels are selected using the pre-processing steps detailed above under pre-processing. Within the hand, pixels are picked using the results of the heatmap, skeletonization. Pixels where all pre-processing steps match are selected.

[047] For selection of pixels outside the hand, only heatmap and manual markings are used. This is how the points for background motion and stomach motion are found. Initially, all three-color channels’ waveforms were plotted. However, it was observed that since colour magnification did not take place across time, the average of the three channel waveforms could be taken as the final waveform for the point.

[048] Figure 7 - Further, these waveforms are synthesized for every selected pixel from video.

Classification:

[049] The labelled waveforms obtained were finally passed into a classification means for classification. The waveforms were limited to the central 10 seconds of data from each subject for inputting into the classification means. These waveform inputs were vectors of the averaged RGB values of the pixels in each region of each frame, i.e. pixels of each subject under a particular label. The classification means is a KNN model algorithm.

[050] For classification, the kNN model was used. Multiple past works and papers have assessed the usefulness of different machine learning (Bagged Trees, Decision Trees, Weighted and Unweighted kNNs, etc.) and deep learning models (Deep Belief Network, Deep Neural Network and Long-Short Term Memory, and Recurrent Neural Networks, etc.) for the classification of waveforms. Alghamdi et al.5 detail the use of kNNs for waveform classification and show promising results. Additionally, for this project, the data collected is highly over-represented by the ‘hand motion’. There is almost 3 times the amount of data for the ‘hand motion’ as there is for ‘background’ or ‘vein motion’. [051] Due to this bias in data, kNNs were preferred over Deep Learning methods. For this project, Google Colab was used to train and test the kNN. The data was split into three random mini - cohorts for training. The cohorts were each trained on a different model. The outputs of the three models were plotted into confusion matrices to assess the performance of the model. Additionally, the accuracy of the models with respect to the value of ‘k’ nearest neighbours was assessed by plotting graphs of the accuracy versus the value of ‘k’. This analysis revealed that the highest accuracy was when k = 3.

Method used to Classify Motions in the Hand:

[052] Figure 11: In accordance with Figure 11 of the present invention, a method for synthesization and classification of micro-motions in the hand are disclosed, comprising the steps of: collecting, a plurality of video samples by means of a communication device; processing the said video samples by using: an image processing technique, a region-based shape extracting process and manually labeling of the micro-motion observed in an extracted result from the said extracting process; a magnification process and manually labelling of the micro-motion observed in the said extracted result; a server to upload the said plurality of video samples collected by means of the said communication device; and classifying the said video samples. Further, the image processing operator is used between the original video and the magnified video which yields a suitable heatmap that can pick up on the subtle changes in position, i.e., the small motions of the various parts.

[053] Figure 12: In accordance with Figure 12 of present invention, it is envisioned that the automated machine learning model for abnormal hand motion detection is available on a cloud server. A user who wants to find out if they have normal or abnormal hand motion records a video and uploads it to the server. The server passes the (temporal) waveform data for each pixel to the kNN which classifies the pixel whether it belongs to the hand. The waveforms corresponding to the entire cluster of such hand pixels with significant motion is next passed to Eulerian micro-motion magnification. These magnified waveforms are then passed to the normal/abnormal classifier network. This network returns a probability of the waveforms coming from a normal hand, and a decision on abnormal/normal is then obtained based on a threshold of 0.5 for this probability. RESULTS AND ANALYSIS:

Magnification Process (Eulerian Video Magnification):

[054] The Eulerian Video Magnification performed satisfactorily without any visible artefacts or noise in the videos of all different subjects. Subjects had different recording environments such as different lighting settings, different frame sizes, and different cameras and camera angles.

Image Processing Technique (Heat Mapping):

[055] Figure 8 - Heat-mapping was done by performing the Bitwise OR operation on the original video and the magnified video. This heat mapping highlighted the regions in the hand which had significant motion after motion magnification. The analysis of results was done using various parameters. One was the closeness of the camera to the hand. It was observed that none of the videos which were far away from the camera yielded useful results, and thus, a correlation can be set up between the distance of the hand from the camera and the quality of the heat-map produced.

[056] The quality of the heat map is determined by the ability of the bitwise OR operator to detect and highlight the motions produced after magnification. This quality was determined manually.

Skeletonization:

[057] Figure 9 - In this method, the skeletonization algorithm was modified to detect key points and draw circles but not draw the lines joining the key points to form the skeleton. The algorithm doesn’t show consistency in its performance. This can be attributed to the fact that the algorithm has been trained for hand pose estimation and not a still, downward-facing hand. Additionally, most hand detection and skeletonization algorithms are trained on images of the palm and the hand facing upwards rather than the downward back of the hand used in this experiment.

[058] Figure 10 - It has been observed that the algorithm detects the hands’ keypoints accurately when the hand is placed flat on a surface and fully visible rather than resting on the arm of a chair where the subject’s resting hand is not detected but the key points of the hand on the chair are detected. [059] Overall Results - Classification of Motions after Pre-Processing: These techniques have gained accuracies of ~ 90.33%, 93.37% and 89%. These high accuracies can be attributed to the large amount of data used for training, the correct selection of pixels which were artefact-free, appropriate pre - processing of raw data, and use of the best value of k, where k = 3

Claims

We Claim:

1. A system for synthesization and classification of a micro-motion, comprising: a. a communication device for collecting plurality of video samples b. a processing unit to simultaneously process the said samples by using i. an image processing technique, ii. a region-based shape extracting means, iii. a labelling means for labelling of the micro-motion videos observed in an extracted result from the said extracting means, iv. an overlapping means for composing video samples resulted from magnification means and labelling means, c. a server to upload the said plurality of video samples collected by means of the said communication device; and d. a classification means for classifying the resulted video samples wherein an image processing operator is used between the original video and the magnified video which yields a suitable heatmap that can pick up on the subtle changes in position, i.e., the small motions of the various parts.

2. The system as claimed in claim 1 wherein: a plurality of key points from the region-based shape extracting means and an intensity of color from the data visualization technique are extracted respectively.

3. The system as claimed in claim 1 wherein: a waveform of the said plurality of video samples is created based on the uploaded plurality of video samples on the cloud-based server and the said data relating to waveform is extracted and passed over to the classification means to classify the waveforms.

4. The system as claimed in claim 1 wherein: the image processing operator is a bitwise OR operator used between the original video and the magnified video which yields a suitable heatmap that can pick up on the subtle changes in position.

5. The system as claimed in claim 1 wherein the communication device is a mobile phone or a laptop.

6. The system as claimed in claim 1 wherein the image processing technique is a heat mapping technique.

7. The system as claimed in claim 1 wherein the region-based shape extracting means is a skeletonization means.

8. The system as claimed in claim 1 wherein the magnification means is a Eulerian Video Magnification means.

9. The system as claimed in claim 1 wherein the classification means is a KNN classifier.

10. A method, comprising: a. collecting, a plurality of video samples by means of a communication device; b. processing the said video samples by using: i. an image processing technique, ii. a region-based shape extracting process and manually labeling of the micro-motion observed in an extracted result from the said extracting process; iii. a magnification process and manually labelling of the micro-motion observed in the said extracted result; iv. a server to upload the said plurality of video samples collected by means of the said communication device; and v. classifying the said video samples wherein an image processing operator is used between the original video and the magnified video which yields a suitable heatmap that can pick up on the subtle changes in position, i.e., the small motions of the various parts.

11. The method as claimed in claim 10 wherein: a plurality of key points from the region-based shape extracting process and an intensity of color from the data visualization technique are extracted respectively.

12. The method as claimed in claim 10 wherein: the image processing operator is a bitwise OR operator used between the original video and the magnified video which yields a suitable heatmap that can pick up on the subtle changes in position.

13. The method as claimed in claim 10 wherein: a waveform of the said plurality of video samples is created based on the uploaded plurality of video samples on the cloud-based server and the said data relating to waveform is extracted and passed over to the classification means to classify the waveforms.

14. The method as claimed in claim 10 wherein the communication device is a mobile phone or a laptop.

15. The method as claimed in claim 10 wherein the image processing technique is a heat mapping process.

16. The method as claimed in claim 10 wherein the region-based shape extracting process is a skeletonization process.

17. The method as claimed in claim 10 wherein the magnification process is a Eulerian Video Magnification.

18. The method as claimed in claim 10 wherein the classification method is a KNN model.

Dated this 19^th day of July, 2021