CN120108640A - A sports training recommendation method and system based on multimodal data fusion - Google Patents

Abstract

The invention discloses a sports training recommendation method and system based on multi-mode data fusion. The method comprises the following steps of obtaining disease types and clinical characteristics of an aged disabled patient, fusing and constructing character portraits, obtaining static characteristics, generating and pushing a first exercise training scheme based on the character portraits, collecting structured data, text data and image data during exercise training, inputting the data into a collaborative filtering model, and combining the character portraits to generate and pushing a next exercise training scheme. By utilizing the invention, an individual targeted exercise training scheme can be formulated, and the cognitive, emotion, movement, speech and other abilities of the aged disabled patient are obviously improved.

Description

Exercise training recommendation method and system based on multi-modal data fusion

Technical Field

The invention relates to a sports training recommendation method based on multi-mode data fusion, and also relates to a corresponding sports training recommendation system, belonging to the technical field of medical care informatics.

Background

According to the double coding theory in cognitive theory, humans are generally considered to have two main information processing systems, speech systems and nonspeech systems. These two systems differ in coding scheme. Studies have shown that brain memory effects and memory speed for visual material are superior to semantic memory, suggesting that nonverbal systems may have advantages in processing images and spatial information. Although the two systems are functionally independent, they can also be mutually activated and affected.

There is a theory in the cognitive field called architecture, the core of which is to say that knowledge is built by individual interactions with the environment. Through communication and cooperation with others, individuals can build and revise their own understanding, and cognitive conflicts can arise when individuals encounter new information that conflicts with existing knowledge. Such conflicts prompt individuals to re-evaluate and adjust their cognitive structure to accommodate the new information. Moreover, knowledge construction is a continuously evolving process. As the experience of individuals increases and cognitive abilities develop, their knowledge structures evolve. Knowledge is built in a specific context, so learning in an actual context can better facilitate knowledge understanding and application

However, there is no current exercise training regimen that fuses information from the verbal and nonverbal systems in the treatment of elderly disabled patients. Moreover, for the disabled risk group, a personalized targeted exercise training scheme needs to be formulated to promote or improve the decline of the cognitive, emotional, motor, speech and other abilities thereof.

Disclosure of Invention

The invention aims to provide a sports training recommendation method based on multi-modal data fusion.

The invention aims to provide a sports training recommendation system based on multi-mode data fusion.

In order to achieve the technical purpose, the invention adopts the following technical scheme:

According to a first aspect of an embodiment of the present invention, there is provided a sports training recommendation method based on multi-modal data fusion, including the steps of:

step one, aiming at the elderly disabled patient, obtaining the disease type and clinical characteristics, demographic information, exercise investigation information and exercise behavior characteristics of the patient so as to fuse and construct character portraits and obtain static characteristics;

step two, generating and pushing a first exercise training scheme based on the character portrait;

Step three, collecting structured data, text data and image data during exercise training;

inputting the data in the third step into a collaborative filtering model, and combining the character portraits to generate and push a next exercise training scheme;

The collaborative filtering model adopts a plurality of automatic encoders and collaborative filtering networks, adopts a graph embedding technology to map entities and relations in a graph structure into a low-dimensional vector space, is realized by a Bayes TransR embedded model, a Bayes stack noise reduction self-encoder and a Bayes stack convolution self-encoder, and is used for respectively converting structured data, text data and image data into structural vectors, text vectors and image vectors, so as to obtain project hidden vectors and user hidden vectors, and is used for collaborative filtering learning to recommend the next motion training scheme.

Preferably, the fourth step comprises the following sub-steps:

1) Determining key variables as one-dimensional data, two-dimensional data and three-dimensional data;

2) Creating a data matrix according to the key variables;

3) Adding hysteresis characteristics as four-dimensional data;

4) Converting the feature vector into a feature vector by using an embedding technology;

5) Based on the feature vectors, project hidden vectors and user hidden vectors are generated for collaborative filtering learning to recommend a athletic training regimen.

Preferably, in the sub-step 4, the entity and the relation are converted into vectors by using a Bayesian TransR knowledge graph embedding method, text data are converted into text vectors by using a Bayesian sparse self-encoder embedding method, and features of the image are extracted by using a Bayesian sparse convolution self-encoder and are converted into image vectors.

Wherein preferably, the multi-modal features and character portrayal features extracted for the patient condition are used as self-variable data, and the sport prescription suggestions clinically issued by doctors are used as dependent variables and are input into the collaborative filtering model;

The loss function of the collaborative filtering model is cosine contrast loss, and is used for maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs under the constraint of the margin.

In the substep 1, preferably, the one-dimensional data is taken as a W axis, the two-dimensional data is taken as a Y axis, the three-dimensional data is taken as an X axis, and the four-dimensional data is taken as a Z axis, so as to construct a personal data space.

Wherein preferably the one-dimensional data comprises time variables, the two-dimensional data comprises a plurality of variables describing movement preferences, the three-dimensional data comprises variables describing physical and psychological characteristics, and the four-dimensional data comprises post-training state data.

Preferably, when the data matrix is created according to the key variables, different diseases correspond to different key variables, and the character portraits comprise a plurality of attributes including cognitive cortex injury, physical function injury and emotion control force injury attributes.

Wherein preferably, the exercise training recommendation method further comprises the following steps:

Step five, structured data, text data and image data in the exercise training period are collected for a plurality of times in the mode of step three, so that exercise training period data are obtained;

Step six, judging whether the exercise training scheme is the Nth one, if so, entering a step seven, otherwise, returning to a step four, and generating a next exercise training scheme according to the characteristics in the step five, wherein N is a positive integer;

Step seven, based on the data acquired by the patient in the step three and the step five, the RRN model is input together to iteratively optimize the exercise training scheme,

Wherein the RRN model is optimized using the following function:

wherein θ represents a parameter to be learned; For a set of tuples observed in a training set, the tuples include patients, exercise training schemes, and time sequences; r _ij |t represents the patient i's score for motion j at time t; Is a predicted value of R _ij t, and R represents a regularization function.

Wherein preferably the RRN model employs an alternating subspace descent strategy, assuming that the motion state is fixed, without propagating gradients into these motion training protocol sequences, while back propagating gradients of all scores of the patient to update patient sequence parameters, and then switching between updating the user sequence and updating the motion training protocol sequence.

According to a second aspect of an embodiment of the present invention, there is provided a multi-modal data fusion-based exercise training recommendation system, including a processor and a memory, wherein the memory is coupled to the processor and is configured to store one or more programs, which when executed by the processor, cause the processor to implement the multi-modal data fusion-based exercise training recommendation method.

Compared with the prior art, the invention has the following beneficial technical effects:

1) The fusion analysis system for the multi-mode data in collaborative filtering and self-coding learning is used, so that the multi-source information is ensured to enter a recommendation model, an analysis source is increased, continuously iterated motion pushing content based on the multi-dimensional motion data of a patient can be realized, the training can be most suitable for the individual and current state of the patient, and the training effect is improved;

2) Constructing a training period data system, dividing the period of a motion training process into a year period, a big period, a middle period and a small period, manufacturing different time sequence data sets, adding time sequence dimension characteristics when a model is introduced to perform motion recommendation calculation, thereby more fully and regularly applying historical motion data and assisting in the generation of a periodic scheme of motion recommendation;

3) Constructing a multi-stage patient portrayal system, gradually and deeply deconstructing and evaluating attribute features, behavior features and comprehensive (including but not limited to cognition, emotion, movement, speech and the like) features of a patient, ensuring that a label system of an individual has fine granularity, accurately recommending movement for a subsequent input model, and providing a large amount of effective information of a patient sequence;

4) And combining an RRN model, and using an effective calculation mode of a circulation network on time sequence data to realize effective backtracking on data of different time periods of movement of a patient, thereby ensuring consistency and sustainability of movement recommendation and improving patient compliance.

Drawings

FIG. 1 is a flow chart of a training recommendation method based on multi-modal data fusion in a first embodiment of the present invention;

FIG. 2 is a diagram illustrating multi-modal data fusion in accordance with a first embodiment of the present invention;

FIG. 3A is a flow chart of generating a training solution based on collaborative filtering model in a first embodiment of the present invention;

FIG. 3B is a diagram illustrating an individual data space according to a first embodiment of the present invention;

FIG. 3C is a schematic diagram of a collaborative filtering model according to a first embodiment of the present invention;

FIG. 4 is a flowchart of a training recommendation method based on multi-modal data fusion according to a second embodiment of the present invention;

fig. 5 is a schematic structural diagram of an RRN (recurrent neural network) model according to a second embodiment of the present invention;

Fig. 6 is a schematic structural diagram of a training recommendation system based on multi-modal data fusion according to a third embodiment of the present invention.

Detailed Description

The technical contents of the present invention will be described in detail with reference to the accompanying drawings and specific examples.

The technical conception of the embodiment of the invention is that the recommendation method based on the content is combined with the collaborative filtering model, so that the characteristic learning and the project recommendation process of the patient or the project are integrated into a unified framework. Firstly, learning hidden features of a patient or a project by using various deep learning models, constructing a unified optimization function by combining a collaborative filtering method to perform parameter training, and then obtaining final hidden vectors of the patient and the project by using the trained models, thereby realizing the recommendation of a training scheme of the movement of the patient (mainly the aged disabled patient).

First embodiment

As shown in fig. 1 to 3, the exercise training recommendation method based on multi-modal data fusion according to the first embodiment of the present invention at least includes the following steps.

Step one, aiming at the elderly disabled patient, obtaining the disease type and clinical characteristics, demographic information, exercise investigation information and exercise behavior characteristics of the patient so as to fuse and construct character portraits and obtain static characteristics.

The method comprises the steps of constructing a core portrait by utilizing a basic model in exercise training recommendation based on multi-mode data fusion and utilizing disease types and clinical characteristics of a patient, constructing a basic portrait by utilizing demographic information, exercise investigation information and exercise behavior characteristics of the patient, and constructing a role portrait of the patient by fusing the core portrait and the basic portrait. In this process, a static feature is obtained.

The method for constructing the basic model can be referred to in the prior patent application with the application number 202411764001.3, named as "figure construction method, exercise training recommendation method and system for disabled risk group", and is briefly described as follows:

The base model is constructed using data from a plurality of patients based on the following steps:

s1, determining portrait mapping key elements according to a pathological mechanism;

s2, constructing a core portrait based on key elements;

s3, constructing a basic portrait according to demographic information, exercise investigation information and exercise behavior characteristics;

S4, constructing the role portrait based on the core portrait set and the basic portrait set.

The key elements comprise cognitive cortex injury, physical function injury and emotion control force injury, the attributes of the core image of the incapacitation disease comprise cognitive cortex injury, physical function injury and emotion control force injury, the different attributes comprise different labels, and the label data are derived from diagnosis information or investigation information of a patient. The exercise investigation information includes exercise preferences and the like.

As shown in fig. 2, cognitive cortical impairment can lead to cognitive dysfunction, manifested in complex attention, executive function, learning and memory, language, and the like. The physical function injury can cause dyskinesia, and the dyskinesia is expressed in muscle strength, muscle tension, coordination ability, balance ability and the like. Mood controlling stress impairments can lead to psychobehavioral disorders, manifesting as hallucinations/delusions or depression/anxiety etc. in psychology, and also manifesting as aggression/hoarding or depression/wander etc. in behavioural aspects. By evaluating the health condition (disorder or disease) of these patients, the label of the patient can be mined.

The label of the A-type cognitive cortex injury at least comprises:

A1. The patients with the forehead She Sunshang can have the symptoms of mental disorder such as hypomnesis, apathy expression and the like, and can also have the symptoms of epilepsy, monopplegia, movement aphasia, hyperpyrexia, hyperhidrosis, vision and smell abnormality and the like.

A2. the parietal lobe injury can cause complex sensory disturbance of contralateral limb, body image disturbance, calculation dysfunction and the like.

A3. temporal lobe injury patients may have sensory aphasia, naming aphasia, fantasy, epilepsy, mental abnormalities, dysmnesia, altered vision, etc.

A4. occipital lobe injury, a visual disorder in patients.

A5. island She Sunshang patients may have increased salivation, nausea, satiety, and other viscera motor and sensory disturbances.

A6. the patients with the edge leaf injury can have mental abnormalities such as emotion, memory abnormality and illusion, and viscera movement disorder.

A7......

The label of the B-type body function injury at least comprises:

B1. Headache pain

B2. nausea or vomiting

B3. fatigue or drowsiness

B4. Tremor, stiffness and bradykinesia

B5. dizziness or loss of balance

B6. sensory problems, e.g. blurred vision, tinnitus, bad breath or changes in smell

B7. sensitive to light or sound

B8......

The label of the C-type emotion control force injury at least comprises:

C1. mood changes or mood swings

C2. Depression or anxiety

C3. Difficulty in falling asleep

C4. Than usual sleepiness

C5......

And (5) quantifying and calibrating the attribute characteristics of the core portrait to construct the core portrait. The attributes of the core portrait can be in one-to-one correspondence with the core key elements or can be more than the core key elements. The labels of each attribute form a label word stock, and different label word stocks represent different disease course, damage types and disabling characteristics. In this step, the patient's core portrayal signature is determined based on the patient's diagnosed disease type and clinical characteristics.

Demographic information, exercise investigation information, exercise behavior characteristics and social participation information are obtained through investigation or medical records and other modes, the label of the patient is obtained, and a basic portrait is constructed.

Demographic information including, for example, gender, age, educational level, etc., and athletic research information including, for example, athletic preferences, activity abilities, etc. Among other sports preferences include sports types (e.g., rope jump, running, etc.), sports venues (e.g., gym, park, etc.), sports atmospheres (calm, dynamic), etc. The activities include BADL (basic daily living activities) and IADL (instrumental daily living activities).

The athletic performance characteristics include frequency of motion, period of motion, and the like.

The social participation information includes participation activities inside the home and participation activities in the society. The participation activities in the family include family decision making, child tending and the like, and the participation activities in the society include work learning, entertainment activities and the like.

The basic representation is not a core element for influencing the incapacitation, but is taken as a basic characteristic, has synergistic influence on the information of the core representation such as the course of disease, damage type, incapacitation characteristics and the like of a patient, and is taken into a representation system as a potential influence variable to participate in analysis.

The attributes of the basic portrait at least comprise:

The sex attribute is a tag which is widely used, the favorites of people with different sexes on different contents are obviously different, and the basic proportion condition of the patient group is easier to analyze through natural attribute tags such as age, region, academic, occupation, marital condition, child condition and the like;

Vertical attribute, which reflects the exercise requirement of the patient, such as different types of exercise preference, etc.;

Training attribute, namely training attribute is also an important attribute category, which is helpful for judging the movement intention, movement period and movement frequency of the patient;

Tag attributes when a patient begins to use the system to generate a first piece of data, the system can assign the first tag, i.e. new, to the patient, and then the patient with low frequency, active and high frequency can be separated gradually as the patient accumulates.

And (5) quantifying and calibrating the basic attribute features to form a basic attribute portrait set.

Through the previous steps, a core portrayal set and a base portrayal set are constructed for each patient. Next, a role representation needs to be built based on the core representation set and the base representation set.

The core portrait and the attribute portrait are in parallel relation, the core portrait is the core feature of an individual, the attribute portrait is the basic feature of the individual, the character portrait adopts a one-dimensional structure by the basic attribute portrait, the core portrait adopts a two-dimensional structure, the features of the core portrait at different time points (different disease development periods) are three-dimensional structures along with the progress of time, and in the step, the portrait sets are added with weight differences to be overlapped (union set) to form the character portrait of the individual.

And merging the portrait sets constructed based on different features according to a preset rule. For example, the features in the two image sets are combined by giving different weights according to the importance of the different features and then performing weighted overlap. Here, the weight of the feature in the core representation is greater than the weight of the feature in the base representation. Specifically, factors to be considered in weight distribution include feature coverage of each image set, consistency of data, rationality of weight distribution and the like, so that the superimposed images can accurately reflect comprehensive features of different patient groups, and balance cannot be lost due to over-emphasis of certain features.

And combining and superposing the basic image set as the coarse particle characteristic of the individual and the core image set as the fine particle characteristic of the individual. By combining features of different granularities, feature information of a user can be more comprehensively captured.

Coarse grain features refer to relatively broad, generalized features that provide general information about an individual or object. These features include basic attributes such as age, gender, occupation, etc., which are helpful in understanding the basic profile of an individual or patient. The feature is mainly applied to the scene of rapidly positioning the overall profile of the individual and is used as a basic data classification feature label. These features aid in the initial screening and classification and provide directions for subsequent in-depth analysis.

Fine particle characteristics are more specific, detailed characteristics that provide more advanced information about an individual or object. These features include a number of specific behaviors, interests, habits, etc., which are important for understanding the uniqueness and individuality needs of an individual or object. The fine particle feature is suitable for fields requiring deep knowledge of individual or object uniqueness, such as personalized recommendation systems, precision training, and behavioral feedback analysis. These features can capture subtle differences, providing more accurate computing and push support.

Thus, coarse particle features provide general information to the user, while fine particle features provide more specific details. The feature representation of the combination of the thickness can improve the accuracy of image mapping, the fine granularity feature selection module can extract key local and fine granularity identification features in the object, the coarse granularity feature selection module can acquire coarse granularity diversity features for providing context information and improve the discrimination of a model, and the model can be better generalized to different scenes and conditions by combining features with different granularity.

Specifically, a basic image set (coarse grain feature) and a core image set (fine grain feature) are combined and superimposed. Different weights need to be assigned to different features in this process. When different weights are given according to the importance of different features, the method satisfies 1) the feature weight in the core image is larger than the feature weight in the basic image, 2) the feature coverage, the data consistency, the weight distribution rationality and the like of each feature in the core image, and 3) the feature coverage, the data consistency, the weight distribution rationality and the like of each feature in the basic image.

And quantifying and calibrating the core attribute features one by one to form a core attribute portrait set.

The character portrayal is a three-dimensional mapping system integrating basic portrayal features and core portrayal features. The attributes of the core portraits include at least cognitive cortex injury, physical function injury and emotion control force injury. When a character representation is obtained, it is necessary to construct a three-dimensional time slice based on the base representation and the core representation, the three-dimensional time slice being obtained by:

First, the phase division is performed. The course of the disease is divided into several stages, including early, medium and late stages, according to the progression of the different types of disabled diseases in the elderly. Each major phase may be further subdivided into more specific sub-phases in order to more carefully understand the progression of the disease process.

And then extracting the characteristics. In each stage, the critical features of the patient are extracted, including cognitive function, emotional state, motor ability, and the like. These features can be measured by standardized assessment tools such as Mini Mental State Examination (MMSE), montreal cognitive assessment (MoCA), berg balance scale, and the like.

And finally, constructing a time axis. The time axis from the time of the discovery of the condition (e.g., every month, every 2 months, every year, etc.) is used to represent the progression of the disease over time. On the time axis (as the time dimension), the start and end points of each phase, as well as the key features of each phase, can be noted. The time axis may be linear or non-linear to reflect the complexity and uncertainty of the course of the disease. Such a time axis helps to more intuitively understand the progress of the disease process and provides guidance for treatment and intervention.

And quantizing the time sequence dimension characteristics in the character portraits according to industry consensus, and carrying out portraits fusion. The knowledge spectrum information such as clinician experience, industry diagnosis standard, expert consensus in the field and the like is brought into, the disease development stage is used as a time dimension dividing node, and clinical characteristics after quantification of different disease development stages are associated with other dimension information in the character portrait. Taking AD-derived dementia as an example, according to industry consensus, the disease progression can be classified into early (asymptomatic brain amyloidosis stage: SCD, MCI stage), mid-stage (amyloid positive + synaptic dysfunction and/or neurodegenerative stage), late (amyloid positive + evidence of neurodegeneration + severe cognitive decline). Different conditions develop and the corresponding exercise training schemes are different. The content of the exercise training regimen is shown in fig. 2 and includes cognitive function rehabilitation, exercise function rehabilitation, mental behavior symptom rehabilitation, activity and participation rehabilitation, comprehensive rehabilitation, and the like.

The clinical characteristics after different disease development periods and quantization are associated with other dimensional information in the character portraits, and the clinical characteristics are used as multidimensional portrait characteristic input of a pushed exercise training scheme to prepare for pushing of accurate exercise training.

And step two, generating and pushing a first exercise training scheme based on the character portrait.

A first athletic training regimen is automatically generated based on the character representation. The first exercise training regimen may be generated based on a pre-trained model (e.g., collaborative filtering model as described below) or may be designed based on the experience of the physician.

And thirdly, collecting structured data, text data and image data during exercise training.

Structured data, text-like data, image-like data of the patient's movements according to a first exercise training scheme are collected. And according to the structured data, the text data and the image data (image data or image data), the self-variable data of the collaborative filtering model is input. Specifically, including but not limited to:

■ Structured data, such as exercise diagnosis, living ability class measurement and evaluation values, demographic characteristic data and the like;

Text data, such as clinical diagnosis of recent 2 years, case report data and the like (including exercise diagnosis data of a patient obtained through cases, laboratory sheets and the like, such as upper limb exercise function injury);

■ Image data such as nuclear magnetic resonance data and CT image data;

and step four, inputting the data in the step three into a collaborative filtering model, and combining the character portraits to generate and push a next exercise training scheme.

In this embodiment, step four may be repeated until the desired effect is achieved or the patient ceases training.

In one aspect, multiple automatic encoders and collaborative filtering networks are employed in a collaborative filtering model, and graph embedding techniques are employed to map entities (e.g., patients, motion types, physiological indicators, etc.) and relationships in a graph structure into a low-dimensional vector space. The step is realized through a Bayesian TransR embedding model, a Bayesian stack noise reduction self-encoder (SDAE), a Bayesian stack convolution self-encoder (SCAE) and other deep learning embedding technologies, and the structured data, the text data and the image data are respectively converted into a structure vector, a text vector and an image vector.

In particular, an automatic encoder is a layer-by-layer unsupervised learning model, mainly comprising 2 processes of decoding and encoding for processing high-dimensional data as a tool for feature extraction, enabling collaborative filtering models to learn low-dimensional representations (hidden vectors) of users and items. These hidden vectors can capture the inherent links between the user and the item, thereby improving the performance of the recommendation system. Here, the automatic encoder fuses 3 types of information (structured data, text data, image data) to realize fusion analysis of multi-modal information and obtain project hidden vectors. Furthermore, based on the character portrayal data, the automatic encoder uses feature extraction to obtain a user hidden vector.

On the other hand, to fully learn the auxiliary information features, the collaborative filtering model learns the vector representation of the text information by using a Bayesian stack noise reduction self-encoder (SDAE), the vector representation of the image information by using a Bayesian stack convolution self-encoder (SCAE), and the vector representation of the embedded model learning structure information by using Bayesian TransR, respectively. SDAE is a deep learning model that extracts high-level features of data through multi-level unsupervised learning, and can capture semantic and structural information in text, which is particularly useful for dealing with noise and uncertainty of text data. SCAE are used to learn the vector representation of the image information because Convolutional Neural Networks (CNNs) have advantages in processing the image data, enabling capture of spatial hierarchies and local features of the image. SCAE by stacking multiple convolution and pooling layers, more abstract and advanced image features can be learned, helping models make more robust predictions in the face of the diversity and complexity of image data.

TransR is an embedded model specially designed for a knowledge graph, and can effectively process complex relations in the knowledge graph. The entity and the relation are respectively mapped to different semantic spaces, and the entity is projected from the entity space to the relation space by utilizing a relation-specific conversion matrix, so that vector operation is carried out in the projected space, and more accurate knowledge graph completion is realized. The design not only can capture the multidimensional relation among the entities, but also can effectively process incomplete information and noise in the knowledge graph. In addition, the flexibility and the expandability of TransR enable the system to adapt to different structured data, and accuracy and robustness in processing the structured data are improved. By converting text, images, and structured data into a unified vector representation TransR ensures that data from different sources is comparable in feature space, thereby providing powerful support for efficient representation and application of knowledge maps.

After feature extraction, the graph embedding technology is beneficial to constructing an individual data space, and multi-dimensional data such as time, motion preference, physiological and psychological characteristics, state after training and the like are integrated together to form a four-dimensional data structure. Further, it integrates the structure vector, text vector and image vector to form the project hidden vector. The project hidden vector integrates the characteristics of various data types and can more comprehensively represent the characteristics of the project.

Specifically, as shown in fig. 3A, the next exercise training scheme is generated and pushed by using the collaborative filtering model, and the method comprises the following sub-steps:

1) Determining key variables as one-dimensional data, two-dimensional data, and three-dimensional data

First, it is necessary to determine which variables are important for assessing and predicting the patient's rehabilitation process. By combining cognitive tasks (e.g., memory, attention tasks) and motor tasks (e.g., walking, balance training), the motor function and balance ability of the patient can be improved. Through the movement, the brain blood flow and blood oxygen level of the patient are improved, neurons in a dormant state are activated, the growth and connection of the neurons are promoted, a specific nerve conduction mode is formed, neurotransmitter level is regulated, and the improvement of the cognitive function is helpful for improving the motor skills and efficiency of the patient, enhancing the motivation and self-control ability (the control ability on emotion, speech and the like) and optimizing the movement strategy.

In summary, different critical independent variables need to be determined for patients with different diseases (different damaged brain areas).

Key independent variables that are important in common for all cognitive patients include:

Time (T)

Sports Type (Type of exact)

Exercise Intensity (Intensity)

Duration (Duration)

Frequency (Frequency)

Patient Status (Patient Status)

Physiological indexes (Physiological Indicators, such as heart rate, blood pressure, etc.)

Psychological indicators (Psychological Indicators, such as anxiety level, depression level, etc.)

Rehabilitation progress (Recovery Progress)

Determining different key variables for patients with different diseases, for example, includes:

AD patient:

the brain injury nuclear magnetic ROI index (Region of Interest) is hippocampus, medial temporal lobe, parietal lobe and forehead She Shousun;

Cognitive ability scores (Cognitive Ability Scores) are hypomnesis, spatial disorientation, inattention, and executive dysfunction, among others.

PD patient:

the brain injury nuclear magnetism ROI index (Region of Interest) is that substantia nigra, striatum, forehead leaf cortex and top leaf are damaged;

an Activity of DAILY LIVING Indicators of movement disorder, cognitive decline, paresthesia, etc.;

Hormone level score (Hormone Level Score) dopamine level, etc.

2) Creating a data matrix from key variables

A data matrix is created based on the key variables for each patient, wherein each row represents a data record for a point in time and each column represents a variable. For example:

one-dimensional data [ time (T) ]

Two-dimensional data (sports preference) [ sports Type (Type of Exercase) ]

Exercise Intensity (Intensity)

Duration (Duration)

Frequency of motion (Frequency)

Three-dimensional data (physiological and psychological characteristics) [ heart rate (HEART RATE) ]

Blood Pressure (Blood Pressure)

Anxiety Level (measure Level)

Depression level (Depression Level)

3) Adding hysteresis features as four-dimensional data

To capture dependencies in the time series, hysteresis features may be added.

Four-dimensional [ Patient Status after training (Patient Status) rehabilitation progress (Recovery Progress) ], for example:

Time of	Patient status after training	Rehabilitation progress
			T1 (e.g. 2024 10 month 3 day 10 am)	Good quality	5%
T2 (e.g. 2024 10 month 3 day 3 pm)	Good quality	5%
			T3 (e.g. 2024 10 month 4 day 10 am)	Good quality	5%
T4 (e.g. 2024 10 month 4 day 3 pm)	Improvements in or relating to	10%
			...	...	...

And forming a four-dimensional data structure based on the one-dimensional data, the two-dimensional data, the three-dimensional data and the four-dimensional data. Namely, a volume data space is constructed by taking one-dimensional data (time) as a W axis, taking two-dimensional data as a Y axis, taking three-dimensional data as an X axis, and taking four-dimensional data as a Z axis.

4) And converting the feature vectors into feature vectors by using an embedding technology.

And (5) carrying out internal processing and fusion by utilizing a hidden layer of the deep learning model, and converting the four-dimensional data into characteristic vector representation by an embedding technology.

As shown in fig. 3A to 3C, the aforementioned structured data in the individual data space of each patient is converted into a structural vector by structured embedding. For example, using the bayesian TransR knowledge graph embedding method, entities and relationships can be converted into vectors that can capture complex relationships between entities.

By text embedding, converting text-like data into text vectors, low-dimensional representations of the text data may be learned, for example, using a Bayesian stack noise reduction self-encoder (SDAE), capturing semantic information of the text.

By image embedding, image class data is converted into image vectors, for example, features of the image can be extracted and converted into vectors using a Bayesian stack convolution self-encoder (SCAE).

The structure vector, text vector, image vector are integrated into project hidden vector using the methods provided by the aforementioned TransR, SDAE and SCAE. The project hidden vector is a vector integrating the characteristics of various data types, can more comprehensively represent the characteristics of different types of patients, and can represent the project characteristics.

5) Based on the feature vectors, project hidden vectors and user hidden vectors are generated for collaborative filtering learning to recommend a athletic training regimen.

And merging feature vectors from different modal data by using an attribute fusion graph roll network AF-GCN, weighting the importance of the different modal data by using an attention mechanism, and carrying out layer-by-layer hidden vectorization on the feature vectors.

Therefore, the embodiment fuses data (such as texts and images) from different sources, takes one-dimensional data as a W axis, takes two-dimensional data as a Y axis, takes three-dimensional data as an X axis and takes four-dimensional data as a Z axis to construct a personal data space, wherein the one-dimensional data comprises time variables, the two-dimensional data comprises a plurality of variables describing movement preference, the three-dimensional data comprises variables describing physical and psychological characteristics, and the four-dimensional data comprises trained state data. Moreover, features of each data type are extracted by using a deep learning model, so that the features are combined with knowledge representation to form a unified feature space, thereby better understanding the interests and requirements of patients and realizing multi-modal information feature level fusion (fusion performed after feature extraction and before decision-making). Moreover, the information in the character portraits of the patient (user) (such as the behavior in the portraits, the preference, the scoring of different training schemes, and the like) is converted into the user hidden vector by using a deep learning model (such as the SDAE), and the user hidden vector is a vector capable of representing the characteristics of the user, so that the user preference of different projects is calculated by inputting the user hidden vector and the user preference of different projects into a collaborative filtering network, and a final recommended scheme (a motion training scheme) is generated according to the user preference.

The present embodiment employs collaborative filtering models that combine graph structures and deep learning techniques, which are used to represent a knowledge base including entities (e.g., sports categories, user attributes, user behavior) and their relationships, so that the graph structures can be used to capture interaction data between users and items, learning user preferences and item features.

The multi-modal feature (project hidden vector) and the character image feature (user hidden vector) extracted for the patient condition are used as self-variable data, and the first exercise training scheme (for example, an exercise prescription clinically prescribed by a doctor) in the previous step is used as a dependent variable and is input into the collaborative filtering model to generate the next exercise training scheme. The loss function of the collaborative filtering model is cosine contrast loss (Cos ine Contrast ive Loss) for maximizing similarity between positive pairs of samples while minimizing similarity for negative pairs of samples under the margin constraint.

As previously described, each entity and relationship in the graph structure of the individual data space is mapped into a low-dimensional vector space by graph embedding techniques, which allows them to be computed and compared to enable collaborative filtering models to analyze interaction data between users and items, capture complex nonlinear relationships between patients and items, learn user preferences and item characteristics, and thereby predict items that may be of interest to a user.

Collaborative filtering models present themselves with cold start problems. In other words, for a new patient, due to the lack of historical behavioral data, the similarity between patients cannot be calculated according to a conventional collaborative filtering model, and thus personalized recommendation is difficult. This results in the difficulty of providing accurate personalized recommendations to the user when the collaborative filtering model is started at the time of entrance to the hospital. However, for structured data, image data, text data, etc., collaborative filtering models can predict the preferences of new users or new projects by analyzing the behavior of other similar patients or similar projects, and can implement recommendations (cold start function) for rapid exercise training programs for similar feature populations using only a small amount of exercise training program data with clinical authoritative diagnosis, without requiring a large amount of training. This is because in the collaborative filtering model provided by the embodiment of the invention, similar patients and similar items can be easily found based on the character portrayal data and the multi-modal data of the new patient.

Namely, the collaborative filtering model provided by the embodiment of the invention plays the role portrait and multi-mode data fusion advantage. For example, in patients with cognitive cortical impairment, nuclear magnetic resonance images of the patient show abnormal signals of specific brain regions, and similar cognitive dysfunction manifestations are also mentioned in clinical diagnosis. In the multi-mode data corresponding to the cognitive cortex injury and the physical function injury, the feature similarity is higher, and the commonality and regularity of the data of different patients on key features can be obtained by using the model, so that the feature similarity judgment accuracy is improved. As another example, two patients, while differing in mood control stress injury, are more likely to be classified as similar patients when the exercise training regimen is recommended if they are highly similar in cognitive cortical injury and physical function injury, thereby recommending a suitable exercise training regimen for the new patient. By using the core portrait with fine granularity, the model can accurately grasp the key requirements of new patients and find similar patients and projects.

According to the rich information in the character portraits, matching and screening are carried out from multiple layers, a patient group similar to a new patient in multiple dimensions is found, and then a sports training scheme which is more in line with the comprehensive characteristics of the new patient is recommended for the new patient, so that the recommended individuation degree is improved. For another example, for patients with severe cognitive cortex injury, more emphasis is placed on cognitive training and motor training protocols for brain function activation, while patients with prominent physical function injury are focused on training for rehabilitation and enhancement of limb motor function. Compared with the traditional collaborative filtering model only relying on historical behavior data, the personalized recommendation mode can better solve the cold start problem, provides accurate and effective exercise training scheme recommendation for new patients, and helps the new patients to improve the ability in the aspects of cognition, emotion, exercise, speech and the like.

In addition, on the basis of basic portraits such as medical history, demographic characteristics and core portraits such as examination results and doctor suggestions, the generated patient state sequence data and different mode data such as imaging data (image data), scale data (text data) and the like are fused, and the correlation between different media modes can be more fully mastered through multi-layer attribute fusion by using a collaborative filtering model, so that the accuracy and generalization capability of the exercise training scheme recommendation are improved.

Second embodiment

Based on the first embodiment, in the present embodiment, the recurrent neural network model (i.e. RRN model) is combined with the collaborative filtering model in the first embodiment, so as to further improve the consistency and sustainability of the exercise training scheme recommendation and promote the patient compliance.

As shown in fig. 4, the exercise training recommendation method based on multi-modal data fusion according to the second embodiment of the present invention at least includes the following steps:

steps one to four, which are the same as the first embodiment (but not repeated many times), are not repeated here.

And fifthly, collecting structured data, text data and image data in the exercise training period for multiple times in the mode of the step three to obtain exercise training period data.

The patient moves for a plurality of times according to the exercise training scheme, and data (structured data, text data and image data) in the plurality of times of movement are collected in a cumulative way according to the mode of the third step, so that exercise training period data can be obtained.

Training period data in a sports training scheme is generally divided into multi-year period data, large period data, medium period data and small period data.

Years of period, namely, the specific analysis of specific problems is needed due to the large individual difference of the old. As in (1), the physical condition and recovery ability are different for each elderly person, so that the exercise rehabilitation cycle may vary from person to person. Generally, the elderly with better physical condition and less disability may recover faster, and may take a period of 2 years. (2) The more severe the disability, the longer the time required for rehabilitation is generally. The elderly who are completely bedridden take 8 years as a period and the elderly who can be partially self-care take 4 years as a period. In the rehabilitation process, the physical condition and rehabilitation progress of the old need to be continuously monitored, and adjustment is performed according to the needs. This helps to ensure the effectiveness of the rehabilitation program and shortens the rehabilitation cycle as much as possible.

The large period is generally defined by taking the current year or one year as a time limit, so that a half-year or full-year training plan is prepared. There are also three large cycles of the year, which are generally associated with sports classification.

The medium period is called stage and month training in training practice, and is generally 4-8 weeks, so that a stage or month training plan is prepared.

Small cycles are known in training practice as weekly training, usually with calendar weeks as a term, to formulate a weekly training plan. A small period of 4 days to 10 days can also be arranged. If a small recovery adjustment is made, the adjustment task can be completed in 4 days, and a recovery period of 4 days can be scheduled.

Step six, judging whether the exercise training scheme is the Nth one, if so, entering a step seven, otherwise, returning to a step four, and generating the next exercise training scheme according to the characteristics in the step five.

N is a preset number of times, for example 3 times. Through the step, the dynamic characteristics of the patient when training is respectively carried out according to a plurality of exercise training schemes can be obtained, and the dynamic characteristics are the characteristics with time sequence dimension. The design is beneficial to the RRN model learning of more complex dynamic evolution characterization.

And step seven, based on the data acquired by the patient in the step three and the step five, jointly inputting an RRN model to iteratively optimize the exercise training scheme.

In this step, more complex dynamic evolution characterization can be better learned using the RRN model as shown in fig. 5. In the RRN model, historical interaction information between a patient and motion is key data for driving the preference of the patient and the motion state to change, so that an evolution hidden representation of the patient and the motion can be captured by using a common evolution model. Here, the training period vector input to the RRN model is utilized, and the RRN model learns the dynamic feature representation of the patient and the motion using the recurrent neural network, resulting in a dynamic evolution representation with a time-series relationship. Training period vectors refer to vectors that are obtained by converting various attributes (e.g., time, frequency, intensity, etc.) in training period data into numeric features and then converting the numeric features by embedding or encoding techniques.

As shown in fig. 5, the RRN model uses 2 long-short term memory networks (LSTM) on the basis of a conventional Recurrent Neural Network (RNN) to learn the temporal variation of patient exercise preference, long-term (e.g., seasonal) evolution of exercise, respectively. In fig. 5, y _i, which represents the dynamic feature vector of patient i over the time series (time t, t+1, etc.), is learned by one LSTM network, and y _j, which represents the dynamic feature vector of motion j over the time series, is learned by another LSTM network. Furthermore, both y _i and y _j are affected by the patient's static characteristics u _i and the static characteristics of motion m _j. This is to allow the RRN model to learn both the static hidden representation of the patient and the static hidden representation of the motion, taking into account the patient's long-term motion preferences as well as the static properties of the motion.

In other words, the RRN model updates the dynamic feature vector by combining the dynamic feature vector of the previous time step of the patient and the current static feature vector with one LSTM network at each time step, and also updates the dynamic feature vector by combining the dynamic feature vector of the previous time step of the motion and the current static feature vector with another LSTM network. Thus, the RRN model combining static and dynamic characteristics can simultaneously consider long-term exercise preference and short-term variation of a patient, as well as long-term characteristics and short-term variation of an exercise training scheme.

Specifically, the RRN model employs LSTM-based recurrent neural networks to model dynamic changes in patient and motor training protocols. For patient i and motion j (in one motion training scenario, multiple motions are included), let u _i denote the static feature vector of patient i and m _j denote the static feature vector of motion j. At time t, let u _it be the dynamic feature vector of patient i (i.e., y _i,t),m_jt in the figure is the dynamic feature vector of motion j (i.e., y _j,t in the figure). Dynamic features u _i,t+1 and m _j,t+1 at time t+1 can be solved in series by one LSTM network, respectively, as follows:

u_i,t+1＝g(u_it,{r_ij|t})

m_j,t+1＝h(m_jt,{r_ij|t})

wherein g and h are functions to be learned, g represents a function used for updating the dynamic characteristics of the patient in the LSTM network, h represents a function used for updating the dynamic characteristics of the exercise training scheme in the LSTM network, and r _ij|t represents the score of the patient i on the exercise j at time. the dynamic characteristics u _i,t+1 of the patient at time t+1 are calculated from the dynamic characteristics u _i, t of the patient at time t and the response r _ij|t by the LSTM network. the dynamic characteristic m _j,t+1 of the motion at the time t+1 is calculated by the LSTM network according to the dynamic characteristic m _j,t of the motion at the time t and the response r _ij|t. This shows that the dynamic characteristics of the patient and the athletic training regimen are updated via the LSTM network based on their state and current response at a previous point in time. Thus, the RRN model is able to capture dynamic characteristics over time, thereby better simulating and predicting the patient's response to different exercise training protocols.

By inputting the dynamic and static characteristics (u _i,t+1,m_j,t+1,u_it,m_jt) of the patient i, the model can predict the current interest (score) of the patient, and the predicted value of r _ij Where f is also a function that needs to be learned.

The loss function of the RRN model is the sum of the squared error loss function and the regularization term. The parameter theta is adjusted to minimize the loss function, so that the overall optimization of the RRN model can be realized. That is, the optimization is performed using the following function:

It can be seen that the objective of RRN model optimization is to make the model push the exercise training scheme more and more conform to the current exercise preference, exercise level and exercise state of the patient, i.e. the prediction is made to be close to the actual through the parameters generated by training. Where θ represents a parameter to be learned, R represents a regularization function for a set of observed (patient, exercise training regimen, time series) tuples in the training set.

While the objective functions and building blocks in the RRN model are very standard, simple back propagation applications cannot easily solve this optimization problem. The key challenge is that the score of each patient is dependent on the patient's state and the exercise training regimen. Back propagation through 2 sequences is computationally prohibitive. This problem is alleviated by back-propagating gradients from the patient's motion feedback, but each score is still dependent on the patient's state, which in turn acts on the complete sequence pushed by the motion training protocol.

Thus, an alternate subspace descent strategy (Subspace Descent) is employed in an embodiment of the present invention. That is, assuming that the motion state is fixed, there is no need to propagate gradients into these motion training program sequences, while back propagating gradients of all scores of the patient to update the patient sequence parameters, and then switch between updating the user sequence and updating the motion training program sequence. In this way, standard feed forward and back propagation can only be performed once for each patient's exercise training protocol push, ultimately achieving model optimization.

In summary, embodiments of the present invention use a hybrid recommendation model (collaborative filtering model and RRN model) as the core tool for athletic training recommendations. Compared with the traditional content recommendation model, the collaborative filtering model in the mixed recommendation model can learn the representation of multi-source data (structured data, text data, image data and training period data) at the same time, and utilizes the structural information, the content information and the image information multidimensional information. Meanwhile, the recurrent neural network in the mixed recommendation model is used for capturing long-term preference evolution characterization and short-term preference characterization of the patient. Therefore, by fusing different models, the effective evaluation of the movement dose, movement state and movement mode of the individual of the aged disabled patient can be realized, and finally the pushing of the optimal movement training scheme is realized.

At each training of the patient, the new exercise training regimen is output in a seventh manner using the data acquired during the previous exercise training. This is repeated until a predetermined goal is reached, such as ability to recover or remain unchanged in terms of cognition, emotion, motion, speech, etc.

As an optional step, a step eight can be added after the step seven, wherein the training effect is estimated according to the data in the previous exercise training scheme, an exercise training report is produced and output or directly ended if the effect is improved, and the exercise training scheme is regenerated by returning to the step seven if the effect is not improved. The method for evaluating the training effect can adopt a conventional evaluation method, including the evaluation by doctors, and can also compare and evaluate the data in the last exercise training scheme with the data in the previous exercise training scheme.

Compared with the prior art, the invention has the following technical advantages:

1) Constructing a multi-stage patient portrayal system, gradually and deeply deconstructing and evaluating attribute features, behavior features and comprehensive features of a patient, ensuring that an individual tag system has fine granularity, and providing a large amount of effective information of a patient sequence for accurately recommending motion of a subsequent input model;

2) A motion recommendation framework that fuses the algorithmic benefits of multiple models. On the one hand, the collaborative filtering and the fusion analysis system of the multi-mode data in the self-coding learning are used, so that the multi-source information is ensured to enter a recommendation model, and the analysis source is increased;

3) Constructing a training period data system, dividing the period of a motion training process into a year period, a big period, a middle period and a small period, manufacturing different time sequence data sets, adding time sequence dimension characteristics when a model is introduced to perform motion recommendation calculation, thereby more fully and regularly applying historical motion data and assisting in the generation of a periodic scheme of motion recommendation;

4) The deep learning-based motion pushing system is constructed, multidimensional motion training pushing logic is constructed as a system flow, and continuously iterated motion pushing content based on multidimensional motion data of a patient can be realized, so that training can be most consistent with the individual and current state of the patient, and the training effect is improved.

Third embodiment

On the basis of the exercise training recommendation method based on the multi-mode data fusion, the third embodiment of the invention further provides an exercise training recommendation system based on the multi-mode data fusion. As shown in fig. 6, the athletic training recommendation system includes a processor and a memory. Wherein the memory is coupled to the processor for storing one or more programs that, when executed by the processor, cause the processor to implement the exercise training recommendation method based on multimodal data fusion as in the above embodiments.

The processor is used for controlling the overall operation of the exercise training recommendation system so as to complete all or part of the steps of the exercise training recommendation method based on the multi-mode data fusion. The processor may be a Central Processing Unit (CPU), a Graphics Processor (GPU), a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), a Digital Signal Processing (DSP) chip, or the like. The memory is used to store various types of data to support operation at the athletic training recommendation system, which may include, for example, instructions for any application or method operating on the athletic training recommendation system, as well as application-related data. The memory may be implemented by any type of volatile or non-volatile memory device or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, etc.

In another exemplary embodiment, the invention also provides a computer readable storage medium comprising program instructions which, when executed by a processor, implement the steps of the exercise training recommendation method based on multimodal data fusion of any of the embodiments described above. For example, the computer readable storage medium may be a memory including program instructions as described above, which are executable by a processor of the system to perform the exercise training recommendation method based on multimodal data fusion as described above, and achieve technical effects consistent with the method described above.

It should be noted that the above embodiments are only examples. The technical schemes of the embodiments can be combined, the sequence of the steps can be changed, and the method is within the protection scope of the invention.

The exercise training recommendation method and system based on multi-mode data fusion provided by the invention are described in detail. Any obvious modifications to the present invention, without departing from the spirit thereof, would constitute an infringement of the patent rights of the invention and would take on corresponding legal liabilities.

Claims (10)

1. A sports training recommendation method based on multi-mode data fusion is characterized by comprising the following steps:

step one, aiming at the elderly disabled patient, obtaining the disease type and clinical characteristics, demographic information, exercise investigation information and exercise behavior characteristics of the patient so as to fuse and construct character portraits and obtain static characteristics;

step two, generating and pushing a first exercise training scheme based on the character portrait;

Step three, collecting structured data, text data and image data during exercise training;

and step four, inputting the data in the step three into a collaborative filtering model, combining character images to generate and push a next motion training scheme, wherein the collaborative filtering model adopts a plurality of automatic encoders and collaborative filtering networks, adopts a graph embedding technology to map entities and relations in a graph structure into a low-dimensional vector space, and is realized through a Bayesian TransR embedding model, a Bayesian stack noise reduction self encoder and a Bayesian stack convolution self encoder so as to respectively convert structured data, text data and image data into structural vectors, text vectors and image vectors, thereby obtaining project hidden vectors and user hidden vectors for collaborative filtering learning to recommend the next motion training scheme.

2. The exercise training recommendation method based on multi-modal data fusion as claimed in claim 1, wherein said step four comprises the sub-steps of:

1) Determining key variables as one-dimensional data, two-dimensional data and three-dimensional data;

2) Creating a data matrix according to the key variables;

3) Adding hysteresis characteristics as four-dimensional data;

4) Converting the feature vector into a feature vector by using an embedding technology;

5) Based on the feature vectors, project hidden vectors and user hidden vectors are generated for collaborative filtering learning to recommend a athletic training regimen.

3. The exercise training recommendation method based on multi-modal data fusion of claim 2, wherein:

In the sub-step 4, the entity and the relation are converted into vectors by using a Bayesian TransR knowledge graph embedding method, text data are converted into text vectors by using a Bayesian sparse self-encoder embedding method, and features of the image are extracted by using a Bayesian sparse convolution self-encoder and are converted into image vectors.

4. The exercise training recommendation method based on multi-modal data fusion of claim 3, wherein:

The multi-modal features and the character portrayal features extracted aiming at the condition of the patient are taken as self-variable data, and the clinically-issued sports prescription suggestions are taken as dependent variables and input into the collaborative filtering model;

the loss function of the collaborative filtering model is cosine contrast loss, which is used for maximizing the similarity between positive sample pairs and minimizing the similarity of negative sample pairs under the edge margin constraint.

5. The multi-modal data fusion based athletic training recommendation method of claim 4, wherein:

In the substep 1, a one-dimensional data is taken as a W axis, a two-dimensional data is taken as a Y axis, a three-dimensional data is taken as an X axis, and a four-dimensional data is taken as a Z axis, so that a personal data space is constructed.

6. The multi-modal data fusion based athletic training recommendation method of claim 5, wherein:

The one-dimensional data includes time variables, the two-dimensional data includes a plurality of variables describing movement preferences, the three-dimensional data includes variables describing physical and psychological characteristics, and the four-dimensional data includes post-training state data.

7. The multi-modal data fusion based athletic training recommendation method of claim 6, wherein:

when a data matrix is created according to the key variables, different diseases correspond to different key variables, and the character portraits comprise a plurality of attributes including cognitive cortex injury, physical function injury and emotion control force injury attributes.

8. The method for training recommendation based on multimodal data fusion as claimed in claim 6, further comprising the steps of:

Step five, structured data, text data and image data in the exercise training period are collected for a plurality of times in the mode of step three, so that exercise training period data are obtained;

Step six, judging whether the exercise training scheme is the Nth one, if so, entering a step seven, otherwise, returning to a step four, and generating a next exercise training scheme according to the characteristics in the step five, wherein N is a positive integer;

Step seven, based on the data acquired by the patient in the step three and the step five, the RRN model is input together to iteratively optimize the exercise training scheme,

Wherein the RRN model is optimized using the following function:

wherein θ represents a parameter to be learned; For a set of tuples observed in the training set, the tuples include patient, exercise training regimen and time series, rij|t represents the score of patient i to exercise j at time t; Is the predicted value of rij|t, and R represents a regularization function.

9. The multi-modal data fusion based athletic training recommendation method of claim 8, wherein:

The RRN model employs an alternating subspace descent strategy, assuming that the motion state is fixed, without propagating gradients into these motion training program sequences, while counter propagating gradients of all scores of the patient to update patient sequence parameters, and then switches between updating the user sequence and updating the motion training program sequence.

10. A multi-modal data fusion based athletic training recommendation system, comprising a processor and a memory, wherein the memory is coupled to the processor for storing one or more programs that, when executed by the processor, cause the processor to implement the multi-modal data fusion based athletic training recommendation method of any of claims 1-9.