WO2024247080A1 - Training device, evaluation device, training method, evaluation method, and program - Google Patents

Abstract

One aspect of the present invention is a training device comprising a control unit that trains a training target by using: action video data that is video data of a video showing an action of an evaluation target, which is a person or animal whose action is to be evaluated; action measurement data indicating time-series results, which are obtained during an action shown in the action video data, of an action measurement sensor that is a sensor attached to at least one of the body of the evaluation target, an item worn by the evaluation target, and an item used by the evaluation target during the action, and obtains a result according to the action of the evaluation target; and correct data indicating a result of evaluating the action. The training target is a mathematical model that evaluates the action on the basis of the action video data and the action measurement data.

Description

Learning device, evaluation device, learning method, evaluation method, and program

The present invention relates to a learning device, an evaluation device, a learning method, an evaluation method, and a program.

Interest is growing in technology that allows computers to evaluate the movements of people being evaluated based on video footage, such as in judging the movements of athletes in competitions such as figure skating, surfing, and diving, and in judging the movements of medical interns during medical procedures such as surgery.

Paritosh Parmar and Brendan Tran Morri, “What and How Well You Performed? A Multitask Learning Approach to Action Quality Assess”, arXiv:1904.04346v2, (2019)

However, the environments in which sports and medical procedures are performed are not necessarily the same. For example, the lighting may be different for an indoor movement, and the weather may be different for an outdoor movement. The footage shows not only the person being evaluated, but also these environmental differences. As a result, even if the movements of the person being evaluated are the same, the computer evaluation results may differ. In other words, the accuracy of the computer evaluation may be poor. This situation is not limited to cases where the person being evaluated is a human, but is the same even when the person being evaluated is an animal, such as a dog or cat.

In view of the above, the present invention aims to provide a technology that improves the accuracy of motion evaluation.

One aspect of the present invention is a learning device that includes a control unit that learns a learning subject using motion video data, which is video data of a video showing the motion of an evaluation subject, which is a human or animal whose motion is to be evaluated, motion measurement data, which is a time series of results obtained during the motion shown in the motion video data by a motion measurement sensor that is a sensor attached to at least one of the body of the evaluation subject, something worn by the evaluation subject, and something used by the evaluation subject when performing the motion, and that obtains results according to the motion of the evaluation subject, during the motion shown in the motion video data, and ground truth data that indicates the result of the evaluation of the motion, and the learning subject is a mathematical model that evaluates the motion based on the motion video data and the motion measurement data.

One aspect of the present invention is an evaluation device that includes an interface unit that acquires a set of motion video data showing the motion of an evaluation target and motion measurement data obtained during the motion shown in the motion video data, a control unit that performs learning of a learning target using motion video data that is video data showing the motion of an evaluation target that is a human or animal whose motion is to be evaluated, motion measurement data that is a time series of results obtained during the motion shown in the motion video data by a motion measurement sensor that is a sensor attached to at least one of the body of the evaluation target, something worn by the evaluation target, and something used by the evaluation target when performing the motion and that obtains a result according to the motion of the evaluation target, and ground truth data that indicates the result of the evaluation of the motion, and the learning target is a mathematical model that evaluates the motion based on the motion video data and the motion measurement data, and a learned evaluation model execution unit that uses the learned mathematical model obtained by the learning device and the set acquired by the interface unit to evaluate the motion shown in the motion video data included in the set.

One aspect of the present invention is a learning method that includes a control step of learning a learning subject using motion video data, which is video data of a video showing the motion of an evaluation subject, which is a human or animal whose motion is to be evaluated, motion measurement data, which is a time series of results obtained during the motion shown in the motion video data by a motion measurement sensor, which is a sensor attached to at least one of the body of the evaluation subject, something worn by the evaluation subject, and something used by the evaluation subject when performing the motion, and which obtains a result according to the motion of the evaluation subject, during the motion shown in the motion video data, and ground truth data indicating the result of the evaluation of the motion, and the learning subject is a mathematical model that evaluates the motion based on the motion video data and the motion measurement data.

One aspect of the present invention is an evaluation method including an interface step of acquiring a set of motion video data showing the motion of an evaluation target and motion measurement data obtained during the motion shown in the motion video data; a control step of performing learning of a learning target using motion video data, which is video data showing the motion of an evaluation target which is a person or animal whose motion is to be evaluated, motion measurement data, which is a time series of results obtained during the motion shown in the motion video data by a motion measurement sensor which is a sensor attached to at least one of the body of the evaluation target, something worn by the evaluation target, and something used by the evaluation target during the motion, and which obtains a result according to the motion of the evaluation target, and ground truth data indicating the result of the evaluation of the motion, and the learning target is a mathematical model which evaluates the motion based on the motion video data and the motion measurement data, and a learned evaluation model execution step of evaluating the motion shown in the motion video data included in the set using the learned mathematical model obtained by the learning method and the set obtained by the interface step.

One aspect of the present invention is a program for causing a computer to function as either or both of the above-mentioned learning device and the above-mentioned evaluation device.

The present invention makes it possible to provide technology that improves the accuracy of motion evaluation.

FIG. 1 is a diagram showing an example of the configuration of an evaluation system according to an embodiment. FIG. 2 is a diagram showing an example of a hardware configuration of a learning device according to an embodiment. FIG. 2 is a diagram showing an example of a hardware configuration of an evaluation apparatus according to an embodiment. 1 is a flowchart showing an example of a flow of a process executed by a learning device in an embodiment. 4 is a flowchart showing an example of a flow of a process executed by an evaluation device in the embodiment. 13A to 13C are diagrams showing an example of a first modification process in the modified example. FIG. 11 is an explanatory diagram illustrating an example of a process for assigning a noteworthy identifier in a modified example.

(Embodiment)
1 is a diagram showing an example of the configuration of an evaluation system 100 according to an embodiment. The evaluation system 100 includes a learning apparatus 1 and an evaluation apparatus 2. The learning apparatus 1 includes a control unit 11 including a processor 91 such as a CPU (Central Processing Unit) and a memory 92 connected by a bus, and executes a program.

The control unit 11 executes a learning process. The learning process is a process for learning a mathematical model of the learning target. The mathematical model of the learning target is a mathematical model (hereinafter referred to as an "evaluation model") that evaluates the movement of the evaluation target using a set of movement video data, movement measurement data, and ground truth data (hereinafter referred to as "learning data").

The motion video data is video data of a video showing the motion of a person or animal (hereafter referred to as the "evaluation subject") whose motion is to be evaluated.

The subject of evaluation may be any person or animal, so long as it is an object of evaluation of motion. The subject of evaluation may be, for example, an athlete in a sport such as figure skating, surfing, or diving. When the subject of evaluation is an athlete, the motions evaluated by the evaluation model are the athlete's motions during the competition. The subject of evaluation may be, for example, a medical intern. When the subject of evaluation is a medical intern, the motions evaluated by the evaluation model are the motions during a medical procedure such as surgery.

The motion measurement data is a time series of results obtained by a motion measurement sensor during the motion captured in the motion video data. The motion measurement sensor is a sensor attached to at least one of the body of the subject to be evaluated, something worn by the subject to be evaluated, and something used by the subject to be evaluated when performing the motion, and obtains results according to the motion of the subject to be evaluated.

The motion measurement sensor is, for example, an acceleration sensor. In such a case, the result obtained by the motion measurement sensor is acceleration. Therefore, in such a case, the time series of the results obtained by the motion measurement sensor is a time series of acceleration. The motion measurement sensor is, for example, an angular velocity sensor. In such a case, the result obtained by the motion measurement sensor is angular velocity. Therefore, in such a case, the time series of the results obtained by the motion measurement sensor is a time series of angular velocity.

Otherwise, the motion measurement sensor may be, for example, an inertial measurement unit (IMU), or a sensor that obtains biosignals that indicate changes in response to the motion of the subject being evaluated, such as heart rate, brain waves, pulse, blood pressure, breathing, and sweating.

If a sensor that obtains results according to the movements of the subject is attached to the body of the subject or to something worn by the subject, the results obtained by the sensor will be less affected by the environment, such as the intensity and color of lighting, than images, and will have a high correlation with the movements of the subject. Therefore, an evaluation based on not only movement video data but also movement measurement data will be more accurate than an evaluation based only on movement video data.

Explain what the subject of evaluation uses when making movements. For example, if the sport is surfing, the subject of evaluation uses a surfboard during the competition. The surfboard makes movements that are highly correlated with the movements of the subject of evaluation in many scenes during the competition. Therefore, if a sensor is attached to the surfboard, results that are highly correlated with the movements of the subject of evaluation can be obtained even if the sensor is not attached to the body of the subject of evaluation. Again, this result is less affected by environmental factors such as lighting intensity and color than video images.

Therefore, even if the sensor is attached to something used during the movement of the subject being evaluated, using the time series of results obtained by the sensor will provide a more accurate evaluation than an evaluation based only on video data of the movement. This is not limited to surfing, but is common to all events where movements using equipment such as skis and snowboards are the subject of grading.

Furthermore, when the motion evaluated by the evaluation model is a medical procedure, the movement of medical equipment used during the medical procedure, such as a scalpel, also has a high correlation with the motion of the object being measured. Therefore, even if the motion is during a medical procedure, if a motion measurement sensor is attached to the medical equipment, the motion measurement sensor will obtain results that are highly correlated with the motion of the object being evaluated, even if the motion measurement sensor is not attached to the body of the object being evaluated. Again, this is a result that is less affected by the environment, such as the intensity and color of lighting, than video.

Therefore, even if the movement being evaluated is a medical procedure and the sensor is attached to something used during the movement of the subject being evaluated, using the time series of results obtained by the sensor will provide a more accurate evaluation than an evaluation based solely on video data of the movement.

If the subject to be evaluated is an animal such as a dog or cat, the motion measurement sensor may be attached to the collar, for example, or, if a tag is attached to the foot or the like, it may be attached to the tag.

Such motion measurement sensors may be sensors that obtain information on the movement of the evaluation subject in each of three linearly independent directions, such as forward/backward, left/right, and up/down, or may be sensors that obtain information in one or two dimensions.

The motion measurement data used by the evaluation model does not have to be the results obtained by a single motion measurement sensor. The evaluation model may use motion measurement data obtained by multiple motion measurement sensors. In such a case, multiple motion measurement data are input to the evaluation model. The multiple motion measurement sensors may be multiple types of motion measurement sensors that obtain different physical quantities, such as a motion measurement sensor that obtains acceleration and a motion measurement sensor that obtains bioinformation. In such a case, the evaluation model uses time series of multiple different types of physical quantities as motion measurement data for evaluation.

Correct answer data is data that indicates the results of the evaluation of the subject's movements.

In the learning process, the control unit 11 updates the evaluation model based on the evaluation result of the evaluation model and the correct answer data so as to reduce the difference between the evaluation result of the evaluation model and the correct answer data. The difference may be expressed, for example, by the L1 norm, the L2 norm, or the linear sum of the L1 norm and the L2 norm.

The learning process is executed until a predetermined end condition for learning (hereinafter referred to as the "learning end condition") is satisfied. The learning end condition is, for example, a condition that the change in the learning object due to learning is smaller than a predetermined change. The learning end condition may be, for example, a condition that the learning object has been updated a predetermined number of times or more.

The evaluation device 2 evaluates the movement of the evaluation subject using a trained evaluation model, which is the evaluation model at the time when the learning termination condition is satisfied. The evaluation device 2 accepts input of a set of movement video data showing the movement of the evaluation subject and movement measurement data obtained during the movement shown in the movement video data (hereinafter referred to as "inference stage data"). The evaluation device 2 performs evaluation based on the accepted inference stage data.

<About the evaluation model>
The evaluation model may be of any type as long as it can evaluate the motion of the evaluation subject using motion video data, motion measurement data, and ground truth data.

The evaluation model includes, for example, a video data feature acquisition process and a measurement data feature acquisition process. The video data feature acquisition process is a process for acquiring the feature of motion video data. The measurement data feature acquisition process is a process for acquiring the feature of motion measurement data.

An evaluation model including a video data feature acquisition process and a measurement data feature acquisition process evaluates the motion of the evaluation subject based on the results of the video data feature acquisition process and the results of the measurement data feature acquisition process.

<Regarding processing targets for obtaining feature amounts>
In the video data feature acquisition process, the feature may be obtained for each group (hereinafter referred to as "partial video data") of the action video data divided into a plurality of groups along the time axis. Therefore, in this case, the feature of the partial video data is obtained for each partial video data.

For example, if the video is motion video data consisting of P x Q frames (P is an integer equal to or greater than 1, and Q is an integer equal to or greater than 2), in the video feature acquisition process, the video is divided into partial video data for every P frames, for example, in the order of frame numbers, and features are acquired for each partial video data. Therefore, in such a case, the video data feature acquisition process obtains Q features from the motion video data.

Note that the process of dividing the motion video data into partial video data (hereinafter referred to as the "video data division process") may be performed at any time before the video data feature acquisition process is performed.

Note that the video data division process is a process of dividing the motion video data along the time axis, for example, for each predetermined number of frames.

Note that action video data that has been divided into partial video data is also a type of action video data. Therefore, the action video data input to the evaluation model may be action video data after video data division processing. When the action video data input to the evaluation model is action video data after video data division processing, the video data division processing may be executed, for example, by the control unit 11, or may be executed by another device other than the learning device 1.

When another device executes, the control unit 11 acquires the results, for example via the interface unit 12 described below, and uses them for evaluation by executing the evaluation model. Note that the motion video data input to the evaluation model may be motion video data before the video data division process, and in such a case, when the video data division process is also executed, the evaluation model executes the video data division process.

The same situation applies to the measurement data feature acquisition process. In the measurement data feature acquisition process, feature values may be obtained for each group (hereinafter referred to as "partial measurement data") after the motion measurement data, which is divided into a plurality of groups at a predetermined time width along the time axis, is divided. Therefore, in this case, feature values for each partial measurement data are obtained.

When feature values are obtained for each of multiple partial measurement data of motion measurement data, the number of feature values obtained is greater than that obtained when the motion measurement data is not divided into partial measurement data.

Note that the process of dividing the motion measurement data into partial measurement data (hereinafter referred to as the "measurement data division process") may be performed at any time before the measurement data feature acquisition process is performed.

The measurement data division process is, for example, a process of dividing the measurement video data, which is a type of time series, along the time axis for each predetermined time width.

Note that motion measurement data that has been divided into partial measurement data is also a type of motion measurement data. Therefore, the motion measurement data input to the evaluation model may be motion measurement data after measurement data division processing. When the motion measurement data input to the evaluation model is motion measurement data after measurement data division processing, the measurement data division processing may be executed, for example, by the control unit 11, or may be executed by a device other than the learning device 1.

When another device executes, the control unit 11 acquires the results, for example via the interface unit 12 described below, and uses them for evaluation by executing the evaluation model. Note that the motion measurement data input to the evaluation model may be the motion measurement data before the measurement data division process, and in such a case, when the measurement data division process is also executed, the evaluation model executes the measurement data division process.

<How features are used for evaluation>
As described above, the evaluation model evaluates the motion of the evaluation object based on the result of the video data feature amount acquisition process and the result of the measurement data feature amount acquisition process. In evaluating the motion of the evaluation object based on the result of the video data feature amount acquisition process and the result of the measurement data feature amount acquisition process, the result of the video data feature amount acquisition process and the result of the measurement data feature amount acquisition process may be used in an integrated state to evaluate the motion of the evaluation object.

In evaluating the motion of the evaluation subject based on the results of the video data feature acquisition process and the results of the measurement data feature acquisition process, the results of the video data feature acquisition process and the measurement data feature acquisition process may be used in an unintegrated state to evaluate the motion of the evaluation subject.

Integration is the process of combining different data into one. For example, when the data is expressed as vectors, it is the process of obtaining the Cartesian product of the vectors. When the data is expressed as matrices, it may be the process of concatenating matrices in the row or column direction.

For the integration, one or more fully connected layers may be used, or the Transformer Encoder described in Reference 1 below may be used. In this case, the number of Transformer Blocks may be any number, for example, two Transformer Blocks may be connected in a row.

Reference 1: Vaswani, Ashish, et al. “Attention is all you need.” Advances in neural information processing systems 30 (2017).

<Example of hardware configuration>
2 is a diagram showing an example of a hardware configuration of the learning device 1 according to the embodiment. As described above, the learning device 1 includes the control unit 11 and executes a program. The learning device 1 functions as a device including the control unit 11, an interface unit 12, and a storage unit 13 by executing the program.

More specifically, the processor 91 reads out a program stored in the storage unit 13 and stores the read out program in the memory 92. The processor 91 executes the program stored in the memory 92, whereby the learning device 1 functions as a device including a control unit 11, an interface unit 12, and a storage unit 13.

The control unit 11 performs, for example, a learning process. The control unit 11 performs, for example, a video data division process. The control unit 11 performs, for example, a measurement data division process. The control unit 11 controls, for example, the operation of each functional unit provided in the learning device 1. The control unit 11 acquires learning data, for example, via the interface unit 12. The control unit 11 acquires, for example, information stored in the memory unit 13. The process of acquiring information stored in the memory unit 13 is specifically a read process.

The interface unit 12 includes a communication interface for connecting the learning device 1 to an external device. The interface unit 12 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits learning data. In such a case, the interface unit 12 acquires learning data by communicating with the device that transmits the learning data.

When an external device performs video data division processing, the interface unit 12 obtains the motion video data after the video data division processing by the external device as motion video data including learning data. When an external device performs measurement data division processing, the interface unit 12 obtains the motion measurement data after the measurement data division processing by the external device as motion measurement data including learning data.

The external device is, for example, the evaluation device 2. The interface unit 12 transmits the trained evaluation model obtained by the learning process (i.e., the trained evaluation model) to the evaluation device 2 through communication with the evaluation device 2.

The interface unit 12 may be configured to include input devices such as a mouse, keyboard, touch panel, etc. The interface unit 12 may be configured as an interface that connects these input devices to the learning device 1. In this way, the interface unit 12 accepts input of various information to the learning device 1 via the input device, either wired or wirelessly. Note that learning data does not necessarily need to be input to a communication interface, and may be input to an input device.

The interface unit 12 may output various types of information. The interface unit 12 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The interface unit 12 may be configured as an interface that connects these display devices to the learning device 1. The interface unit 12 outputs information that has been input to the interface unit 12, for example.

The memory unit 13 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The memory unit 13 stores various information related to the learning device 1. The memory unit 13 stores, for example, information necessary for the learning process. The memory unit 13 stores, for example, an evaluation model in advance. The memory unit 13 stores, for example, various information generated by the operation of the control unit 11. Thus, the memory unit 13 stores, for example, a learned evaluation model. The memory unit 13 stores, for example, information acquired by the interface unit 12.

FIG. 3 is a diagram showing an example of the hardware configuration of the evaluation device 2 in an embodiment. As described above, the evaluation device 2 includes a control unit 21 (an example of a trained evaluation model execution unit) and executes a program. By executing the program, the evaluation device 2 functions as a device including the control unit 21, an interface unit 22, and a memory unit 23.

More specifically, the processor 93 reads the program stored in the storage unit 23 and stores the read program in the memory 94. The processor 93 executes the program stored in the memory 94, whereby the evaluation device 2 functions as a device including the control unit 21, the interface unit 22, and the storage unit 23.

The control unit 21 executes the learned evaluation model, for example, with the inference stage data as the execution target. This allows the control unit 21 to evaluate the actions depicted in the action video data included in the inference stage data. In other words, the control unit 21 uses the learned evaluation model and the inference stage data to evaluate the actions depicted in the action video data included in the inference stage data.

If the control unit 11 has performed video data splitting processing during the learning stage of the evaluation model but the video data splitting processing is not included in the evaluation model, the control unit 21 performs the video data splitting processing before executing the learned evaluation model. If the control unit 11 has performed measurement data splitting processing during the learning stage of the evaluation model but the measurement data splitting processing is not included in the evaluation model, the control unit 21 performs the measurement data splitting processing before executing the learned evaluation model.

The control unit 21 controls the operation of each functional unit of the evaluation device 2, for example. The control unit 21 acquires various information acquired, for example, via the interface unit 22. Thus, for example, the control unit 21 acquires data on which the trained evaluation model is to be executed (i.e., inference stage data). The control unit 21 acquires information stored in the memory unit 23, for example. The process of acquiring information stored in the memory unit 23 is specifically a read process.

The interface unit 22 includes a communication interface for connecting the evaluation device 2 to an external device. The interface unit 22 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits inference stage data. In such a case, the interface unit 22 acquires the inference stage data by communicating with the device that transmits the inference stage data. The external device is, for example, the learning device 1. The interface unit 22 acquires the trained evaluation model by communicating with the learning device 1.

When an external device executes a video data division process, the interface unit 22 obtains the motion video data after the video data division process by the external device as motion video data including inference stage data. When an external device executes a video data division process during the learning stage of the evaluation model, the evaluation device 2 also obtains the result of the video data division process executed by the external device.

When an external device executes the measurement data division process, the interface unit 22 obtains the motion measurement data after the measurement data division process by the external device as motion measurement data including inference stage data. When an external device executes the measurement data division process during the learning stage of the evaluation model, the evaluation device 2 also obtains the result of the measurement data division process executed by the external device.

The interface unit 22 includes input devices such as a mouse, keyboard, and touch panel. The interface unit 22 may be configured as an interface that connects these input devices to the evaluation device 2. In this way, the interface unit 22 accepts input of various information to the evaluation device 2 via the input device, either wired or wirelessly. Note that the inference stage data does not necessarily need to be input to a communication interface, and may be input to an input device.

The interface unit 22 outputs various types of information. The interface unit 22 includes a display device such as a CRT display, a liquid crystal display, or an organic EL display. The interface unit 22 may be configured as an interface that connects these display devices to the evaluation device 2. The interface unit 22 outputs, for example, information input to the interface unit 22.

The storage unit 23 is configured using a computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 23 stores various information related to the evaluation device 2. The storage unit 23 stores, for example, a trained evaluation model. The storage unit 23 stores, for example, various information generated by the operation of the control unit 21. The storage unit 23 stores, for example, information acquired by the interface unit 22.

FIG. 4 is a flowchart showing an example of the flow of processing executed by the learning device 1 in the embodiment. The control unit 11 acquires learning data (step S101). The control unit 11 executes a learning process using the acquired learning data until a learning end condition is satisfied (step S102). The control unit 11 outputs the trained evaluation model to a predetermined output destination (step S103). The predetermined output destination is, for example, the memory unit 13. Note that output to the memory unit 13 means writing to the memory unit 13. The predetermined output destination may be, for example, an external device such as the evaluation device 2 that is communicatively connected via the interface unit 12.

FIG. 5 is a flowchart showing an example of the flow of processing executed by the evaluation device 2 in an embodiment. For ease of explanation, it is assumed in the example of the flowchart in FIG. 5 that the trained evaluation model has been stored in advance in the storage unit 23.

Inference stage data is input to the interface unit 22, and the control unit 21 acquires the inference stage data input to the interface unit 22 (step S201). Inputting inference stage data to the interface unit 22 means that the interface unit 22 acquires the inference stage data.

The control unit 21 executes the learned evaluation model on the inference stage data obtained in step S201 (step S202). By the processing of step S202, the action shown in the action video data included in the inference stage data obtained in step S201 is evaluated. After step S202, the control unit 21 outputs the evaluation result obtained in step S202 to a predetermined output destination (step S203). The predetermined output destination is, for example, the memory unit 23. Note that output to the memory unit 23 means writing to the memory unit 23. The predetermined output destination may be, for example, a predetermined external device connected for communication via the interface unit 22.

The learning device 1 configured in this manner learns an evaluation model based not only on movement video data but also on movement measurement data. Because the movement measurement data is obtained by a movement measurement sensor, it is less affected by the environment, such as lighting intensity and color, than video. Therefore, an evaluation model learned based not only on movement video data but also on movement measurement data can be evaluated with higher accuracy than when based only on movement video data. Therefore, the learning device 1 can improve the accuracy of movement evaluation.

The evaluation device 2 configured in this way evaluates the movement using the learned mathematical model obtained by the learning device 1. Therefore, the evaluation device 2 can improve the accuracy of the movement evaluation.

The evaluation system 100 configured in this manner also includes a learning device 1. As a result, the evaluation system 100 can improve the accuracy of the evaluation of actions.

(Modification)
In addition, in evaluating the motion of the evaluation target object based on the feature amounts obtained in the video data feature amount acquisition process and the feature amounts obtained in the measurement data feature amount acquisition process, for example, the following first transformation process may be executed. The first transformation process is a process in which the dimensions of the feature amounts obtained in the video data feature amount acquisition process and the feature amounts obtained in the measurement data feature amount acquisition process are aligned via a linear layer, input to a transformer block via embedding processing, and evaluated through an estimator. The estimator is, for example, a plurality of fully connected layers.

In the first transformation process, the processing of the Linear layer is a process for aligning the dimensions of the features obtained in the video data feature acquisition process and the feature obtained in the measurement data feature acquisition process, so if the dimensions of the features obtained in the video data feature acquisition process and the feature obtained in the measurement data feature acquisition process are originally the same, it does not necessarily have to be executed. Furthermore, the structure of the Linear layer may be any, and may be, for example, a single fully connected layer.

The embedding process is explained below. Because the Transformer block handles inputs in parallel, it cannot take into account the chronological order of the inputs, or whether the input is the result of video data feature acquisition processing or measurement data feature acquisition processing. "Not being able to take into account" means that such information cannot be used. Therefore, when using the Transformer, information indicating the attributes of the features input to the Transformer must be provided by the embedding process to the Transformer block, which is the layer that executes the Transformer.

The information indicating the attribute of a feature includes, for example, information as to whether the feature is the result of a video data feature acquisition process or a measurement data feature acquisition process. For example, if the feature is the result of a video data feature acquisition process, the information indicating the attribute may include information indicating which partial video data the feature is a feature of. For example, if the feature is the result of a measurement data feature acquisition process, the information indicating the attribute may include information indicating which partial measurement data the feature is a feature of.

The embedding process obtains information indicating the attributes of the features used in the Transformer so that the Transformer process is based on information indicating the attributes of the features.

The embedding process in the first transformation process is, for example, sensor embedding. Sensor embedding is a process that assigns an identifier to each feature input to the Transformer, identifying whether it is the result of a video data feature acquisition process or a measurement data feature acquisition process. The identifier can be any identifier as long as it can identify whether it is the result of a video data feature acquisition process or a measurement data feature acquisition process.

Therefore, the identifier may be, for example, "1" indicating that it is the result of the video data feature acquisition process, and "0" indicating that it is the result of the measurement data feature acquisition process. Also, instead of integers such as 1 or 0, it may be a decimal point such as 0.1, or it may not even be a number.

The identifier to be assigned is determined in advance based on the results of the video data feature acquisition process and the measurement data feature acquisition process.

The embedding process in the first transformation process is, for example, positional embedding. Positional embedding is a process that assigns an identifier indicating which partial video data the feature is from when the feature is the result of a video data feature acquisition process, and assigns an identifier indicating which partial measurement data the feature is from when the feature is the result of a measurement data feature acquisition process. The assigned identifier is, for example, an identifier that indicates the temporal order of the partial video data or partial measurement data.

FIG. 6 is a diagram showing an example of the first transformation process in the modified example. In the example of FIG. 6, one piece of motion video data and two pieces of motion measurement data are used. In the example of FIG. 6, one piece of motion video data is divided into partial video data, and both pieces of motion measurement data are divided into partial measurement data. In the example of FIG. 6, both the partial video data and the partial measurement data are expressed as "Clip". In the example of FIG. 6, Position Embedding and Sensor Embedding are performed. In the example of FIG. 6, there is a Linear layer, and the output of the Linear layer is input to a Transformer block together with the results of Position Embedding and Sensor Embedding. In the example of FIG. 6, evaluation is performed based on the results of the Transformer block.

<Technology enabling the processing of heterogeneous data>
In the above explanation, the importance of each sample in the time series of the motion video data and the motion measurement data is not explicitly assigned to the samples when evaluating the motion. In other words, all samples are treated uniformly for evaluation.

However, video footage data and movement measurement data may contain movements that are not necessarily suitable for evaluating movements. For example, in judged sports, this data may include data on the period before the competition begins or the period after the competition ends when players are substituted. When evaluating movements, the movements indicated by this data are not subject to evaluation, and may act as noise when evaluating movements.

The learning device 1 may therefore suppress the occurrence of such a situation by making the time series samples in the action video data and the action measurement data non-uniform. Specifically, the evaluation model also performs evaluation based on a focus identifier, which is an identifier that indicates that an influence on the evaluation of an action is stronger than others in the action video data and the action measurement data. In such a case, the evaluation model is capable of evaluation with higher accuracy.

The attention identifier is assigned to samples from a period that includes the time when a specified index showing the intensity of the movement is highest in the graph shown by the movement measurement data and that satisfies a specified condition. The evaluation model evaluates samples that have been assigned an attention identifier by weighting them more heavily than samples that have not been assigned an attention identifier.

The predetermined condition may be any condition that specifies a period of time that includes the time when a predetermined index indicating the degree of intensity of the movement is highest in the graph shown by the movement measurement data. For example, the predetermined condition is a condition that specifies each time span before and after the time when a predetermined index indicating the degree of intensity of the movement is highest in the graph shown by the movement measurement data as a predetermined time span.

As described above, the evaluation model weights samples that have been assigned a focus identifier more heavily than samples that do not. Therefore, since the more intense the movement, the stronger the impact on the evaluation of the movement should be, the focus identifier is assigned to samples from the period that includes the time when a specified index showing the degree of movement intensity is highest in the graph shown by the movement measurement data, and that satisfies specified conditions. As a result, the evaluation model can perform evaluation with greater accuracy.

Note that assigning a focus identifier to data means determining that the data being evaluated satisfies the content indicated by the focus identifier, and recording information indicating the evaluation result in a specified storage device from which an evaluation model, such as storage unit 13 or storage unit 23, can read information.

The process of making the time series samples in the action video data and the action measurement data non-uniform includes a process of dividing the action video data and the action measurement data into a plurality of Cilps, and in the process of dividing into Clips, the action video data and the action measurement data may be divided into a plurality of Cilps such that the number of Clips included in a period that includes the time when a predetermined index indicating the degree of intensity of the action is highest in the graph shown by the action measurement data and that satisfies a predetermined condition is greater than in other periods.

FIG. 7 is an explanatory diagram illustrating an example of the process of assigning an attention identifier in a modified example. FIG. 7 shows three types of time series. The three types of time series are IMU signals. In the example of FIG. 7, an attention identifier is assigned to motion video data and motion measurement data within a period that satisfies a predetermined attention period condition that includes at least the condition that the maximum differential coefficient time is included within the period.

The maximum derivative time is the time that gives the maximum derivative value in the graph of the movement measurement data. When moving, the value of the movement measurement data should change more drastically than when not moving.

Therefore, if an attention identifier is assigned so that it includes the time at which the maximum derivative coefficient is obtained in the graph of the movement measurement data, the sample to which the attention identifier is assigned is highly likely to be data obtained while the movement is being performed. Therefore, in the example of Figure 7, the time of maximum derivative coefficient is an example of the time at which a specified index showing the degree of intensity of the movement is highest in the graph shown by the movement measurement data.

In the example of Figure 7, the set of samples that have been assigned a focus identifier are samples from the period indicated as "Technique". In Figure 7, the horizontal axis indicates time. In Figure 7, the "Preparation Period" is the period when the athlete is preparing. In Figure 7, a "Technique" is the period when the athlete is performing an action. In Figure 7, "Post-Execution" is the period when the athlete has finished performing an action and is not performing an action. In Figure 7, the time t = ta is an example of the maximum derivative time.

Hereinafter, the process of assigning an attention identifier to the action video data and action measurement data within a period that satisfies the attention period condition based on the action measurement data is referred to as the attention identifier assignment process.

The attention period condition may be any condition including a condition that the maximum derivative time is included within the period. The attention period condition may be, for example, a condition that the period has a predetermined time width before and after the maximum derivative time. The attention period condition may be, for example, a condition that the period has a predetermined time width before and after the maximum derivative time, and is longer than a period that does not satisfy the attention period condition.

The attention period condition may be, for example, a period having a predetermined time width before and after the maximum differential coefficient time, and the length of the period may be a predetermined length that is at least 1/3 of the time from the start to the end of the image shown by the motion image data.

The meaning of 1/3 will be explained. The image shown by the motion video data may show a preparatory period when the athlete is getting ready, a period when the athlete is performing, and a period after the athlete has finished performing. In such a case, if the preparatory period when the athlete is getting ready, the period when the athlete is performing, and the period after the athlete has finished performing are all of the same duration, it is preferable to give the evaluation model a period centered on the maximum differential coefficient time and that is 1/3 the length of the video shown by the motion video data.

However, when evaluating an action, it is expected that the user will delete unnecessary data and input the data as action video data to the learning device 1. In such a case, it is preferable that the attention period condition is, for example, a period having a predetermined time width before and after the maximum differential coefficient time, and the length of the period is a predetermined length that is at least 1/3 of the time from the start to the end of the video shown by the action video data.

When the attention identifier assignment process is executed, it may be executed at any timing before the evaluation model determines the evaluation of the action. The attention identifier assignment process is executed, for example, before the video data feature acquisition process and the measurement data feature acquisition process are executed. In such a case, the evaluation model acquires features by weighting data to which an attention identifier has been assigned more heavily than other data in the video data feature acquisition process and the measurement data feature acquisition process.

If the attention identifier assignment process is executed before the video data feature acquisition process and the measurement data feature acquisition process are executed, the attention identifier assignment process may be executed, for example, after the video data division process and the measurement data division process are executed. Note that, if the video data division process and the measurement data division process are executed, the attention identifier assignment process may be assigned to each partial video data and partial measurement data, thereby assigning an attention identifier to the sample.

If the attention identifier assignment process is executed after the video data division process and the measurement data division process, for example, it may be executed after the video data feature acquisition process and the measurement data feature acquisition process. In such a case, since an attention identifier and feature are obtained for each partial video data and partial measurement data, the evaluation model may determine the evaluation by giving a large weight to the feature to which the attention identifier has been assigned among the obtained feature values.

The attention identifier assignment process may be executed by the control unit 11 during the learning stage, or may be executed by an external device when the video data division process and the measurement data division process are executed by the external device. When executed by an external device, the control unit 11 obtains the results of the attention identifier assignment process together with the results of the division of the video data division process and the measurement data division process via the interface unit 12.

In addition, when the attention identifier assignment process is executed in the learning phase, the evaluation device 2 also uses the result of the attention identifier assignment process. For example, when the control unit 11 executes the attention identifier assignment process in the learning phase, the control unit 21 obtains the result of the attention identifier assignment process by, for example, the control unit 21 executing the attention identifier assignment process. For example, when the control unit 11 executes the attention identifier assignment process in the learning phase, the control unit 21 may obtain the result of the attention identifier assignment process by having an external device execute the attention identifier assignment process and obtaining the result.

For example, if an external device executes the attention identifier assignment process in the learning phase, the control unit 21, for example, acquires the result of the attention identifier assignment process by having the control unit 21 execute the attention identifier assignment process. For example, if an external device executes the attention identifier assignment process in the learning phase, the control unit 21 may acquire the result of the attention identifier assignment process by having the external device execute the attention identifier assignment process and acquiring the result.

The learning device 1 may be implemented using multiple information processing devices connected to each other so that they can communicate with each other via a network. In this case, each process executed by the control unit 11 may be executed in a distributed manner by the multiple information processing devices.

The evaluation device 2 may be implemented using multiple information processing devices connected to each other so that they can communicate with each other via a network. In this case, each process executed by the control unit 21 may be executed in a distributed manner by the multiple information processing devices.

The learning device 1 and the evaluation device 2 do not necessarily need to be implemented in different housings, and may be integrated into a single housing. In such a case, the learning device 1 and the evaluation device 2 may be a computer that functions as both the learning device 1 and the evaluation device 2 by executing a program.

All or part of the functions of the learning device 1 and the evaluation device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

The control unit 21 is an example of a trained evaluation model execution unit.

　Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs that do not deviate from the gist of the present invention.

100...Evaluation system, 1...Learning device, 2...Evaluation device, 11...Control unit, 12...Interface unit, 13...Memory unit, 21...Control unit, 22...Interface unit, 23...Memory unit, 91...Processor, 92...Memory, 93...Processor, 94...Memory

Claims (8)

a control unit that learns a learning object using motion video data, which is video data of a video showing a motion of an evaluation subject that is a human or animal whose motion is to be evaluated, motion measurement data, which is a time series of results obtained during the motion shown in the motion video data by a motion measurement sensor that is attached to at least one of the body of the evaluation subject, an item worn by the evaluation subject, and an item used by the evaluation subject when performing the motion and that obtains results according to the motion of the evaluation subject, and ground truth data that indicates the result of evaluation of the motion;
Equipped with
The learning object is a mathematical model that evaluates the movement based on the movement video data and the movement measurement data.
Learning device. The mathematical model obtains a feature amount of the partial video data for each partial video data set of the motion video data divided into a plurality of sets along a time axis.
The learning device according to claim 1 . The mathematical model obtains a feature amount of the partial measurement data for each partial measurement data set obtained after dividing the motion measurement data into a plurality of sets at a predetermined time width along a time axis.
The learning device according to claim 1 . The mathematical model performs the evaluation based on a note identifier that indicates a stronger influence on the evaluation of the action than other identifiers in the action video data and the action measurement data.
The learning device according to claim 1 . an interface unit that acquires a set of motion image data showing a motion of an evaluation target and motion measurement data obtained during the motion shown in the motion image data;
a control unit that performs learning of a learning object using motion video data, which is video data of a video showing a motion of an evaluation object, which is a human or animal whose motion is to be evaluated, motion measurement data, which is a time series of results obtained during the motion captured in the motion video data by a motion measurement sensor, which is a sensor attached to at least one of the body of the evaluation object, an item worn by the evaluation object, and an item used by the evaluation object when performing the motion, and which obtains a result according to the motion of the evaluation object, and ground truth data that indicates the result of evaluation of the motion, wherein the learning object is a mathematical model that evaluates the motion based on the motion video data and the motion measurement data, and a learned evaluation model execution unit that uses the learned mathematical model obtained by the learning device and the set obtained by the interface unit to evaluate the motion captured in the motion video data included in the set;
An evaluation device comprising: a control step of learning a learning object using motion video data, which is video data of a video showing the motion of an evaluation object, which is a human or animal whose motion is to be evaluated, motion measurement data, which is a time series of results obtained during the motion shown in the motion video data by a motion measurement sensor, which is a sensor attached to at least one of the body of the evaluation object, an item worn by the evaluation object, and an item used by the evaluation object when performing the motion, and which obtains results according to the motion of the evaluation object, and ground truth data, which indicates the result of evaluation of the motion;
having
The learning object is a mathematical model that evaluates the movement based on the movement video data and the movement measurement data.
How to learn. an interface step of acquiring a set of motion image data showing a motion to be evaluated and motion measurement data obtained during the motion shown in the motion image data;
a control step of learning a learning object using motion video data, which is video data of a video showing a motion of an evaluation object, which is a human or animal to be evaluated for motion, motion measurement data, which is a time series of results obtained during the motion captured in the motion video data by a motion measurement sensor, which is a sensor attached to at least one of the body of the evaluation object, an item worn by the evaluation object, and an item used by the evaluation object when performing the motion, and which obtains a result according to the motion of the evaluation object, and ground truth data, which indicates a result of evaluation of the motion, wherein the learning object is a mathematical model that evaluates the motion based on the motion video data and the motion measurement data; and a learned evaluation model execution step of evaluating the motion captured in the motion video data included in the set, using the learned mathematical model obtained by the learning method and the set obtained by the interface step;
An evaluation method having the following: A program for causing a computer to function as either or both of the learning device described in any one of claims 1 to 4 and the evaluation device described in claim 5.

Images