JP7057959B2 - Motion analysis device - Google Patents

Description

The present invention relates to a motion analysis device, method and program for analyzing the motion of an object, and a model building device, method and program for constructing a model for analyzing the motion of an object, and particularly analyzes a golf swing. With respect to suitable equipment, methods and programs.

Conventionally, a device for photographing a golf swing with a camera and analyzing the golf swing based on the image at this time has been known (Patent Documents 1, 2, etc.). The results of the analysis are used for various purposes such as fitting a golf club suitable for a golfer, improving the form of a golfer, and developing golf equipment. In the analysis of the golf swing as described above, three-dimensional measurement of the part of interest such as the grip and head of the golf club and the joint of the golfer shown in the image is often performed. Patent Documents 1 and 2 disclose that a golf swing is photographed from a plurality of directions by a plurality of cameras, and three-dimensional measurement of a region of interest is performed based on a triangulation method, a DLT method, or the like.

In recent years, in addition to two-dimensional cameras, a device called kinect (registered trademark) equipped with a range image sensor that enables three-dimensional measurement has become widespread, and research on motion analysis of people using this device has been active. (For example, Patent Documents 3, 4, etc.).

Japanese Unexamined Patent Publication No. 2013-215349 Japanese Unexamined Patent Publication No. 2004-313479 Japanese Unexamined Patent Publication No. 2015-106281 Japanese Unexamined Patent Publication No. 2016-75693

However, the camera configuration as in Patent Documents 1 and 2 can increase the size of the equipment. On the other hand, the operation analysis technique based on the depth image from the distance image sensor, such as Patent Documents 3 and 4, is still in the development stage, and further improvement of the analysis accuracy is desired.

The present invention provides a motion analysis device, a method and a program capable of analyzing the motion of an object easily and with high accuracy, and a model building device, a method and a program for constructing a model for analyzing the motion of an object. With the goal.

The motion analysis device according to the first aspect is a motion analysis device for analyzing the motion of an object, and includes an acquisition unit and a derivation unit. The acquisition unit acquires a depth image obtained by capturing the movement of the object with a distance image sensor. The derivation unit derives the operation value by inputting the depth image acquired by the acquisition unit into a neural network that outputs an operation value that quantitatively represents the operation of the object.

The motion analysis device according to the second aspect is the motion analysis device according to the first aspect, and the depth image is an image obtained by photographing a golf swing.

The motion analysis device according to the third aspect is the motion analysis device according to the second aspect, and the motion value is the rotation angle of the golfer's waist.

The motion analysis device according to the fourth aspect is the motion analysis device according to the third aspect, and the acquisition unit further acquires skeleton data representing the skeleton of the human body measured by the distance image sensor. The derivation unit extracts the waist region in the vicinity of the waist from the depth image based on the skeleton data, and then inputs the image of the waist region to the neural network.

The motion analysis device according to the fifth aspect is the motion analysis device according to the third aspect or the fourth aspect, and the neural network is a dropout layer that invalidates the arm region representing the golfer's arm from the depth image. Has.

The motion analysis device according to the sixth aspect is the motion analysis device according to the second aspect, and the motion value is a value representing the weight shift of the golfer.

The motion analysis device according to the seventh aspect is the motion analysis device according to the sixth aspect, and the acquisition unit further acquires skeleton data representing the skeleton of the human body measured by the distance image sensor. The derivation unit extracts a golfer region in the vicinity of the golfer from the depth image based on the skeleton data, and then inputs the image of the golfer region to the neural network.

The motion analysis device according to the eighth aspect is the motion analysis device according to the second aspect, and the motion value is the rotation angle of the golfer's shoulder.

The motion analysis device according to the ninth aspect is the motion analysis device according to the eighth aspect, and the acquisition unit further acquires skeleton data representing the skeleton of the human body measured by the distance image sensor. The derivation unit extracts a shoulder region in the vicinity of the shoulder from the depth image based on the skeleton data, and then inputs the image of the shoulder region to the neural network.

The motion analysis device according to the tenth viewpoint is the motion analysis device according to any one of the first to ninth viewpoints, and the neural network has a convolution layer.

The motion analysis device according to the eleventh viewpoint is a motion analysis device according to any one of the first to tenth viewpoints, and the acquisition unit acquires the depth image in time series. The derivation unit derives the operation value of the time series by inputting the depth image of the time series into the neural network.

The motion analysis device according to the twelfth viewpoint is a motion analysis device according to any one of the first viewpoint, the second viewpoint, the tenth viewpoint and the eleventh viewpoint, and the acquisition unit is measured by the distance image sensor. Further acquire skeleton data representing the skeleton of the human body. Based on the skeleton data, the derivation unit extracts a region of interest in the vicinity of the region of interest of the object from the depth image, and then inputs an image of the region of interest to the neural network.

The motion analysis device according to the thirteenth viewpoint is a motion analysis device according to any one of the first viewpoint, the second viewpoint and the tenth viewpoint to the twelfth viewpoint, and the neural network pays attention to the object from the depth image. It has a dropout layer that nullifies non-focused areas that represent areas that are not.

The model building apparatus according to the fourteenth aspect is a model building device for constructing a model for analyzing the motion of an object, and includes a first acquisition unit, a second acquisition unit, and a learning unit. The first acquisition unit acquires a large number of depth images obtained by capturing the movement of the object with a distance image sensor. The second acquisition unit acquires a large number of motion values that quantitatively represent the motion of the object, respectively, corresponding to the large number of depth images. The learning unit uses the large number of operation values acquired by the second acquisition unit as a teacher signal based on the large number of depth images acquired by the first acquisition unit, and inputs the depth image to the learning unit. Learn a neural network that outputs operation values.

The model building apparatus according to the fifteenth viewpoint is the model building device according to the fourteenth viewpoint, and the second acquisition unit represents the rotation angle of the object from the output value of the angular velocity sensor attached to the object. Get the operation value of.

The model building device according to the 16th viewpoint is the model building device according to the 14th viewpoint, and the second acquisition unit represents the position of the center of gravity of the object from the output value of the floor reaction force meter on which the object is placed. Get a large number of operation values.

The motion analysis system according to the seventeenth aspect is a motion analysis system for analyzing the motion of an object, and includes a distance image sensor and a motion analysis device. The distance image sensor captures a depth image that captures the movement of the object. The motion analysis device derives the motion value by inputting the depth image captured by the distance image sensor into a neural network that outputs an motion value that quantitatively represents the motion of the object.

The motion analysis method according to the eighteenth aspect is a motion analysis method for analyzing the motion of an object, and includes the following steps.
(1) A step of taking a depth image that captures the movement of the object by a distance image sensor.
(2) A step of deriving the operation value by inputting the depth image captured by the distance image sensor into a neural network that outputs an operation value that quantitatively represents the operation of the object.

The motion analysis program according to the nineteenth aspect is a motion analysis program for analyzing the motion of an object, and causes a computer to execute the following steps.
(1) A step of acquiring a depth image of the movement of the object taken by a distance image sensor.
(2) A step of deriving the operation value by inputting the acquired depth image into a neural network that outputs an operation value that quantitatively represents the operation of the object.

According to the first aspect, a depth image obtained by capturing the movement of an object with a distance image sensor is acquired, the depth image is input to a neural network, and an operation value that quantitatively represents the movement of the object is output as an output of the neural network. Derived. That is, the neural network can quantitatively evaluate the movement of the object directly from the depth image. From the above, it is possible to analyze the movement of an object easily and with high accuracy.

The figure which shows the whole structure of the motion analysis system including the motion analysis apparatus which concerns on 1st Embodiment of this invention. The functional block diagram of the motion analysis system which concerns on 1st Embodiment. The figure which shows the model structure of the neural network for deriving the rotation angle of the waist. The flowchart which shows the flow of the motion analysis processing based on the neural network which concerns on 1st Embodiment. The figure which shows the example of the normalized depth image. The figure which shows the example of the skeleton data. The figure which shows the example of the image of the waist region of time series. The figure which shows the example of the image of the waist region of the time series which the arm region was removed. Graph of estimated hip rotation angle based on neural network. A flowchart showing the flow of the learning process of the neural network for deriving the rotation angle of the waist. The figure which shows the whole structure of the motion analysis system including the motion analysis apparatus which concerns on 2nd Embodiment of this invention. The functional block diagram of the motion analysis system which concerns on 2nd Embodiment. The figure which shows the model structure of the neural network for deriving the amount of movement of the center of gravity of a body. The figure which shows the model structure of the neural network for deriving the rotation angle of a shoulder. The flowchart which shows the flow of the motion analysis processing based on the neural network which concerns on 2nd Embodiment. The figure which shows the person area image corresponding to the depth image of FIG. The figure which shows the cut-out image created from the depth image of FIG. 5 using the person area image of FIG. 16A. The figure explaining the example of the setting method of the golfer area with respect to the pigeon tail. A graph of the estimated value of the movement of the center of gravity of the body in the X direction based on the neural network. A graph of the estimated value of the movement of the center of gravity of the body in the Y direction based on the neural network. Graphs of the locus of the center of gravity of the body in a plan view based on the estimates of FIGS. 18A and 18B. The figure explaining the example of the setting method of the shoulder area with respect to the center of the shoulder. Graph of estimated shoulder rotation angle based on neural network. A flowchart showing the flow of learning processing of a neural network for deriving the amount of movement of the center of gravity of the body. A flowchart showing the flow of the learning process of the neural network for deriving the rotation angle of the shoulder.

Hereinafter, the motion analysis device, the method and the program, and the model building device, the method and the program according to some embodiments of the present invention will be described with reference to the drawings. The following embodiment will be described by taking a scene of analyzing a golf swing as an example.

<1. First Embodiment>
<1-1. Overview of motion analysis system>
1 and 2 show an overall configuration diagram of an operation analysis system 100 including an operation analysis device 1 according to the present embodiment. The motion analysis system 100 is a system for photographing the swing motion of the golf club 5 by the golfer 7 as a moving image and analyzing the swing motion based on the moving image. The above shooting is performed by the distance image sensor 2. The motion analysis device 1 constitutes a motion analysis system 100 together with the range image sensor 2, and analyzes the swing motion by analyzing the image data including the depth image acquired by the range image sensor 2. The result of the analysis by the motion analysis device 1 is used for various purposes such as fitting of a golf club 5 suitable for the golfer 7, improvement of the form of the golfer 7, development of golf equipment, and the like.

The analysis of the swing motion is performed based on the neural network 8 (see FIG. 3) that inputs the depth image. The neural network 8 is a model for analyzing a swing motion, and outputs an motion value that quantitatively represents the swing motion. The neural network 8 is constructed by pre-learning. Hereinafter, the details of each part of the motion analysis system 100 will be described, and then the model configuration of the neural network 8, the motion analysis method based on the neural network 8, and the learning method of the neural network 8 will be described in order.

<1-2. Details of each part>
<1-2-1. Distance image sensor>
The distance image sensor 2 is a three-dimensional measurement camera having a distance measuring function of taking a picture of a golfer 7 trying to hit a golf club 5 as a two-dimensional image and measuring the distance to a subject. Therefore, the distance image sensor 2 can output a depth image together with the two-dimensional image. The two-dimensional image referred to here is an image obtained by projecting an image of the shooting space into a plane orthogonal to the optical axis of the camera. The depth image is an image in which the depth data (depth data) of the subject in the optical axis direction of the camera is assigned to pixels within the same imaging range as the two-dimensional image.

The distance image sensor 2 used in the present embodiment captures a two-dimensional image as an infrared image (hereinafter referred to as an IR image). Further, the depth image can be obtained by a method such as a time-of-flight method using infrared rays or a dot pattern projection method. Therefore, as shown in FIG. 1, the distance image sensor 2 has an IR light emitting unit 21 that emits infrared rays toward the front and an IR that receives infrared rays that are emitted from the IR light emitting unit 21 and reflected on the subject and returned. It has a light receiving unit 22. The IR light receiving unit 22 is a camera having an optical system, an image pickup device, and the like. In the dot pattern projection method, the infrared dot pattern emitted from the IR light emitting unit 21 is read by the IR light receiving unit 22, the dot pattern is detected by image processing inside the distance image sensor 2, and the depth is calculated based on this. To. In the present embodiment, the IR light emitting unit 21 and the IR light receiving unit 22 are housed in the same housing 20 and are arranged in front of the housing 20. In the present embodiment, the distance image sensor 2 is installed in front of the golfer 7 in order to photograph the golfer 7 from the front side, and the IR light emitting unit 21 and the IR light receiving unit 22 are directed toward the golfer 7.

In addition to the CPU 23 that controls the entire operation of the distance image sensor 2, the distance image sensor 2 has a built-in memory 24 that stores captured image data at least temporarily. The control program that controls the operation of the distance image sensor 2 is stored in the memory 24. Further, the distance image sensor 2 also has a built-in communication unit 25, and the communication unit 25 transfers the captured image data to an external device such as the motion analysis device 1 via a wired or wireless communication line 17. Can be output. In the present embodiment, the CPU 23 and the memory 24 are also housed in the housing 20 together with the IR light emitting unit 21 and the IR light receiving unit 22. It should be noted that the transfer of image data to the motion analysis device 1 does not necessarily have to be performed via the communication unit 25. For example, if the memory 24 is removable, it can be removed from the housing 20 and inserted into the reader of the motion analysis device 1 (corresponding to the communication unit 15 described later) to obtain image data in the motion analysis device 1. It can be read.

<1-2-2. Motion analysis device>
The configuration of the motion analysis device 1 will be described with reference to FIG. 2. The motion analysis device 1 is a general-purpose computer as hardware, and is realized as, for example, a desktop computer, a notebook computer, a tablet computer, or a smartphone. The motion analysis device 1 is manufactured by installing the motion analysis program 3 on a general-purpose computer from a computer-readable recording medium 30 such as a CD-ROM or a USB memory, or via a network such as the Internet. The motion analysis program 3 is software for analyzing the golf swing based on the image data sent from the distance image sensor 2, and causes the motion analysis device 1 to execute the motion described later.

The motion analysis device 1 includes a display unit 11, an input unit 12, a storage unit 13, a control unit 14, and a communication unit 15. These units 11 to 15 are connected to each other via the bus line 16 and can communicate with each other. The display unit 11 can be configured as a liquid crystal display or the like, and displays the result of golf swing analysis or the like to the user. The term "user" as used herein is a general term for golfers 7 themselves, their instructors, developers of golf equipment, and other persons who require the results of golf swing analysis. The input unit 12 can be composed of a mouse, a keyboard, a touch panel, or the like, and receives an operation from the user on the motion analysis device 1.

The storage unit 13 can be configured by a hard disk or the like. In addition to storing the motion analysis program 3, the storage unit 13 stores image data sent from the distance image sensor 2. Further, in the storage unit 13, information that is learned by the learning process described later and defines the neural network 8 used in the motion analysis process described later is stored. The control unit 14 can be composed of a CPU, a ROM, a RAM, and the like. By reading and executing the motion analysis program 3 in the storage unit 13, the control unit 14 virtually serves as the first acquisition unit 14a, the second acquisition unit 14b, the derivation unit 14c, the learning unit 14d, and the display control unit 14e. Operate. Details of the operation of each part 14a to 14e will be described later. The communication unit 15 functions as a communication interface for receiving data from an external device such as the distance image sensor 2 via the communication line 17.

<1-3. Neural network model configuration>
Next, the model configuration of the neural network 8 used in the motion analysis process described later will be described with reference to FIG. As described above, the neural network 8 is a network that inputs a depth image and outputs an operation value that quantitatively represents a swing operation. In the present embodiment, the neural network 8 quantitatively analyzes the hip rotation motion of the golfer 7 during the swing motion, and more specifically, the hip rotation angle is quantitatively derived.

As shown in FIG. 3, the neural network 8 according to the present embodiment is a convolutional neural network, and has an identification unit 81 and a feature extraction unit 82 connected to the input side of the identification unit 81. The identification unit 81 is a multi-layer perceptron. The feature extraction unit 82 is formed in a multilayer structure having a first intermediate layer 83 and a second intermediate layer 84, and extracts features of a depth image. The first intermediate layer 83 has a convolution layer 83A, a pooling layer 83B and a normalization layer 83C, and similarly, the second intermediate layer 84 also has a convolution layer 84A, a pooling layer 84B and a normalization layer 84C. Therefore, in the neural network 8, the convolution, pooling, and normalization processes are repeated a plurality of times by passing through the plurality of intermediate layers 83 and 84.

Further, the neural network 8 according to the present embodiment has a dropout layer 85 on the input side of the feature extraction unit 82. The general dropout layer is a layer in which units of a multi-layer network are stochastically selected and learned, and the units other than the selected units are invalidated, that is, treated as if they do not exist. As a result, the degree of freedom of the network can be forcibly reduced during learning, and overfitting can be avoided. On the other hand, the dropout layer 85 according to the present embodiment is a layer for removing a region that hinders the analysis in the input image to be analyzed. The input of the neural network 8 is a depth image. In the present embodiment, in the depth image input to the dropout layer 85, as will be described later, the pixel values of the pixels corresponding to the background and arm regions that are known to interfere with the analysis in advance are predetermined pixel values (this). In the embodiment, it is set to "0"). The dropout layer 85 is a layer that invalidates the unit of pixels having a predetermined pixel value "0" in the input depth image (64 x 32 pixels in the example of FIG. 3), and is an output image (output image (64 × 32 pixels)). In the example of FIG. 3, 64 × 32 pixels) is output. That is, the dropout layer 85 does not probabilistically select the units to be invalidated, but selects and invalidates the region that is known to interfere with the analysis in advance. Specifically, the dropout layer 85 is a layer in which the weight is determined using a portion other than "0", and the weight of the filter response value including "0" is increased to compensate.

After that, the output image from the dropout layer 85 is subjected to convolution processing by a large number of weight filters G ₁ , G ₂ , ..., GN ( _N is an integer of 2 or more. In the example of FIG. 3, , N = 16). As a result, _N feature maps A ₁ , A ₂ , ..., AN are generated. The feature maps A ₁ , A ₂ , ..., AN are input to the _N units constituting the convolution layer 83A, respectively. Specifically, the inner product of the input image and the weight filter G _n is repeatedly calculated by raster scan, and the weight filter G _n is convoluted into the input image to calculate the feature map A _n (n =). 1, 2, ..., N). In the example of FIG. 3, the weight filter G _n is 5 × 5 pixels, and the size of the feature map _An is 60 × 28 pixels.

The weight filters G ₁ , G ₂ , ..., _GN , also called a filter kernel, are finer images (or an array of values) than a depth image, and each has a certain pattern (or an array of values) included in the input image. It is a filter for detecting and emphasizing (feature). The feature map A _n is an image (or an array of values) in which the features of the weight filter G _n are emphasized in the input image in response to the features of the weight filter G _n . Here, the size of the input image is H1 × H2 pixels, and the pixels of the input image are represented by indexes (i, j) (i = 0,1, ..., H1-1, j = 0,1, ... ·, H2-1), the pixel value of the pixel (i, j) of the input image is expressed as x _{i, j} . Further, the size of the weight filter G _n is set to H × H pixels, and the pixels of the weight filter G _n are represented by indexes (p, q) (p = 0,1, ..., H-1, q = 0,1). , ..., H-1), the pixel values of the pixels (p, q) of the weight filter G _n are expressed as h _{p, q} . At this time, the pixel values t _{i, j of} the feature map An convoluted in the pixels (i, j) of the input image can be calculated as follows.

Next, pooling processing is executed for each of the feature maps A ₁ , A ₂ , ..., AN, and as a result, _N feature maps B ₁ , B ₂ , ..., _BN are generated. The map. The feature maps B ₁ , B ₂ , ..., BN are input to the _N units constituting the pooling layer 83B, respectively. The pooling process is a process of converting the feature map A _n into a new feature map B _n by outputting a response value representing a small area included in the feature map A _n . By this pooling process, the size of the feature map _Ann can be reduced. Further, in the pooling process, since a large number of pixel values included in the small area of the input image A _n are aggregated into the response value, the position sensitivity of the output image B _n is slightly lowered. Therefore, even if the position of the feature to be detected changes slightly in the depth image, the change can be absorbed and the output image B _n after the pooling process can be brought close to a constant value.

More specifically, in the pooling process, the input image _Ann is divided into small areas, and one pixel value as a response value is determined based on the pixel values included in each small area. In the example of FIG. 3, the feature map _Ann is divided into small areas of 2 × 2 pixels and reduced to 1/2 the size. Therefore, the size of the feature map B _n is 30 × 14 pixels in the example of FIG. Various methods for determining the response value can be considered. For example, the average value of the pixel values in the small area can be used as the response value (mean pooling), or the maximum value can be used as the response value (maximum pooling). Further, as in the method called Lp pooling, the response value can be determined so as to increase the influence of the large pixel value in the small region and leave the influence of the small pixel value to some extent.

Subsequently, each of the feature maps B ₁ , B ₂ , ..., BN is normalized, and as a result, _N feature maps C ₁ , C ₂ , ..., _CN are generated. The feature maps C ₁ , C ₂ , ..., CN are input to the _N units constituting the normalization layer 83C, respectively. The normalization referred to here is local contrast normalization, and in the present embodiment, subtraction normalization is executed. By this normalization, a pixel value having a large change with respect to the pixel value in the peripheral portion is detected on the input image B _n , and the pixel value is emphasized on the output image C _n .

From the above, the processing in the first intermediate layer 83 is completed, the process proceeds to the processing in the second intermediate layer 84, and the second convolution processing is executed. In the second _convolution process, a predetermined number of feature maps C ₁ , C ₂ , ..., CN output from the first intermediate layer 83, and four feature maps in this embodiment. Are randomly selected to form one set, and M sets of such sets are created (in the example of FIG. 3, M = 256). Further, a large number of weight filters L ₁ , L ₂ , ..., L _R are newly prepared (R is an integer of 2 or more. In this embodiment, R = 16), and these weight filters L ₁ , L are prepared. From ₂ , ..., _LR , the same number of feature maps as the number of feature maps included in one set of feature maps, that is, in this embodiment, four weight filters are randomly selected to form one set. Create M sets like this. The set of weight filters is made to correspond one-to-one with the set of feature maps selected in this way, and convolution is performed. More specifically, the four feature maps included in a certain set and the four weight filters included in the set corresponding to the set have a one-to-one correspondence, and convolution is performed according to this correspondence. In the present embodiment, the weight filter L _r here is 5 × 5 pixels (r = 1, 2, ..., R). The method of convolution is the same as the processing in the first intermediate layer 83. Then, the feature maps in which the weight filters L _r are convoluted are averaged for each set, and as a result, _M feature maps D ₁ , D ₂ , ..., DM are generated. The feature maps D ₁ , D ₂ , ..., DM are input to the _M units constituting the convolution layer 84A, respectively. The size (m = 1, 2, ..., M) of the feature map D _m is 26 × 10 pixels in the example of FIG.

After that, the same pooling process as the process in the first intermediate layer 83 is executed for each of the feature maps D ₁ , D ₂ , ..., D _M , and as a result, M feature maps E ₁ , E are executed. ₂ , ..., _EM is generated. The feature maps E ₁ , E ₂ , ..., EM are input to the _M units constituting the pooling layer 84B, respectively. Subsequently, normalization similar to the processing in the first intermediate layer 83 is executed for each of the feature maps E ₁ , E ₂ , ..., EM, and as a result, _M feature maps F ₁ , ... F ₂ , ..., _FM are generated. The feature maps F ₁ , F ₂ , ..., FM are input to the _M units constituting the normalization layer 84C, respectively. The size of the feature maps E _m and F _m is 13 × 5 pixels in the example of FIG. The feature maps F ₁ , F ₂ , ..., FM are the final output _images of the feature extraction unit 82 and are input to the identification unit 81.

The identification unit 81 has an input layer 86, an intermediate layer 87, and an output layer 88, and these layers 86 to 88 constitute a fully connected layer. All the pixel values U ₁ , U ₂ , ..., U I 1 included in the output _images F ₁ , F ₂ , ..., FM are input to each of the _I1 input units constituting the input layer 86, respectively. To. In the example of FIG. 3, I1 = 13 × 5 (number of pixels) × 256 (number of images) = 16640.

The intermediate layer 87 is composed of two intermediate units of I, and is set to I2 = 1000 in the example of FIG. In the i-th intermediate unit, the values V _i calculated based on the values U ₁ , U ₂ , ..., U _I 1 included in the input unit are input (i = 1, 2, ..., I2). V _i is calculated according to the following formula. In the following equation, u _{i, 1} , u _{i, 2} , ..., u _{i, I 1} are weighting coefficients, and b _i is a bias.

The output layer 88 is composed of I3 output units, and in this embodiment, I3 = 1. A value W ₁ calculated based on the values V ₁ , V ₂ , ..., V _I 2 included in the intermediate unit is input to the output unit, and in the present embodiment, the rotation angle of the golfer 7's waist is quantified. The operation value W ₁ to be represented is input. W ₁ is calculated according to the following formula. In the following equation, v ₁ , v ₂ , ..., V _I2 are weighting factors, and b is a bias.

<1-4. Motion analysis processing based on neural network>
Hereinafter, the motion analysis process of the golf swing will be described with reference to FIG. 4. As described above, in the present embodiment, the motion value W ₁ that quantitatively represents the rotation angle of the waist of the golfer 7 during the swing motion is derived based on the neural network 8. The image data to be analyzed is a moving image. Therefore, hereinafter, the IR image and the depth image may be referred to as an IR frame and a depth frame, respectively, and may be simply referred to as a frame.

As a preparation for executing the motion analysis process, first, the golfer 7 is made to try out the golf club 5, and the state is photographed as a moving image by the distance image sensor 2. The time-series IR frames and depth frames captured by the distance image sensor 2 are sent from the distance image sensor 2 to the motion analysis device 1. On the motion analysis device 1 side, the first acquisition unit 14a acquires time-series IR frames and depth frames from the distance image sensor 2 and stores them in the storage unit 13 (step S1).

Further, in step S1, the distance image sensor 2 measures time-series skeleton data representing the skeleton of the golfer 7's body, and the skeleton data is sent from the distance image sensor 2 to the motion analysis device 1. The skeleton data is data representing the positions (three-dimensional coordinates) of the main joints of the human body, and can be derived from the depth frame. Kinect (registered trademark), which is one of the distance image sensors, has a function of deriving skeleton data from a depth frame and outputting it together with the depth frame. The first acquisition unit 14a also acquires time-series skeleton data from the distance image sensor 2 and stores it in the storage unit 13. When the distance image sensor 2 that does not output the skeleton data is used, the first acquisition unit 14a may acquire this from the depth frame. Specifically, the first acquisition unit 14a reads out the time-series depth frames stored in the storage unit 13 during the swing operation, and based on these frames, the skeleton data at each timing during the swing operation. To get. Kinect® provides a library for deriving skeleton data from depth images, which can be used to obtain skeleton data.

Subsequently, the derivation unit 14c normalizes the depth frame at each timing during the swing operation (step S2). The normalization referred to here is a process of scaling the gradation of the depth frame according to the depth of the subject including the golfer 7. Specifically, the derivation unit 14c reads out a time-series depth frame stored in the storage unit 13 during the swing operation. At this time, the depth data, which is the pixel value of the depth frame, has a gradation according to the standard of the distance image sensor 2, and in this embodiment, 16 bits are assigned to one pixel, and each pixel is 0. It takes a pixel value of ~ 65535. Further, the imaging range in the depth direction of the distance image sensor 2 is also defined by the standard of the distance image sensor 2. On the other hand, in order to estimate the rotation angle of the waist of the golfer 7, depth data other than the golfer 7 is not particularly required. Therefore, the gradation of the depth frame is scale-converted so that the pixel value included in the area where the golfer 7 is captured from the depth frame (hereinafter referred to as the person area) takes a value in the range of 0 to 65535.

The installation position of the distance image sensor 2 and the standing position of the golfer 7 are fixed. Therefore, the range of values that the depth data can take in the person area (hereinafter referred to as the person depth range) is set in advance. In the present embodiment, the derivation unit 14c scale-converts the pixel value (depth data) of each pixel in the depth frame according to the following equation. However, the person depth range is min _z to max _z , and z on the right side is the pixel value of each pixel in the depth frame. Z on the left side is a pixel value after conversion.

The above scale conversion is a process of extracting a region having a pixel value within the person depth range from the depth frame. FIG. 5 is a depth frame at a specific timing after the above scale conversion. As can be seen from the figure, in the depth frame after the scale conversion, the pixel value "0" (black) is given to the region that mainly captures the region other than the golfer 7, that is, the background region. As a result, the person area is extracted by the above scale conversion. Note that FIG. 6 shows skeleton data measured by Kinect (registered trademark) corresponding to the timing of FIG.

Subsequently, the derivation unit 14c is based on the skeleton data acquired in step S1 from the depth frame normalized in step S2 to the region near the waist of the golfer 7 at each timing during the swing operation (hereinafter, The waist region) is extracted (step S3). Specifically, the derivation unit 14c acquires the coordinates of the waist in the depth frame from the skeleton data. Then, a region having a predetermined size based on the coordinates of the waist is cut out from the depth frame as a waist region. FIG. 7 shows an image of a time-series waist region during a swing motion cut out from a depth frame as shown in FIG. In this example, the image of FIG. 5 is 512 × 214 pixels, and the image of FIG. 7 is an image of 128 × 64 pixels centered on or substantially centered on the waist.

The movement of the hips that attracts attention is independent of the movements of other parts such as the shoulders, legs, and arms. Therefore, learning a neural network based on an image showing the entire human body may reduce the accuracy of detecting features corresponding to the appearance of the waist. Also, if the size of the image is too large, it may be difficult to analyze by the neural network. In step S3, the waist region is extracted from the depth frame to be analyzed so that the characteristics of the movement of the waist can be accurately detected based on the neural network.

Subsequently, the derivation unit 14c is a region representing the arm of the golfer 7 at each timing during the swing motion from the image of the waist region acquired in step S3 based on the skeleton data acquired in step S1 (hereinafter,). The arm region) is removed (step S4). Specifically, the derivation unit 14c extracts the arm region from the image of the waist region and sets the pixel value of the pixel included in the arm region to a predetermined pixel value (0 in this embodiment). The extraction of the arm region is performed based on the three-dimensional coordinates of the three joints (in this embodiment, the left shoulder, the right shoulder and the wrist of the left hand) included in the skeleton data. More specifically, the derivation unit 14c derives a plane K passing through the three-dimensional coordinates of these three joints, and derives a perpendicular distance d from an arbitrary point (pixel) in the image of the waist region to the plane K. do. The three-dimensional coordinates of any point (pixel) are acquired from the image of the waist region, which is a depth image. Further, the plane K and the perpendicular distance d can be calculated by geometric calculation. Next, the derivation unit 14c determines whether or not the perpendicular distance d is included in the predetermined range, and if it is included in the predetermined range, the arm is a point (pixel) corresponding to the perpendicular distance d. It is determined that it is included in the area, and if it is out of the predetermined range, it is determined that it is not included in the arm area. FIG. 8 shows an image in which the arm region is removed from the image of the waist region in the time series during the swing motion of FIG. 7. In the figure, the pixel value of the pixel in the arm region is converted to “0” (black).

Step S4 is a process for invalidating the arm region in the dropout layer 85 when an image is input to the neural network 8 in step S5 described later. That is, step S4 is a step of removing a region that hinders the analysis from the depth frame to be analyzed based on the neural network 8. In this embodiment, also in step S2, the region that hinders the analysis, that is, the background region is removed.

In the following step S5, the derivation unit 14c sequentially inputs the images of the waist region of the time series during the swing operation acquired in step S4 to the neural network 8. As a result, the output unit of the neural network 8 sequentially outputs the motion value W ₁ that quantitatively represents the swing motion, and in the present embodiment, the rotation angle W ₁ of the waist of the golfer 7. However, in the present embodiment, the 128 × 64 pixel image acquired up to step S4 is compressed to a size of 64 × 32 pixels and then input to the neural network 8.

Subsequently, the derivation unit 14c smoothes and interpolates the time _- series data of the rotation angle W1 derived in step S5 (step S6). FIG. 9 shows an example in which spline interpolation is performed by smoothing by a moving average of 5 points and converting data at 33 ms intervals into data at 1 ms intervals. As a result, the smoothed and interpolated time-series data of the smooth rotation angle W ₁ is acquired.

After that, the display control unit 14e displays the rotation angle W1 in the time series during the swing operation derived in step S6, the time series change thereof, and these graphs as shown in FIG. ₉ on the display unit 11 (step). S7). As a result, the user can grasp the operation of the rotation of the waist of the golfer 7.

<1-5. Neural network learning method ＞
Next, the learning method of the neural network 8 will be described with reference to FIG. In the following, a learning data set for constructing the neural network 8 will be described, and then a flow of learning processing based on the data set will be described.

<1-5-1. Data set for training ＞
The training data set is a pair of data of the depth frame and skeleton data captured by the distance image sensor 2 and the rotation angle (true value) of the golfer 7's waist at the timing of capturing the depth frame and skeleton data. Yes, many such training datasets are collected. The rotation angle (true value) included in the training data set is a teacher signal at the time of training of the neural network 8.

In the present embodiment, the teacher signal regarding the rotation angle of the golfer 7's waist is acquired by the angular velocity sensor 4 (see FIGS. 1 and 2) attached to the golfer 7's waist. The angular velocity data measured by the angular velocity sensor 4 is output from the angular velocity sensor 4 to the motion analysis device 1 via a wired or wireless communication line.

The teacher signal is required in the learning scene of the neural network 8, but is not particularly required in the motion analysis scene based on the neural network 8. Therefore, after the neural network 8 is learned and stored in the storage unit 13, the angular velocity sensor 4 can be omitted from the motion analysis system 100. Further, the motion analysis system 100 and the model construction system for constructing the model of the neural network 8 can be realized by different hardware. That is, the motion analysis system 100 may acquire the neural network 8 learned by another system and use it for the analysis. However, the motion analysis system 100 according to the present embodiment also serves as a model construction system for constructing a model of the neural network 8.

<1-5-2. Flow of learning process>
Next, the learning process of the neural network 8 will be described with reference to FIG. First, in order to acquire a learning data set, a golfer 7 having an angular velocity sensor 4 attached to the waist is made to try out a golf club 5, and the state is photographed as a moving image by a distance image sensor 2. At this time, preferably, a large number of swing motions are performed by a plurality of golfers 7 in order to enhance the learning effect. Then, similarly to step S1, the first acquisition unit 14a acquires the time-series IR frame, depth frame, and skeleton data sent from the distance image sensor 2 and stores them in the storage unit 13 (step S21). .. Further, in step S21, the time-series angular velocity data measured by the angular velocity sensor 4 during the swing operation is also transmitted to the motion analysis device 1. Then, the second acquisition unit 14b acquires the angular velocity data of this time series, converts it into data representing the rotation angle of the waist in the time series (rotation angle data of the waist), and then stores it in the storage unit 13. .. The storage unit 13 stores IR frames, depth frames, skeleton data, and rotation angle data corresponding to a large number of swing operations. At this time, the IR frame, the depth frame, and the skeleton data are synchronized with the rotation angle data, and the data at the same timing are associated with each other and stored in the storage unit 13.

Subsequently, the learning unit 14d executes steps S22 to S24 for the time-series depth frames corresponding to the multiple swing motions acquired in step S21. Steps S22 to S24 are steps of cutting out an image of the waist region as shown in FIG. 8 to be input to the neural network 8 from the depth frame acquired in step S21. Since steps S22 to S24 are the same steps as steps S2 to S4 described above, detailed description thereof will be omitted here.

In the following step S25, the learning unit 14d trains the neural network 8 using the image of the waist region acquired in step S24 as an input and the rotation angle data of the waist acquired in step S21 as a teacher signal. More specifically, the learning unit 14d inputs an image of the waist region to the current neural network 8, acquires a rotation angle W ₁ as an output value, and has an error between the rotation angle W ₁ and the rotation angle data of the waist. Update the parameters of the neural network 8 so as to minimize. The parameters to be learned here are the above _- mentioned weight filters G ₁ , G ₂ , ..., GN, L ₁ , _{L 2} , ..., LR, and weight coefficients u _{i, 1} , u. _{i, 2} , ..., u _{i, I1} , weighting factors v ₁ , v ₂ , ..., v _I2 , bias b _i , b, etc. Then, in this way, the neural network 8 is optimized while applying the learning data sets one after another. Since various supervised learning methods of the neural network 8 are known, detailed description thereof will be omitted here, but for example, a stochastic gradient descent method using an error backpropagation method can be used. With the above, the learning process is completed.

<2. 2nd Embodiment>
11 and 12 show an overall configuration diagram of the motion analysis system 101 according to the present embodiment. As is clear from comparing these figures with FIGS. 1 and 2, the motion analysis system 101 according to the second embodiment is common to the motion analysis system 100 according to the first embodiment in many respects. Hereinafter, for the sake of simplicity, the same reference numerals will be given to the elements common to the first embodiment, and the differences from the first embodiment will be mainly described.

The main differences between the first and second embodiments will be described. In the first embodiment, the rotation motion of the golfer 7's waist during the swing motion is quantitatively analyzed, whereas in the second embodiment, the rotation motion is quantitatively analyzed. In addition to this, the weight shift of the golfer 7 during the swing motion and the rotational motion of the shoulder are also quantitatively analyzed. In the present embodiment, the neural network 108 shown in FIG. 13 quantitatively derives the amount of movement of the center of gravity, which represents the weight movement of the body of the golfer 7, as the movement value that quantitatively represents the movement of the golfer 7. The amount of movement of the center of gravity is an operation value representing the position of the center of gravity. Further, the neural network 208 shown in FIG. 14 quantitatively derives the rotation angle of the shoulder as an operation value that quantitatively represents the operation of the golfer 7. Similar to the neural network 8, these neural networks 108 and 208 also input a depth image. Hereinafter, the model configuration of the neural networks 108 and 208, the motion analysis method based on the neural networks 108 and 208, and the learning method of the neural networks 108 and 208 will be described in order.

<2-1. Neural network model configuration>
As is clear from comparing FIGS. 3 and 13, the neural networks 8 and 108 have similar configurations. As for the main difference between the two, in the neural network 108, the amount of movement of the center of gravity of the golfer 7's body is evaluated two-dimensionally, so that the output layer 88 is composed of two output units. An operation value W ₁₀₁ that quantitatively represents the amount of movement of the center of gravity in the X direction (direction of the golf ball's flight line) is input to one output unit, and the Y direction (from the back of the golfer 7) is input to the other output unit. An operation value W ₁₀₂ that quantitatively represents the amount of movement of the center of gravity in the direction toward the abdomen is input. W ₁₀₁ and W ₁₀₂ are calculated according to the following formula. Note that v _1,1 , v _1,2 , ..., v _{1, I2} and v _2,1 , v _2,2 , ..., v _{2, I2} in the following equation are weighting coefficients. b ₁₀₁ and b ₁₀₂ are biases.

Further, although the neural network 108 does not have the dropout layer 85, it can of course be configured to have the dropout layer 85. Further, the size of the feature map, the number of units forming the layer, and the like are appropriately changed.

Further, as is clear from comparing FIGS. 3 and 14, the neural networks 8 and 208 also have a similar configuration. The main difference between the two is that the neural network 208 does not have a dropout layer 85, but of course the neural network 208 can also be configured to have a dropout layer 85. The output layer 88 of the neural network 208 is composed of one output unit. An operation value W ₂₀₁ that quantitatively represents the rotation angle of the shoulder of the golfer 7 is input to the output unit. W ₂₀₁ is calculated according to the equation of equation 3 in the same manner as W ₁ . In the description of the neural networks 8, 108, 208 for deriving the rotation angle of the waist, the rotation angle of the shoulder, and the amount of movement of the center of gravity, the same symbols are often used, such as Vi and bi. Since the analysis target that is finally output is different and the input image data is also different, the specific values represented by each symbol are naturally different.

The configurations of the neural networks 8, 108, 208 disclosed in the first embodiment and the second embodiment are examples. Therefore, the number of units forming each layer of each neural network 8, 108, 208, the number and size of feature maps, the number and size of filters, and the like can be appropriately changed.

<2-2. Motion analysis processing based on neural network>
Next, the operation analysis process of the golf swing according to the second embodiment will be described with reference to FIG. As described above, here, the motion value W ₁ that quantitatively represents the rotation angle of the golfer 7 during the swing motion is derived based on the neural network 8, and the swing motion is based on the neural network 108. The operating values W _{101 and} W ₁₀₂ that quantitatively represent the amount of movement of the center of gravity of the golfer 7 in the golfer 7 are derived. Further, based on the neural network 208, an operation value W ₂₀₁ that quantitatively represents the rotation angle of the waist of the golfer 7 during the swing operation is also derived. Hereinafter, the process of FIG. 15 will be described, but the process of FIG. 15 includes the process of FIG. 4, and the new steps S102, S103, S105, S106, S203, S205, and S206 are also part of the process of FIG. Similar. Therefore, in the following, the differences between the two processes will be mainly described with reference to the description of FIG.

First, step S1 and step S2 are executed in the same manner as in the process of FIG. That is, the distance image sensor 2 photographs the golfer 7 swinging the golf club 5, acquires time-series IR frames, depth frames, and skeleton data, and stores them in the storage unit 13 of the motion analysis device 1. .. In step S1 of the present embodiment, the distance image sensor 2 creates a person area image representing the range occupied by the golfer 7's body within the imaging range of the depth frame, and this person area image operates from the distance image sensor 2. It is sent to the analyzer 1. The person area image is an image that binaryally distinguishes the area occupied by the person from the other area, and is a binary image as shown in FIG. 16A. The person area image can be derived from the depth frame, and Kinect (registered trademark), which is one of the distance image sensors, has a function of deriving the person area image from the depth frame and outputting it together with the depth frame. There is. The first acquisition unit 14a also acquires a time-series person area image from the distance image sensor 2 and stores it in the storage unit 13. When the distance image sensor 2 is used so that the person area image is not output, the first acquisition unit 14a may acquire the distance image sensor 2 from the depth frame. Specifically, the first acquisition unit 14a reads out the time-series depth frames stored in the storage unit 13 during the swing operation, and based on these frames, the person area at each timing during the swing operation. Create an image.

In the following step S2, the depth frame is normalized. Then, when step S2 is completed, steps S3 to S6 are executed in the same manner as in the process of FIG. 4, and as _a result, the rotation angle W1 of the waist in the time series is calculated, and smoothing and spline interpolation are executed for this. To. On the other hand, in parallel with steps S3 to S6, steps S102, S103, S105, S106 for deriving the amount of movement of the center of gravity of the body in time series W _101, W ₁₀₂ and the rotation angle W ₂₀₁ of the shoulder in time series are derived. Steps S203, S205, and S206 are executed. It should be noted that steps S3 to S6, steps S102, S103, S105, and S106 and steps S203, S205, and S206 do not need to be executed in parallel, and may be executed in an appropriate order, for example.

First, steps S102, S103, S105, and S106 for deriving the center of gravity movement amounts W _{101 and} W ₁₀₂ will be described. In step S102, the derivation unit 14c creates an image (hereinafter referred to as a cut image) in which only the area occupied by the golfer 7 is cut out from the depth frame normalized in step S2 at each timing during the swing operation. As described above, the normalized depth frame is an image mainly containing information on the depth of the golfer 7, but strictly speaking, it also includes information on the depth of the ground near the feet of the golfer 7. In step S102, in order to remove such information on the depth of the ground, only the area occupied by the golfer 7 is cut out from the depth frame normalized in step S2 based on the person area image acquired in step S1. FIG. 16B shows a cropped image created from the normalized depth frame shown in FIG. 5 using the person region image of FIG. 16A.

In the following step S103, the derivation unit 14c extracts a region in the vicinity of the golfer 7 (hereinafter referred to as a golfer region) from the clipped image based on the skeleton data acquired in step S1. Specifically, the derivation unit 14c acquires the coordinates of the dovetail in the depth frame from the skeleton data. Then, a region having a predetermined size based on the coordinates of the pigeon tail is cut out from the depth frame as a golfer region. The image on the left side of FIG. 17 shows an example of a method of setting a golfer area with reference to the dovetail, and the image on the right side of FIG. 17 shows a golfer cut out from the cut-out image of FIG. 16B according to the example of this setting method. An image of the area is shown.

Information on the ground may not be needed when trying to assess the position of the center of gravity of the body. Therefore, learning a neural network based on an image containing information on the depth of the ground may reduce the accuracy of detecting the center of gravity. Also, if the size of the image is too large, it may be difficult to analyze by the neural network. In steps S102 and S103, the golfer region is extracted from the depth frame to be analyzed so that the position of the center of gravity of the body can be accurately detected based on the neural network.

In the following step S105, the derivation unit 14c sequentially inputs the images of the golfer region of the time series during the swing operation acquired in step S103 to the neural network 108. As a result, the output units of the neural network 108 sequentially output the amount of movement of the center of gravity of the body of the golfer 7 in the time series during the swing operation, W _{101 and} W ₁₀₂ . However, in the present embodiment, the 348 × 192 pixel image acquired up to step S103 is compressed to a size of 116 × 64 pixels by nearest neighbor interpolation or the like, and then input to the neural network 108.

Subsequently, the derivation unit 14c smoothes and interpolates the time-series data of the center of gravity movement amounts W _{101 and} W ₁₀₂ derived in step S105 (step S106). FIGS. 18A and 18B show an example in which the time-series data of the center of gravity movement amounts W _{101 and} W ₁₀₂ are smoothed by a moving average of 5 points, and spline interpolation is performed to convert the data at 33 ms intervals into the data at 1 ms intervals. ing. As a result, time-series data of smooth center of gravity movement amounts W _{101 and} W ₁₀₂ are acquired. FIG. 18C is a graph of the locus of the center of gravity of the body in a plan view created by combining the movement amounts of the center of gravity W _{101 and} W ₁₀₂ in the X direction and the Y direction.

Next, steps S203, S205, and S206 for deriving the shoulder rotation angle W ₂₀₁ will be described. First, the derivation unit 14c is a region near the shoulder of the golfer 7 at each timing during the swing operation (hereinafter, shoulder) from the depth frame normalized in step S2 based on the skeleton data acquired in step S1. Region) is extracted (step S203). Specifically, the derivation unit 14c acquires the coordinates of both shoulders in the depth frame from the skeleton data, and specifies the coordinates of the center of both shoulders (hereinafter, the center of the shoulder). Then, a region having a predetermined size based on the coordinates of the center of the shoulder is cut out from the depth frame as a shoulder region. The image on the left side of FIG. 19 shows an example of how to set the shoulder region with respect to the center of the shoulder, and the image on the right side of FIG. 19 is from the clipped image of FIG. 16B or according to the example of this setting method. The image of the shoulder region cut out from the depth frame such as is shown.

The movement of the shoulder that is noticed is independent of the movement of other parts such as the waist, legs, and arms. Therefore, learning a neural network based on an image showing the entire human body may reduce the accuracy of detecting features corresponding to the appearance of the shoulder. Also, if the size of the image is too large, it may be difficult to analyze by the neural network. In step S203, the shoulder region is extracted from the depth frame to be analyzed so that the characteristics of the shoulder movement can be accurately detected based on the neural network.

In the following step S205, the derivation unit 14c sequentially inputs the images of the shoulder region of the time series during the swing operation acquired in step S203 to the neural network 208. As a result, the rotation angle W ₂₀₁ of the shoulder of the golfer 7 is sequentially output from the output unit of the neural network 208. However, in the present embodiment, the 128 × 64 pixel image acquired up to step S203 is compressed to a size of 64 × 32 pixels by nearest neighbor interpolation or the like, and then input to the neural network 208.

Subsequently, the derivation unit 14c smoothes and interpolates the time-series data of the shoulder rotation angle W ₂₀₁ derived in step S205 (step S206). FIG. 20 shows an example in which the time-series data of the shoulder rotation angle W ₂₀₁ is smoothed by a moving average of 5 points, and spline interpolation is performed to convert the data at 33 ms intervals into the data at 1 ms intervals. As a result, time-series data of a smooth shoulder rotation angle W ₂₀₁ is acquired.

After that, the display control unit 14e displays the rotation angle W1 of the waist during the swing operation derived in step S6, the time _- series change thereof, and the graph as shown in FIG. 9 on the display unit 11 (step). S7). Further, in step S7 of the present embodiment, the display control unit 14e shows the time-series body center of gravity movement amounts W _{101 and} W ₁₀₂ and their time-series changes during the swing motion derived in step S106, as well as FIGS. 18A to 18C. A graph as shown is displayed on the display unit 11. Further, the display control unit 14e displays the rotation angle W ₂₀₁ of the shoulder in the time series during the swing operation derived in step S206, the time series change thereof, and the graph as shown in FIG. 20 on the display unit 11. As a result, the user can grasp in detail the movement of the rotation of the waist and shoulders of the golfer 7 and the weight shift of the golfer 7.

<2-3. Neural network learning method ＞
Next, the learning method of the neural networks 108 and 208 will be described with reference to FIGS. 21 and 22. In the following, a learning data set for constructing each of the neural networks 108 and 208 will be described, and a flow of learning processing based on each data set will be described.

<2-3-1. Data set for learning (movement of the center of gravity of the body)>
The training data set for constructing the neural network 108 includes the depth frame, the skeleton data, and the person area image acquired by the distance image sensor 2, and the amount of movement of the center of gravity (true value) of the golfer 7 at the timing when these are acquired. ), And a large number of such training data sets are collected. The amount of movement of the center of gravity (true value) included in the training data set is a teacher signal during training of the neural network 108.

In the present embodiment, the teacher signal regarding the amount of movement of the center of gravity of the golfer 7 is acquired based on the output value (floor reaction force data) of the floor reaction force meter 104 (see FIGS. 11 and 12) installed at the foot of the golfer 7. Will be done. The floor reaction force meter 104 is composed of a pair of left and right force plates 104L and 104R. During the golf swing, the golfer 7 rides on the force plates 104L and 104R. At this time, the left foot of the golfer 7 is positioned on the force plate 104L, and the right foot is positioned on the force plate 104R.

The force plates 104L and 104R each have a plurality of force sensors 121. The force sensors 121 are dispersedly arranged in the plate-shaped case of the force plates 104L and 104R (for example, arranged at the four corners), receive the weight of the golfer 7, and act on the force plates 104L and 104R. Is detected. The force plates 104L and 104R are communication-connected to the motion analysis device 1, and the floor reaction force data detected by the force sensor 121 is transmitted from the force plates 104L and 104R via a wired or wireless communication line. It is output to 1.

After the neural network 108 is learned and stored in the storage unit 13, the floor reaction force meter 104 can be omitted from the motion analysis system 101. Further, the motion analysis system 101 and the model construction system for constructing the model of the neural network 108 can be realized by different hardware.

<2-3-2. Flow of learning process (movement of the center of gravity of the body)>
Next, the learning process of the neural network 108 will be described with reference to FIG. First, in order to acquire a learning data set for constructing the neural network 108, a golfer 7 riding on the floor reaction force meter 104 is made to test-hit the golf club 5, and the state is photographed as a moving image by the distance image sensor 2. do. At this time, preferably, a large number of swing motions are performed by a plurality of golfers 7 in order to enhance the learning effect. Then, as in step S1, the first acquisition unit 14a acquires the time-series IR frame, depth frame, skeleton data, and person area image sent from the distance image sensor 2 and stores them in the storage unit 13. (Step S121). Further, in step S121, the time-series floor reaction force data measured by the floor reaction force meter 104 is also transmitted to the motion analysis device 1. Then, the second acquisition unit 14b acquires the data of the amount of movement of the center of gravity of the golfer 7's body (hereinafter, the center of gravity position data) based on the floor reaction force data of this time series, and stores it in the storage unit 13. At this time, the IR frame, the depth frame, the skeleton data, the person area image, and the center of gravity position data are synchronized, and the data at the same timing are stored in the storage unit 13 in association with each other.

Subsequently, the learning unit 14d executes steps S122, S132, and S133 for the time-series depth frames corresponding to the multiple swing motions acquired in step S121. Steps S122, S132, and S133 are steps for cutting out an image of the golfer region to be input to the neural network 108 from the depth frame acquired in step S121. Since steps S122, S132, and S133 are the same steps as steps S2, S102, and S103 described above, detailed description thereof will be omitted here.

In the following step S135, the learning unit 14d trains the neural network 108 using the image of the golfer region acquired in step S133 as an input and the center of gravity position data acquired in step S121 as a teacher signal. More specifically, the learning unit 14d inputs an image of the golfer region to the current neural network 108, acquires the center of gravity movement amounts W ₁₀₁ and W ₁₀₂ as output values, and obtains the center of gravity movement amounts W ₁₀₁ and W ₁₀₂ . The parameters of the neural network 108 are updated so as to minimize the error from the center of gravity position data. The parameters to be learned here are the above _- mentioned weight filters G ₁ , G ₂ , ..., GN, L ₁ , _{L 2} , ..., LR, and weight coefficients u _{i, 1} , u. _{i, 2} , ..., u _{i, I1} , v _1,1 , v _1,2 , ..., v _{1, I 2} and v 2, ₁ , v 2, ₂ , ..., v _{2, I2} , Bias _bi , b ₁₀₁ , b ₁₀₂ , etc. Then, in this way, the neural network 108 is optimized while applying the training data sets one after another. Since various supervised learning methods of the neural network 108 are known, detailed description thereof will be omitted here, but for example, a stochastic gradient descent method using an error backpropagation method can be used. With the above, the learning process is completed.

<2-3-3. Data set for learning (shoulder rotation angle)>
The training data set for constructing the neural network 208 consists of the depth frame and skeleton data acquired by the distance image sensor 2 and the rotation angle (true value) of the golfer 7's shoulder at the timing when these are acquired. It is a pair of data, and many such training data sets are collected. The rotation angle (true value) of the shoulder included in the training data set becomes a teacher signal at the time of training of the neural network 208.

In the present embodiment, the teacher signal regarding the rotation angle of the shoulder of the golfer 7 is acquired by the angular velocity sensor 204 (see FIGS. 11 and 12) attached to the center of the shoulder of the golfer 7. The angular velocity data measured by the angular velocity sensor 204 is output from the angular velocity sensor 204 to the motion analysis device 1 via a wired or wireless communication line.

After the neural network 208 is learned and stored in the storage unit 13, the angular velocity sensor 204 can be omitted from the motion analysis system 101. Further, the motion analysis system 101 and the model construction system for constructing the model of the neural network 208 can be realized by different hardware.

<2-3-4. Flow of learning process (shoulder rotation angle)>
FIG. 22 is a flowchart showing the flow of the learning process of the neural network 208. First, in order to acquire a training data set for constructing the neural network 208, a golfer 7 having an angular velocity sensor 204 attached to the shoulder is made to test-hit the golf club 5, and the state is photographed as a moving image by the distance image sensor 2. .. At this time, preferably, a large number of swing motions are performed by a plurality of golfers 7 in order to enhance the learning effect. Then, similarly to step S1, the first acquisition unit 14a acquires the time-series IR frame, depth frame, and skeleton data sent from the distance image sensor 2 and stores them in the storage unit 13 (step S221). .. Further, in step S221, the time-series angular velocity data measured by the angular velocity sensor 204 during the swing operation is also transmitted to the motion analysis device 1. Then, the second acquisition unit 14b acquires the angular velocity data of this time series, converts it into data representing the rotation angle of the shoulder in the time series (rotation angle data of the shoulder), and then stores it in the storage unit 13. .. The storage unit 13 stores IR frames, depth frames, skeleton data, and shoulder rotation angle data corresponding to a large number of swing movements. At this time, the IR frame, the depth frame, and the skeleton data are synchronized with the rotation angle data, and the data at the same timing are associated with each other and stored in the storage unit 13.

Subsequently, the learning unit 14d executes steps S222 and S223 for the time-series depth frames corresponding to the multiple swing motions acquired in step S221. Steps S222 and S223 are steps to cut out an image of the shoulder region as shown in FIG. 19 to be input to the neural network 208 from the depth frame acquired in step S221. Since steps S222 and S223 are the same steps as steps S2 and S203 described above, detailed description thereof will be omitted here.

In the following step S225, the learning unit 14d trains the neural network 208 using the image of the shoulder region acquired in step S223 as an input and the rotation angle data of the shoulder acquired in step S221 as a teacher signal. More specifically, the learning unit 14d inputs an image of the shoulder region to the current neural network 208, acquires the shoulder rotation angle W ₂₀₁ as an output value, and obtains the shoulder rotation angle W ₂₀₁ and the shoulder rotation angle. Update the parameters of the neural network 208 to minimize the error with the data. The parameters to be learned here are the above _- mentioned weight filters G ₁ , G ₂ , ..., GN, L ₁ , _{L 2} , ..., LR, and weight coefficients u _{i, 1} , u. _{i, 2} , ..., u _{i, I1} , weighting factors v ₁ , v ₂ , ..., v _I2 , bias b _i , b, etc. Then, in this way, the neural network 208 is optimized while applying the training data sets one after another. Since various supervised learning methods of the neural network 208 are known, detailed description thereof will be omitted here, but for example, a stochastic gradient descent method using an error backpropagation method can be used. With the above, the learning process is completed.

<3. Modification example>
Although some embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the spirit of the present invention. For example, the following changes are possible. In addition, the gist of the following modifications can be combined as appropriate.

<3-1>
The motion value derived in the motion analysis process is not limited to the rotation angle and position (coordinates) of the region of interest, and may be, for example, a rotation speed, a rotation acceleration, or the like, or may be a speed, an acceleration, or the like. Further, the region of interest is not limited to the waist, shoulders and center of gravity of the golfer 7, and may be the arm, head, or the like of the golfer 7. Further, not only the golf swing but also any movement of any object can be analyzed.

<3-2>
The distance image sensor may capture only a depth image, or may capture a color image instead of an IR image. In the latter case, the distance image sensor may be equipped with a visible light receiving unit (for example, an RGB camera) that receives visible light.

<3-3>
In the first embodiment, the rotation angle of the waist was calculated, and in the second embodiment, the rotation angle of the waist, the rotation angle of the shoulder, and the amount of movement of the center of gravity were calculated. However, it is possible to calculate one or more elements arbitrarily selected from the three of the rotation angle of the waist, the rotation angle of the shoulder, and the amount of movement of the center of gravity. For example, only the rotation angle of the waist and the rotation angle of the shoulder may be calculated, or only the rotation angle of the shoulder and the amount of movement of the center of gravity may be calculated.

1 Motion analysis device (model construction device)
2 Distance image sensor 3 Motion analysis program 4,204 Angular velocity sensor 104 Floor platform 5 Golf club 7 Golfer (object)
8,108,208 Neural network 81 Identification layer 82 Feature extraction unit 83A, 84A Convolution layer 85 Dropout layer 14a First acquisition unit (acquisition unit)
14b 2nd acquisition unit 14c Derivation unit 14d Learning unit 100,101 Motion analysis system

Claims (21)

It is a motion analysis device for analyzing the motion of an object.
An acquisition unit that acquires a depth image of the movement of the object taken by a distance image sensor, and
A neural network that outputs an operation value that quantitatively represents the operation of the object is provided with a derivation unit that derives the operation value by inputting the depth image acquired by the acquisition unit .
The depth image is an image of a golf swing.
Motion analysis device. The operation value is the rotation angle of the golfer's waist.
The motion analysis device according to claim 1 . The acquisition unit further acquires skeleton data representing the skeleton of the human body measured by the distance image sensor.
The derivation unit extracts the waist region near the waist from the depth image based on the skeleton data, and then inputs the image of the waist region to the neural network.
The motion analysis device according to claim 2 . The neural network has a dropout layer that nullifies the arm region representing the golfer's arm from the depth image.
The motion analysis device according to claim 2 or 3 . The operation value is a value representing the weight shift of the golfer.
The motion analysis device according to claim 1 . The acquisition unit further acquires skeleton data representing the skeleton of the human body measured by the distance image sensor.
The derivation unit extracts a golfer region in the vicinity of the golfer from the depth image based on the skeleton data, and then inputs the image of the golfer region to the neural network.
The motion analysis device according to claim 5 . The operation value is the rotation angle of the golfer's shoulder.
The motion analysis device according to claim 1 . The acquisition unit further acquires skeleton data representing the skeleton of the human body measured by the distance image sensor.
The derivation unit extracts a shoulder region in the vicinity of the shoulder from the depth image based on the skeleton data, and then inputs the image of the shoulder region to the neural network.
The motion analysis device according to claim 7 . The neural network has a convolution layer.
The motion analysis device according to any one of claims 1 to 8 . The acquisition unit acquires the depth image in time series and obtains the depth image.
The derivation unit derives the operation value of the time series by inputting the depth image of the time series into the neural network.
The motion analysis device according to any one of claims 1 to 9 . The acquisition unit further acquires skeleton data representing the skeleton of the human body measured by the distance image sensor.
The derivation unit extracts a region of interest in the vicinity of the region of interest of the object from the depth image based on the skeleton data, and then inputs the image of the region of interest to the neural network.
The motion analysis device according to any one of claims 1, 9 and 10 . The neural network has a dropout layer that nullifies non-focused areas representing unfocused parts of the object from the depth image.
The operation analysis device according to any one of claims 1 and 9 to 11 . It is a motion analysis device for analyzing the motion of an object.
An acquisition unit that acquires a depth image of the movement of the object taken by a distance image sensor, and
A derivation unit that derives the operation value by inputting the depth image acquired by the acquisition unit into a neural network that outputs an operation value that quantitatively represents the operation of the object.
Equipped with
The neural network has a dropout layer that nullifies non-focused areas representing unfocused parts of the object from the depth image.
Motion analysis device. A model building device that builds a model for analyzing the movement of an object.
A first acquisition unit that acquires a large number of depth images obtained by capturing the movement of the object with a distance image sensor, and
A second acquisition unit that acquires a large number of motion values that quantitatively represent the motion of the object, respectively, corresponding to the large number of depth images.
Based on the large number of depth images acquired by the first acquisition unit, the large number of operation values acquired by the second acquisition unit are used as a teacher signal, the depth image is input, and the operation value is output. Equipped with a learning unit to learn neural networks
The second acquisition unit acquires a large number of operation values representing the rotation angle of the object from the output values of the angular velocity sensor attached to the object.
Model building equipment. A model building device that builds a model for analyzing the movement of an object.
A first acquisition unit that acquires a large number of depth images obtained by capturing the movement of the object with a distance image sensor, and
A second acquisition unit that acquires a large number of motion values that quantitatively represent the motion of the object, respectively, corresponding to the large number of depth images.
Based on the large number of depth images acquired by the first acquisition unit, the large number of operation values acquired by the second acquisition unit are used as a teacher signal, the depth image is input, and the operation value is output. With a learning unit that learns neural networks
Equipped with
The second acquisition unit acquires a large number of operation values representing the position of the center of gravity of the object from the output value of the floor reaction force meter on which the object rides.
Model building equipment. It is a motion analysis system for analyzing the motion of an object.
A distance image sensor that captures a depth image that captures the movement of the object,
A neural network that outputs an operation value that quantitatively represents the operation of the object is provided with an operation analysis device that derives the operation value by inputting the depth image taken by the distance image sensor .
The depth image is an image of a golf swing.
system. It is a motion analysis system for analyzing the motion of an object.
A distance image sensor that captures a depth image that captures the movement of the object,
An motion analysis device that derives the motion value by inputting the depth image captured by the distance image sensor into a neural network that outputs an motion value that quantitatively represents the motion of the object.
Equipped with
The neural network has a dropout layer that nullifies non-focused areas representing unfocused parts of the object from the depth image.
system. It is a motion analysis method for analyzing the motion of an object.
A step of taking a depth image that captures the movement of the object with a distance image sensor,
A step of deriving the motion value by inputting the depth image captured by the distance image sensor into a neural network that outputs an motion value that quantitatively represents the motion of the object is included.
The depth image is an image of a golf swing.
Motion analysis method. It is a motion analysis method for analyzing the motion of an object.
A step of taking a depth image that captures the movement of the object with a distance image sensor,
A step of deriving the motion value by inputting the depth image captured by the distance image sensor into a neural network that outputs an motion value that quantitatively represents the motion of the object.
Including
The neural network has a dropout layer that nullifies non-focused areas representing unfocused parts of the object from the depth image.
Motion analysis method. A motion analysis program for analyzing the motion of an object.
A step of acquiring a depth image of the movement of the object taken by a distance image sensor, and
By inputting the acquired depth image into a neural network that outputs an operation value that quantitatively represents the operation of the object, a computer is made to execute a step of deriving the operation value .
The depth image is an image of a golf swing.
Motion analysis program. A motion analysis program for analyzing the motion of an object.
A step of acquiring a depth image of the movement of the object taken by a distance image sensor, and
A step of deriving the motion value by inputting the acquired depth image into a neural network that outputs an motion value that quantitatively represents the motion of the object.
Let the computer run
The neural network has a dropout layer that nullifies non-focused areas representing unfocused parts of the object from the depth image.
Motion analysis program.

Images