JP2019096228A - Human body shape model visualization system, human body shape model visualization method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily present time change data related to exercise and exercise.
A human body shape model visualization system acquires human body whole body shape model data, human body motion data, and time change data, and generates time change data, an initial arrangement of the human body whole body shape model, and a stop motion model generation time point. Determine and apply human body movement data to initial arrangement of human body whole body shape model according to reproduction operation command, generate animated human body whole body shape model, and generate timeline showing time lapse from reproduction start, reproduction operation Temporal change data position information and human body whole body shape model position information and stop motion model group position based on the instruction and initial arrangement of time change data in virtual space and animated human whole body shape model and timeline and stop motion model group Information and generate them, and the position and attitude of the image presentation unit Zui is, to produce a video image presentation unit is presented to the user.
[Selected figure] Figure 1

Description

  The present invention relates to a human body shape model visualization system, a human body shape model visualization method, and a program.

Research and development on industrial applications of virtual reality technology has a history of over 20 years. Until the early 2000s, presentation equipment such as head-mounted displays (HMDs) were expensive, and research was conducted as a training simulator for mastering mission-critical situations such as maintenance of power plants and petrochemical plants. The
In recent years, with the lower price of HMD (several hundred dollars to one thousand dollars) and the improvement of graphics processing capability of consumer PCs, XR technology (XR technology is a virtual reality positioned as a flow of virtual reality technology) The scope of application of (VR), Augmented Reality (AR), Mixed Reality (MR), etc. is expanded from the above-mentioned mission-critical situation to a broader industry from "2016: VR first year" It was also reported that. For example, in the United States, the application of XR technology has expanded to the fields of sports watching (360 degree video distribution), media, news coverage, architectural visualization, and game entertainment. In Japan, housing exhibition halls and wedding halls offer HMD-based preview services with 360-degree videos. According to the survey, the XR-related software market is expected to grow to ¥ 86 billion in fiscal 2016, and to ¥ 106.5 billion in fiscal 2021, and the scope of application is expected to expand in the future.

By the way, in sports coaching theory, it is considered important to systematically understand and practice both coaching science as a general theory and sports specialized subjects as each theory. For example, the coach (= researcher) extracts the action to be focused on for each subject, such as “start movement” in track and field track competition, “rebound movement” in basketball competition, and “crossing-off movement” in a jumping box, and the quality of the movement It is conducted to create or select a quantitative index to measure and to evaluate the change in index value before and after intervention.
Sports coaching science can be divided into the research areas of "kinetics", "morphology", "exercise physiology" and "statistics", motion measurement is motion capture and high-speed camera, morphology measurement is a maltin instrument and non-contact dimension measurement Physiological measurement corresponds to the device and physiological information.
A survey of the past five years of the “Coaching Studies” published by the Japan Coaching Society on the Internet shows that the characteristic behavior is that the coach (= researcher) observes so to speak craftsmanship for each subject and player It was found by force, and no standard behavior was defined.
Therefore, in the coaching scene, it is necessary to measure various data focusing on the operation extracted by the craftsman-like observation power, and to visualize the data in order to make the data a common language with the player.

The visualization of movement can be traced back to Malay (Buddha, 1830-1904), who invented a photo gun and created a series of photographs.
FIG. 9 is a view for explaining a continuous photo taken by a photo camera of Malay. In the example shown in FIG. 9, the viewpoint of visualization can not be changed later.

  And until today, visualization methods of motion and measurement information are designed on the premise of a two-dimensional display. As products, a visualization software LabChart for video and physiological measurement data by ADInstruments is known and widely used. By using this software, video and physiological measurement data can be reproduced synchronously. However, when this software is used, video and physiological information are drawn in separate windows, so it takes a lot of effort for the user to watch motion and physiological information at the same time, making it easy to understand Is hard to say.

  Also, conventionally, there is known a method of mixing or combining a computer-generated three-dimensional object and an image feed from a film camera in real time (see, for example, Patent Document 1). In the method described in Patent Document 1, the body of the film camera is movable in three dimensions. Also, a sensor mounted in or on the camera provides real time position data that defines the 3D position or 3D orientation of the camera or allows calculation of the 3D position. On the other hand, Patent Document 1 does not describe means for efficiently performing motion analysis or visualization of physiological measurement data.

Japanese Patent Publication No. 2015-521419 gazette

  The present invention has been made in view of the above circumstances, and it is an object of the present invention to provide a human body shape model visualization system, a human body shape model visualization method, and a program capable of presenting time change data related to movement and movement in an easy-to-understand manner. I assume.

  The present invention provides the following means in order to solve the above problems.

(1) A human body shape model visualization system according to an aspect of the present invention includes a data registration unit for acquiring human body whole body shape model data, human body motion data, and time change data, and time change data acquired by the data registration unit. Based on the temporal change data initial arrangement determination unit for determining the initial arrangement of temporal change data in the virtual space, and the initial arrangement of the human body whole body shape model in the virtual space based on the human whole body shape model data acquired by the data registration unit A human body whole body shape model initial arrangement determining unit for determining arrangement, and a stop motion model generation for determining a generation time point of a stop motion model generated based on human body whole shape model data and human body movement data acquired by the data registration unit A time point determination unit, a reproduction operation instruction unit for transmitting a reproduction operation instruction, and the reproduction operation instruction unit The human body animated by applying the human body motion data acquired by the data registration unit to the initial arrangement of the human body whole body shape model determined by the human body whole body shape model initial arrangement determination unit according to the reproduction operation command issued An animation human body whole body shape model generation unit that generates a whole body shape model and generates a time line indicating a time lapse from the start of reproduction, and a stop motion model based on generation time points determined by the stop motion model generation time point determination unit A stop motion model group generation unit for generating a group; a reproduction operation instruction sent by the reproduction operation instruction unit; an initial arrangement of time change data in the virtual space determined by the time change data initial arrangement determination unit; The animation generated by the human body whole body shape model generation unit Time-varying data position information indicating the position of time-varying data in the virtual space based on the human body whole-body shape model and the timeline and the stop motion model group generated by the stop motion model group generation unit; An arrangement movement unit for generating position information of a human body whole body shape model indicating a position of an animated human body whole body shape model and stop motion model group position information indicating a position of a stop motion model group in the virtual space; The image presentation unit uses the time change data position information generated by the unit, the human body whole-body shape model position information and the stop motion model group position information, and the position and orientation of the image presentation unit that presents the image to the user And a video generation unit for generating a video to be presented to a person.

(2) In the human body shape model visualization system according to (1) above, the time change data is time series physiological measurement data obtained by the physiological measurement device, and the video generation unit On the timeline showing the values of the physiological measurement data at that time (graph) and the animated human whole body shape model at that time, and the animated human whole body shape model placed on the timeline is The posture of the human body at that time may be indicated.

(3) In the human body shape model visualization system according to (1) or (2) above, the stop motion model generation time point determination unit interacts with the setting file, the external program, and the user of the video presentation unit. The generation interval of the stop motion model in the virtual space may be determined based on at least one of the above.

(4) In the human body shape model visualization system according to any one of (1) to (3) above, the stop motion model generation time point determination unit may calculate the time point extracted as human body motion data in a rule base. It may be determined as the generation time of the stop motion model in the virtual space.

(5) In the human body shape model visualization system according to any one of (1) to (4) above, when the reproduction operation instruction unit transmits a reproduction operation instruction in the forward direction, the placement and movement unit is: Move the animated whole human body shape model and the time line showing the time lapse from the start of reproduction to the right, or the waveform of the time change data and the stop motion model group generated by the stop motion model group generator to the left You may move it to

(6) In the human body shape model visualization system according to any one of the above (1) to (5), the human body whole body shape model data acquired by the data registration unit is a shape model created by imitating a person. It may be data or shape model data obtained by converting point cloud data obtained by the shape measuring machine.

(7) In the human body shape model visualization system according to any one of (1) to (6), the human body movement data acquired by the data registration unit is time series data acquired by the human body movement measurement device. It may be.

(8) In the human body shape model visualization system described in (1) above, the time change data acquired by the data registration unit may be time series data acquired by an external program.

(9) In the human body shape model visualization system according to (1), the reproduction operation instruction unit reproduces in response to an operation of a user of the video presentation unit and / or according to a command from an external program. In the reproduction operation instruction sent out by the reproduction operation instruction unit, an operation instruction is sent out: forward reproduction operation instruction, reverse direction reproduction operation instruction, forward direction speed reproduction operation instruction, reverse direction speed reproduction operation An instruction, a forward acceleration playback operation instruction, a reverse acceleration playback operation instruction, a pause instruction, a stop instruction and / or a loop playback instruction may be included.

(10) A human body shape model visualizing method according to an aspect of the present invention includes a data registration step of acquiring human body whole body shape model data, human body motion data, and time change data, and time change data acquired in the data registration step. Based on the time change data initial arrangement determining step of determining the initial arrangement of time change data in the virtual space, and the human whole body shape model in the virtual space based on the human whole body shape model data acquired in the data registration step. Human body whole body shape model initial arrangement determining step of determining the initial arrangement of the human body, and stop determining the generation time point of the stop motion model generated based on the human body whole body shape model data and human body movement data acquired in the data registration step. Motion model generation time point determination step And an initial arrangement of a human body whole body shape model determined in the human body whole body shape model initial arrangement determining step according to a reproduction operation instruction step of transmitting a reproduction operation instruction and the reproduction operation instruction transmitted in the reproduction operation step. An animated human whole body shape model generating step of generating an animated human whole body shape model by applying the human body motion data acquired in the data registration step, and generating a time line indicating time lapse from the start of reproduction; A stop motion model group generation step of generating a stop motion model group based on a generation time point determined in the stop motion model generation time point determination step; a reproduction operation instruction sent out in the reproduction operation instruction step; Time Initial arrangement of temporal change data in the virtual space determined in the change data initial arrangement determining step, an animated human whole body shape model and timeline generated in the animation human body whole body shape model generating step, and the stop motion model Time change data position information indicating the position of time change data in the virtual space based on the stop motion model group generated in the group generation step, and a human body indicating the position of the animated human whole body shape model in the virtual space An arrangement movement step for generating whole body shape model position information and stop motion model group position information indicating the position of the stop motion model group in the virtual space; and time change data generated in the arrangement movement step Video presented by the video presentation unit to the user based on the position information of the human body shape model and the stop motion model group position and the position and posture of the video presentation unit presenting the video to the user And V. generating an image.

(11) A program according to an aspect of the present invention is a computer that includes a data registration step of acquiring human body whole-body shape model data, human body motion data, and time change data, and time change data acquired in the data registration step. Based on a time change data initial arrangement determining step of determining an initial arrangement of time change data in the virtual space, and a human body whole body shape model in the virtual space based on the human whole body shape model data acquired in the data registration step. A human body whole body shape model initial arrangement determining step for determining an initial arrangement, and a stop motion for determining a generation time point of a stop motion model generated based on human body whole body shape model data and human body motion data acquired in the data registration step Model generation point determination step And a reproduction operation instruction step of transmitting a reproduction operation instruction, and an initial arrangement of a human body whole body shape model determined in the human body whole body shape model initial arrangement determining step according to the reproduction operation instruction transmitted in the reproduction operation instruction step. By generating the animated human whole body shape model by applying the human body movement data acquired in the data registration step, and generating a timeline indicating the time lapse from the start of reproduction; A stop motion model group generation step of generating a stop motion model group based on a generation time point determined in the stop motion model generation time point determination step; a reproduction operation instruction sent out in the reproduction operation instruction step; Said Initial arrangement of temporal change data in the virtual space determined in the inter-change data initial arrangement determination step, an animated human whole-body shape model and timeline generated in the animation human body whole-body shape model generation step, and the stop motion Indicates time-varying data position information indicating the position of time-varying data in the virtual space based on the stop motion model group generated in the model group generation step, and the position of the animated human whole body shape model in the virtual space An arrangement movement step of generating human body whole-body shape model position information and stop motion model group position information indicating the position of the stop motion model group in the virtual space; and time change generated in the arrangement movement step Based on data position information, human body whole shape shape position information and stop motion model group position information, and the position and posture of the image presentation unit presenting the image to the user, the image presented to the user by the image presentation unit is And a video generation step to generate.

  According to the present invention, it is possible to provide a human body shape model visualization system, a human body shape model visualization method, and a program capable of presenting time change data related to movement and movement in an easy-to-understand manner.

It is a figure which shows an example of a structure of the human body shape model visualization system of embodiment. It is a figure for demonstrating the imaging | video which the imaging | video production | generation part produced | generated. It is a figure for demonstrating the imaging | video of the time of the time t5 which the imaging | video production | generation part produced | generated. It is a figure for demonstrating the image | video of the reproduction completion time which the image | video production | generation part produced | generated. It is a figure for demonstrating the imaging | video immediately before the reproduction | regeneration start which an imaging | video presentation part presents to a user. It is a figure for demonstrating the other image in front of the reproduction | regeneration start which a video presentation part presents to a user. It is a figure for demonstrating the other imaging | video which an image | video presentation part presents to a user. It is a flowchart for demonstrating the process performed by a human body shape model visualization system. It is a figure for demonstrating the continuous photography image | photographed with the photo camera of Malay.

  Hereinafter, embodiments of a human body shape model visualization system, a human body shape model visualization method, and a program according to the present invention will be described in detail with reference to the drawings. In the drawings used in the following description, in order to make features of the present invention intelligible, the features that are the features may be enlarged and shown for convenience, and the dimensional ratio of each component is different from the actual one. There is. The materials, dimensions, and the like exemplified in the following description are merely examples, and the present invention is not limited to them, and can be appropriately modified and implemented within the scope of achieving the effects of the present invention.

FIG. 1 is a diagram showing an example of the configuration of a human body shape model visualization system 1 according to the embodiment.
In the example shown in FIG. 1, human body whole body shape model data 110, human body motion data 120, and time change data 130 are configured to be able to be input to human body shape model visualization system 1 via AND operation unit 150.
The human body whole body shape model data 110 input to the human body shape model visualization system 1 is, for example, shape model data created to simulate a person.
In another example, a shape obtained by converting point cloud data obtained by a shape measuring machine (not shown) such as a digitizer into the whole human body shape model data 110 input to the human body shape model visualization system 1 It may be model data.

In the example shown in FIG. 1, the human body motion data 120 input to the human body shape model visualization system 1 is, for example, time series data such as each joint angle obtained by a human body motion measuring device (not shown).
The time change data 130 input to the human body shape model visualization system 1 is, for example, time series data obtained by an external program. The time-series data obtained by the external program is, for example, time-series physiological measurement data obtained by the physiological measurement device. Examples of time-series physiological measurement data obtained by the physiological measurement device include time-series data such as myoelectric potential, heartbeat, respiratory metabolism, consumed calories, and the like.
In another example, as the time change data 130, the subject of human body movement data and time change data measures the state of mood during exercise recorded using a portable terminal (not shown) during the measurement, etc. May be input to the human body shape model visualization system 1.

In the example shown in FIG. 1, the image presentation unit 200 generates an animated human body whole-body shape model (operated by the human body shape model visualization system 1 based on the human body whole-body shape model data 110 and the human body motion data 120) An image of a moving image of the human body whole body shape model is presented to the user of the human body shape model visualization system 1 and the image presentation unit 200.
In addition, the image presentation unit 200 is a stop motion model generated by the human body shape model visualization system 1 based on the human body whole body shape model data 110 and the human body motion data 120 (with the operating whole body shape model at a predetermined timing). Present the user with the image of
In addition, the image presentation unit 200 presents the user with an image of the waveform of the time change data generated by the human body shape model visualization system 1 based on the time change data 130.
The image presentation unit 200 is, for example, a known virtual reality (VR) device or an augmented reality (AR) device. Information on the position and orientation of the image presentation unit 200 is input to the human body shape model visualization system 1. When the position or posture of the video presentation unit 200 changes with the change of the position or posture of the user wearing the video presentation unit 200, the angle or the like of the video presented to the user by the video presentation unit 200 changes.
In order to detect the position and orientation of the image presentation unit 200, for example, techniques such as VR tracking technology "Lighthouse" using an external sensor, SLAM (Simultaneous Localization and Mapping) using a built-in camera, and the like are used.

In the example shown in FIG. 1, the human body shape model visualization system 1 includes a data registration unit 1A, a time change data initial arrangement determination unit 1B, a human whole body shape model initial arrangement determination unit 1C, and a stop motion model generation time point determination unit 1D. , A reproduction operation command unit 1E, an animation human body whole-body shape model generation unit 1F, a stop motion model group generation unit 1G, an arrangement movement unit 1H, and a video generation unit 1I.
The data registration unit 1A acquires the human body whole body shape model data 110, the human body motion data 120, and the time change data 130 input to the human body shape model visualization system 1.

The time change data initial arrangement determination unit 1B is connected to the data registration unit 1A. The time change data initial arrangement determination unit 1B determines the initial arrangement of time change data in the virtual space based on the time change data 130 acquired by the data registration unit 1A. The virtual space is a space displayed on the display of the VR device or AR device.
That is, the time change data initial arrangement determination unit 1B determines the initial position of the waveform of the time change data in the space displayed on the display of the VR device or AR device (specifically, for example, the value of the waveform of the time change data at time t0 Determine the display position).

The human body whole-body shape model initial arrangement determination unit 1C is connected to the data registration unit 1A. The human body whole body shape model initial arrangement determining unit 1C determines the initial arrangement of the human body whole body shape model in the virtual space based on the human body whole shape model data 110 acquired by the data registration unit 1A.
That is, the human body whole body shape model initial arrangement determination unit 1C is an initial position of the human body whole body shape model in the space displayed on the display of the VR device or the AR device (in detail, for example, the display position of the human body whole body shape model at time t0) Decide.

The stop motion model generation time point determination unit 1D is connected to the data registration unit 1A. The stop motion model generation time point determination unit 1D generates a stop motion model generation time point (for example, time t0, t1, t2, etc.) generated based on the human body whole body shape model data 110 and the human body motion data 120 acquired by the data registration unit 1A. To determine).
The stop motion model generation time point determination unit 1D determines a stop motion model generation interval (for example, generation interval t1-t0, t2-t1) based on, for example, a setting file.
In another example, the stop motion model generation time point determination unit 1D may determine the stop motion model generation interval based on an external program, and based on an operation like interaction with a user (for example, a user request). The generation interval of the stop motion model may be determined.
Alternatively, the stop motion model generation time point determination unit 1D may determine a time point extracted from the human body motion data 120 on a rule basis as a generation time point of the stop motion model.
The stop motion model generation time point determination unit 1D may combine them to determine the generation interval of the stop motion model.

In the example shown in FIG. 1, the reproduction operation instruction unit 1E sends out a reproduction operation instruction. The reproduction operation instruction is an instruction for instructing the reproduction operation of the above-described animated human whole body shape model (moving image of a moving human whole body shape model) or the like.
The reproduction operation instruction unit 1E sends out a reproduction operation instruction according to the operation of the user of the human body shape model visualization system 1 and the image presentation unit 200.
In another example (for example, when the human body shape model visualization system 1 is incorporated into the archive viewing system), the reproduction operation instruction unit 1E may send out the reproduction operation instruction according to the command from the external program. The reproduction operation command unit 1E may send out the reproduction operation command according to both the user operation and the command from the external program.

In the example shown in FIG. 1, the reproduction operation instruction sent by the reproduction operation instruction unit 1E includes, for example, a reproduction operation instruction in the forward direction, a reproduction operation instruction in the reverse direction, and a time reproduction operation instruction in the forward direction. And a reverse playback instruction, a pause instruction, a stop instruction, and a loop playback instruction.
In another example, the reproduction operation instruction sent by the reproduction operation instruction unit 1E includes at least one of the above-described instructions.

In the example shown in FIG. 1, an animation human body whole body shape model generation unit 1F is connected to a human body whole body shape model initial arrangement determination unit 1C and a reproduction operation command unit 1E.
The animated human body whole body shape model generation unit 1F is an animated human body whole body shape model by applying the human body motion data 120 to the initial arrangement of the human body whole body shape model according to the reproduction operation command sent out by the reproduction operation command unit 1E. (A moving image of an operating human body whole-body shape model) is generated. The animated whole body shape model is displayed on the display of the VR device or AR device as described later.
In addition, the animation human body whole-body shape model generation unit 1F generates a time line indicating time lapse from the start of reproduction. The timeline is displayed on the display of the VR device or AR device, as described later. Further, the position of the timeline in the virtual space displayed on the display of the VR device or AR device changes with the passage of time after the start of reproduction.

The stop motion model group generation unit 1G is connected to the stop motion model generation time point determination unit 1D. The stop motion model group generation unit 1G generates a stop motion model group based on the generation time point determined by the stop motion model generation time point determination unit 1D.
For example, when the stop motion model generation time point determination unit 1D determines the time t0, the time t1, and the time t2 as the generation time of the stop motion model, the stop motion model group generation unit 1G generates the animated whole human body. A shape model (moving image of an operating human body whole-body shape model) is stopped at time t0 (a still image of the human whole body shape model) is generated as a stop motion model at time t0. In addition, the stop motion model group generation unit 1G generates a stop motion model of time t1 by stopping the animated human whole body shape model at time t1 as a stop motion model of time t1, and stops the animated human whole body shape model at time t2. Is generated as a stop motion model at time t2.
In addition, when the stop motion model generation time point determination unit 1D determines the time t0, the time t1, and the time t2 as the generation time of the stop motion model, the stop motion generated by the stop motion model group generation unit 1G The model group is configured of a stop motion model at time t0, a stop motion model at time t1, and a stop motion model at time t2.

The arrangement movement unit 1H is connected to a time change data initial arrangement determination unit 1B, a reproduction operation instruction unit 1E, an animation human body whole body shape model generation unit 1F, and a stop motion model group generation unit 1G.
That is, to the arrangement moving unit 1H, the reproduction operation instruction sent out by the reproduction operation instruction unit 1E, the initial arrangement of the time change data in the virtual space determined by the time change data initial arrangement determining unit 1B, and the animation human body whole body shape model generation The animated human whole body shape model and timeline generated by the unit 1F and the stop motion model group generated by the stop motion model group generation unit 1G are input.
The arrangement movement unit 1H, based on the input, temporal change data position information indicating the position of time change data in the virtual space, and a human body whole body shape model position indicating the position of the animated human whole body shape model in the virtual space Information and stop motion model group position information indicating the position of the stop motion model group in the virtual space are generated.
The time change data position information is, for example, information on where in the virtual space displayed on the display of the VR device or the AR device, the waveform of the time change data should be arranged.
For example, in the virtual space displayed on the display of the VR device or AR device, for example, an animated human whole body shape model (moving image of the human whole body shape model) should be placed anywhere in the virtual space displayed on the display of the VR device or AR device. It is the information of the day.
The stop motion model group position information includes, for example, the stop motion model at time t0, the stop motion model at time t1, and the stop motion model at time t2 anywhere in the virtual space displayed on the display of the VR device or AR device. It is information of what should be arranged.

In the example shown in FIG. 1, when the reproduction operation instruction unit 1E transmits a reproduction operation instruction in the forward direction, the arrangement moving unit 1H includes an animated human whole body shape model and a time line indicating time lapse from the start of reproduction. Move to the right.
In another example, when the reproduction operation instruction unit 1E sends a reproduction operation instruction in the forward direction, the arrangement moving unit 1H generates the waveform of the time change data and the stop motion model group generated by the stop motion model group generation unit 1G. May be moved leftward.

In the example shown in FIG. 1, the video generation unit 1I is connected to the arrangement movement unit 1H.
That is, the image generation unit 1I receives the time change data position information generated by the placement and movement unit 1H, the human body whole body shape model position information, and the stop motion model group position information. Also, information on the position and orientation of the video presentation unit 200 is input to the video generation unit 1I.
The video generation unit 1I sends the video presentation unit 200 a message based on the time change data position information, the human whole body shape model position information, the stop motion model group position information, and the position and orientation of the video presentation unit 200. Generate a video to be presented.

FIG. 2 is a diagram for explaining a video generated by the video generation unit 1I. In detail, FIG. 2 shows an image at time t3 generated by the image generation unit 1I.
In the example shown in FIG. 2, the video generated by the video generation unit 1I includes the waveform TD1 of the time change data, the waveform TD2 of the time change data, the waveform TD3 of the time change data, and the waveform TD4 of the time change data. It is done. Waveforms TD1, TD2, TD3, and TD4 of the time change data are waveforms of time-series physiological measurement data obtained by the physiological measurement device. In detail, waveforms TD1 to TD4 of the time change data indicate, for example, electromyogram waveforms. The waveform TD1 of the time change data indicates, for example, a myoelectric waveform of the gluteus maximus muscle. The waveform TD2 of the time change data indicates, for example, an electromyographic waveform of the quadriceps femoris muscle. The waveform TD3 of the time change data shows, for example, a myoelectric waveform of Hamstrings. The waveform TD4 of the time change data indicates, for example, an electromyographic waveform of the tibialis anterior muscle.
In addition, the video generated by the video generation unit 1I includes a timeline TL indicating an elapse of time from the start of reproduction (time t0). The timeline TL in FIG. 2 indicates that time (t3-t0) has elapsed from the start of reproduction (time t0).
The intersection of the waveform TD1 of the time change data and the timeline TL indicates the value of the myoelectric potential of the gluteus maximus at time t3. The intersection of the waveform TD2 of the time change data and the timeline TL indicates the value of the myoelectric value of the quadriceps femoris muscle at the time t3. The intersection of the waveform TD3 of the time change data and the timeline TL indicates the value of the Hamstrings myoelectric potential at the time t3. The intersection of the waveform TD4 of the time change data and the timeline TL indicates the value of the myoelectric value of the tibialis anterior muscle at time t3.
Further, on the timeline TL, an animated human whole body shape model AM at time t3 is arranged. The animated whole human body shape model AM indicates the posture of the human body at time t3.

Further, in the example shown in FIG. 2, the video generated by the video generation unit 1I includes the stop motion model SMt0 at time t0, the stop motion model SMt1 at time t1, and the stop motion model SMt2 at time t2. .
As described above, the stop motion model SMt0 corresponds to an animated human whole body shape model (moving image of a moving human whole body shape model) AM stopped at time t0. The stop motion model SMt1 corresponds to stopping the animated human whole body shape model AM at time t1, and the stop motion model SMt2 corresponds to stopping the animated human whole body shape model AM at time t2. Do.
In the example shown in FIG. 2, the generation intervals (generation intervals t1-t0 and t2-t1) of the stop motion models SMt0, SMt1 and SMt2 in the virtual space are constant time intervals (e.g. 0. 0.) based on the setting file or the external program. 3 seconds).

FIG. 3 is a diagram for explaining a video at time t5 generated by the video generation unit 1I.
In the example shown in FIG. 3, the position of the video presentation unit 200 is different from that in the example shown in FIG. 2. Further, in the example shown in FIG. 3, the time point when the stop motion model generation time point determination unit 1D is extracted from the human body motion data 120 on a rule basis (for example, the time point when only landing moment is extracted Times t2, t3 and t4) are determined as generation time points of the stop motion models SMt2, SMt3 and SMt4 in the virtual space.
By configuring in this way, for example, it is possible to (semi) automatically generate a stop motion model based on the needs of the user (coach or player), and to make the observation more efficient.
Depending on the subject of exercise, or the researcher, the user's attentional movement can vary depending on the subject, ie, there is a case where one wants to observe the moment of heel contact and one wants to observe the moment of kicking even in the same running subject. It can.
The moment of generating the stop motion model is not only the constant time interval specified as a parameter, but also the moment specified based on a rule (for example, the moment of heel contact in running operation based on the coordinates of the vertical axis of the ankle joint) It can be extracted and adopted.
As a result, it is possible to (semi) automatically generate a characteristic motion during exercise, such as a heel contact stop motion model of a sprinting motion, and its generation interval can be arbitrarily changed after measurement .

FIG. 4 is a diagram for explaining the video at the reproduction completion time point generated by the video generation unit 1I.
In the example shown in FIG. 4, the position of the image presentation unit 200 is different compared to the examples shown in FIGS. 2 and 3. Further, in the example shown in FIG. 4, since the reproduction of the moving image is completed, the animated human whole body shape model AM, that is, the moving human whole body shape model does not exist in the image. As a result, there is no timeline TL in the video.

FIG. 5 is a diagram for explaining a video immediately before the start of reproduction presented to the user by the video presentation unit 200. In particular, FIG. 5 shows the view of the head mounted display wearer.
In the example shown in FIG. 5, the position of the image presentation unit 200 is different compared to the examples shown in FIGS. 2 to 4. Further, in the example shown in FIG. 5, since the reproduction of the moving image has not started yet, there is no animated human whole body shape model AM, that is, a moving human whole body shape model in the image. As a result, there is no timeline TL in the video.
As described above, the video shown in FIG. 5 is not a video generated by the video generation unit 1I, but a video presented by the video presentation unit 200 to the user. Accordingly, FIG. 5 includes the image for the left eye of the user (image in the circle on the left side of FIG. 5) and the image for the right eye of the user (image in the right circle of FIG. 5). Be

FIG. 6 is a diagram for explaining another video immediately before the start of reproduction (that is, at the same time as the video shown in FIG. 5) presented to the user by the video presenting unit 200.
In the example shown in FIG. 6, the position of the image presentation unit 200 is different compared to the example shown in FIG. Further, in the example shown in FIG. 6, as in the example shown in FIG. 5, the animated human whole body shape model AM does not exist in the image, and the timeline TL also does not exist.
Also, in FIG. 6, similarly to FIG. 5, an image for the left eye of the user (image in a circle on the left side of FIG. 6) and an image for the right eye of the user (in the circle on the right side of FIG. And the image).

FIG. 7 is a diagram for explaining another video presented by the video presenting unit 200 to the user.
The video shown in FIG. 7 includes the stop motion model SMt3 included in the video shown in FIG. In the example illustrated in FIG. 7, for example, in order to confirm the landing habit, the image presentation unit 200 approaches the stop motion model SMt3.
In the example shown in FIGS. 5 to 7, by combining the image presentation unit 200 (HMD) that presents an image according to the position and orientation of the head of the user, the exercise currently performed just in front of the eyes is observed As a result, the user can observe while selecting an intuitive and free viewpoint. As a result, the user can observe, for example, a portion to be focused on from a free viewpoint, such as the knee joint extension angle and the ankle angle at the moment of landing in running.

FIG. 8 is a flowchart for explaining the process executed by the human body shape model visualization system 1.
In the example illustrated in FIG. 8, in step S11, the data registration unit 1A acquires human body whole-body shape model data 110, human body motion data 120, and time change data 130.
Next, in step S12, based on the time change data 130, the time change data initial arrangement determining unit 1B initially arranges the waveforms TD1, TD2, TD3, and TD4 of time change data in the virtual space (that is, the waveforms of time change data). The position of the left end of TD1, TD2, TD3, and TD4 is determined.
Next, in step S13, the human body whole body shape model initial arrangement determining unit 1C determines the initial arrangement of the human body whole body shape model in the virtual space (that is, the animated human whole body shape model AM moves) based on the human whole body shape model data 110. Determine where to begin.
Next, in step S14, stop motion model generation time point determination unit 1D generates stop motion models SMt0, SMt1, and SMt2 generated based on human body whole-body shape model data 110 and human body motion data 120 (time t0 and t1). , T2).

Next, in step S15-1, the reproduction operation command unit 1E determines whether the user's operation or a command from an external program has been received. If there is a user operation or a command from an external program, the process proceeds to step S15-2. If there is neither the user's operation nor the command from the external program, step S15-1 is repeatedly executed until there is the user's operation or the command from the external program.
In step S15-2, the reproduction operation instruction unit 1E sends out a reproduction operation instruction.
Next, in step S16, the animation human body whole body shape model generation unit 1F acquires the data registration unit 1A in the initial arrangement of the human body whole body shape model determined by the human body whole body shape model initial arrangement determination unit 1C according to the reproduction operation command. By applying the human body motion data 120, an animated human body whole-body shape model AM is generated, and a timeline TL indicating time lapse from the start of reproduction is generated.
Next, in step S17, the stop motion model group generation unit 1G determines the stop motion model group (stop motion model SMt0, SMt0, t2 based on the generation time points (time t0, t1, t2) determined by the stop motion model generation time point determination unit 1D. Generate SMt1 and SMt2 groups).

Next, in step S18, the arrangement moving unit 1H performs the reproduction operation instruction, the initial arrangement of the waveforms TD1, TD2, TD3, and TD4 of the time-varying data in the virtual space, the animated human whole body shape model AM, and the timeline TL. Time-varying data position information indicating positions of waveforms TD1, TD2, TD3, and TD4 of time-varying data in the virtual space, based on the stop motion model group (the group of stop motion models SMt0, SMt1, and SMt2); Human body whole body shape model position information indicating the position of the animated human body whole body shape model AM and stop motion model group position information indicating the position of the stop motion model group (group of stop motion models SMt0, SMt1, and SMt2) in virtual space And It is formed.
Next, in step S19, the video generation unit 1I displays the time change data position information generated by the placement and movement unit 1H, the human body whole-body shape model position information and the stop motion model group position information, and the video presentation unit Based on the position and orientation of 200, the video presenting unit 200 generates a video to be presented to the user.

In detail, for example, as shown in FIG. 2, the video generation unit 1I generates time change data (time t3 in the example shown in FIG. 2) on the timeline TL indicating time lapse from the start of reproduction. The values of waveforms TD1, TD2, TD3 and TD4 of physiological measurement data) and the animated human whole body shape model AM at that time are arranged. Also, the animated human whole body shape model AM placed on the timeline TL indicates the posture of the human body at that time (time t3).
That is, the example shown in FIG. 2 specifies a specific arrangement method of the physiological measurement data in the virtual space, the timeline TL indicating the elapsed time from the start of reproduction, and the animated human whole body shape model AM.
Therefore, the image presentation unit 200 can present time change data related to exercise and exercise to the user in an easy-to-understand manner. The user can easily grasp time change data related to exercise and exercise.
By presenting the physiological measurement data as waveforms TD1, TD2, TD3 and TD4 of the time change data, the user can easily gaze at human motion and physiological information at the same time as long as he / she pays attention to the timeline TL. Become.

In the example shown in FIG. 2, the animated human whole body shape model AM moves from the left side of FIG. 2 to the right side of FIG. Therefore, in the example shown in FIG. 2, when the reproduction operation instruction unit 1E sends a reproduction operation instruction in the forward direction, the arrangement moving unit 1H indicates an animated human whole body shape model AM and time lapse from the start of reproduction. The timeline TL is moved to the right in FIG. Further, the placement moving unit 1H places the stop motion models SMt0, SMt1, and SMt2 at the positions of the animated whole human body shape model AM after passing through. That is, the stop motion models SMt0, SMt1, and SMt2 are not arranged (displayed) immediately after the start of reproduction, but are arranged (displayed) as soon as the animated whole human body shape model AM passes.
On the other hand, for example, in the case where the animated whole human body shape model AM performs "vertical jump", the arrangement moving unit 1H temporarily makes the animated human whole body shape model AM and the stop motion models SMt0, SMt1, and SMt2 the same. If placed in a position, they will overlap and it will be difficult to see.
Therefore, for example, in the case where the animated whole human body shape model AM performs vertical jumping, the arrangement moving unit 1H includes the waveforms TD1, TD2, TD3, and TD4 of time change data, the stop motion model group (stop motion model SMt0 , SMt1, and SMt2) toward the left. Therefore, even if the position of the animated human whole body shape model AM does not change, the animated human whole body shape model AM and the stop motion models SMt0, SMt1, and SMt2 do not overlap. As a result, it is possible to make the image easy to see.

  As described above, the human body shape model visualization system 1 of the embodiment applies the XR technology to visualize motion / physiological measurement data in an easy-to-understand manner.

  The human body shape model visualization system, the human body shape model visualization method and program of the present invention can be widely used in the fields of life, welfare, health care, sports, health support and the like.

1 Human body shape model visualization system 1A Data registration unit 1B Time change data initial arrangement determination unit 1C Human body whole body shape model initial arrangement determination unit 1D Stop motion model generation time point determination unit 1E Reproduction operation instruction unit 1F Animation human body Whole body shape model generation unit 1G ... Stop motion model group generation unit 1H ... Arrangement movement unit 1I ... Video generation unit 110 ... Human body whole body shape model data 120 ... Human body movement data 130 ... Time change data 150 ... AND operation unit 200 ... Image presentation unit

Claims (11)

A data registration unit for acquiring human body whole body shape model data, human body motion data, and time change data;
A time change data initial arrangement determination unit that determines an initial arrangement of time change data in a virtual space based on the time change data acquired by the data registration unit;
A human body whole body shape model initial arrangement determination unit which determines an initial arrangement of the human body whole body shape model in the virtual space based on the human body whole body shape model data acquired by the data registration unit;
A stop motion model generation time point determination unit that determines a generation time point of a stop motion model generated based on human body whole-body shape model data and human body motion data acquired by the data registration unit;
A reproduction operation instruction unit that sends out a reproduction operation instruction;
Applying the human body motion data acquired by the data registration unit to the initial arrangement of the human body whole body shape model determined by the human body whole body shape model initial arrangement determination unit according to the reproduction operation instruction sent by the reproduction operation instruction unit; An animated human whole body shape model generation unit that generates an animated human whole body shape model and generates a timeline indicating a time lapse from the start of reproduction;
A stop motion model group generation unit that generates a stop motion model group based on the generation time point determined by the stop motion model generation time point determination unit;
The reproduction operation instruction sent by the reproduction operation instruction unit, the initial arrangement of time change data in the virtual space determined by the time change data initial arrangement determination unit, and the animation generated by the animation human body whole body shape model generation unit Time change data position information indicating the position of time change data in the virtual space based on a human body whole body shape model and a time line, and the stop motion model group generated by the stop motion model group generation unit, and in the virtual space An arrangement movement unit that generates human body whole body shape model position information indicating the position of an animated human whole body shape model, and stop motion model group position information indicating the position of the stop motion model group in the virtual space;
The image presentation based on the time change data position information generated by the arrangement moving unit, the human body whole-body shape model position information and the stop motion model group position information, and the position and posture of the image presentation unit presenting the image to the user. A video generation unit for generating a video to be presented to the user by the unit;
Human body shape model visualization system. The time change data is time series physiological measurement data obtained by the physiological measurement device,
The video generation unit arranges values of physiological measurement data at that time and an animated whole human body shape model at that time on a time line indicating time lapse from the start of reproduction,
An animated whole human body shape model placed on the timeline shows the posture of the human body at that time,
The human body shape model visualization system according to claim 1. The stop motion model generation time point determination unit determines a stop motion model generation interval in the virtual space based on at least one of a setting file, an external program, and an interactive operation with the user of the image presentation unit. ,
The human body shape model visualization system according to claim 1 or 2. The stop motion model generation time point determination unit determines a time point extracted on a rule basis from human body motion data as a generation time point of a stop motion model in the virtual space.
The human body shape model visualization system according to any one of claims 1 to 3. When the reproduction operation instruction unit transmits the reproduction operation instruction in the forward direction, the arrangement movement unit moves the animated human body whole-body shape model and the time line indicating the elapsed time from the start of reproduction to the right, or Moving the waveform of the time change data and the stop motion model group generated by the stop motion model group generation unit in the left direction;
The human body shape model visualization system according to any one of claims 1 to 4. The human body whole-body shape model data acquired by the data registration unit may be shape model data created by imitating a person, or shape model data obtained by converting point cloud data obtained by a shape measuring machine. is there,
The human body shape model visualization system according to any one of claims 1 to 5. The human body movement data acquired by the data registration unit is time-series data acquired by the human body movement measurement device,
The human body shape model visualization system according to any one of claims 1 to 6. The time change data acquired by the data registration unit is time series data acquired by an external program,
The human body shape model visualization system according to claim 1. The reproduction operation instruction unit sends out a reproduction operation instruction according to an operation of a user of the video presentation unit and / or according to a command from an external program.
The playback operation command sent out by the playback operation command unit includes a forward playback operation command, a reverse playback operation command, a forward playback time playback operation command, a backward playback speed operation instruction, and an increase in forward direction. And at least one of a fast playback operation instruction, a reverse acceleration playback operation instruction, a pause instruction, a stop instruction, and a loop playback instruction.
The human body shape model visualization system according to claim 1. A data registration step of acquiring human body whole body shape model data, human body motion data and time change data;
A time change data initial arrangement determination step of determining an initial arrangement of time change data in a virtual space based on the time change data acquired in the data registration step;
A human body whole body shape model initial arrangement determining step of determining an initial arrangement of the human body whole body shape model in the virtual space based on the human body whole body shape model data acquired in the data registration step;
A stop motion model generation time point determination step of determining a generation time point of a stop motion model generated based on human body whole body shape model data and human body motion data acquired in the data registration step;
A reproduction operation instruction step for transmitting a reproduction operation instruction;
The human body movement data acquired in the data registration step is set in the initial arrangement of the human body whole body shape model determined in the human body whole body shape model initial arrangement determination step according to the reproduction operation instruction sent in the reproduction operation instruction step. An animated human whole body shape model generating step of generating an animated human whole body shape model and generating a timeline indicating a time lapse from the start of reproduction by applying;
A stop motion model group generation step of generating a stop motion model group based on the generation point determined in the stop motion model generation point determination step;
The reproduction operation instruction sent in the reproduction operation instruction step, the initial arrangement of time change data in the virtual space determined in the time change data initial arrangement determination step, and the animation human body whole body shape model generation step Time-varying data position information indicating the position of time-varying data in the virtual space based on the animated whole human body shape model and timeline, and the stop motion model group generated in the stop motion model group generation step; A configuration for generating stop motion model group position information indicating the position of a stop motion model group in the virtual space, and human body whole body shape model position information indicating the position of an animated human body whole body shape model in the virtual space And the moving step,
The image based on the time change data position information, the human body whole body shape model position information and the stop motion model group position information generated in the arrangement movement step, and the position and posture of the image presentation unit that presents the image to the user And a video generation step of generating a video to be presented to the user by the presentation unit.
Human body shape model visualization method. On the computer
A data registration step of acquiring human body whole body shape model data, human body motion data and time change data;
A time change data initial arrangement determination step of determining an initial arrangement of time change data in a virtual space based on the time change data acquired in the data registration step;
A human body whole body shape model initial arrangement determining step of determining an initial arrangement of the human body whole body shape model in the virtual space based on the human body whole body shape model data acquired in the data registration step;
A stop motion model generation time point determination step of determining a generation time point of a stop motion model generated based on human body whole body shape model data and human body motion data acquired in the data registration step;
A reproduction operation instruction step for transmitting a reproduction operation instruction;
The human body movement data acquired in the data registration step is set in the initial arrangement of the human body whole body shape model determined in the human body whole body shape model initial arrangement determination step according to the reproduction operation instruction sent in the reproduction operation instruction step. An animated human whole body shape model generating step of generating an animated human whole body shape model and generating a timeline indicating a time lapse from the start of reproduction by applying;
A stop motion model group generation step of generating a stop motion model group based on the generation point determined in the stop motion model generation point determination step;
The reproduction operation instruction sent in the reproduction operation instruction step, the initial arrangement of time change data in the virtual space determined in the time change data initial arrangement determination step, and the animation human body whole body shape model generation step Time-varying data position information indicating the position of time-varying data in the virtual space based on the animated whole human body shape model and timeline, and the stop motion model group generated in the stop motion model group generation step; A configuration for generating stop motion model group position information indicating the position of a stop motion model group in the virtual space, and human body whole body shape model position information indicating the position of an animated human body whole body shape model in the virtual space And the moving step,
The image based on the time change data position information, the human body whole body shape model position information and the stop motion model group position information generated in the arrangement movement step, and the position and posture of the image presentation unit that presents the image to the user And a video generation step of generating a video to be presented to a user by the presentation unit.