JP7066116B2 - Human body shape model visualization system, human body shape model visualization method and program - Google Patents

Description

The present invention relates to a human body shape model visualization system, a human body shape model visualization method and a program.

Research and development on industrial applications of virtual reality technology has a history of more than 20 years. Until the early 2000s, presentation equipment such as head-mounted displays (HMDs) were expensive, and research was conducted as a training simulator to master mission-critical situations such as maintenance of power plants and petrochemical plants. rice field.
In recent years, with the background of lower prices of HMDs (hundreds of dollars to 1,000 dollars) and improvement of graphics processing capacity of consumer PCs, XR technology (XR technology is a virtual reality that is positioned in the flow of virtual reality technology). (VR), Augmented Reality (AR), Mixed Reality (MR), etc.) The application range has expanded from the above-mentioned mission-critical situation to a wider industry, and "2016 is the first year of VR." Was also reported. For example, in the United States, the application of XR technology is expanding to the fields of watching sports (delivery of 360-degree video), media, news, architectural visualization, and game entertainment. In Japan, housing exhibition halls and wedding halls use HMDs to provide a 360-degree video preview service. According to the survey, the XR-related software market was 8.6 billion yen in FY2016, and is expected to grow to 106.5 billion yen in FY2021, and the scope of application is expected to continue to expand.

By the way, in sports coaching theory, it is important to systematically understand and practice both coaching studies as a general theory and sports specialized subjects as specific theories. For example, the coach (= researcher) extracts notable movements for each subject, such as "start movement" in athletics, "rebound movement" in basketball competition, and "railroad crossing movement" in the vaulting box, and the quality of the movement. Quantitative indicators are created or selected, and changes in the indicator values before and after the intervention are evaluated.
Sports coaching science can be divided into research areas of "kinematics", "morphology", "exercise physiology", and "statistics". Physiological measurement corresponds to the device and physiological information.
According to a survey of the past five years of the academic journal "Coaching Studies" published on the Internet by the Japan Coaching Society, the characteristic movements are observed by coaches (= researchers) for each subject and each player, so to speak, as a craftsman. It was found by force, and no standard behavior was defined.
Therefore, in the coaching scene, it is necessary to focus on the movements extracted by the craftsman's observing ability, measure various data, and visualize the data so that it can be used as a common language with the athletes.

The visualization of motion can be traced back to Murray (France, 1830-1904), who invented the photographic gun and created a series of photographs.
FIG. 9 is a diagram for explaining a series of photographs taken by a Murray photogun. In the example shown in FIG. 9, the viewpoint of visualization cannot be changed later.

And to this day, the visualization method of motion and measurement information is designed on the premise of a two-dimensional display. As a product, LabChart, a visualization software for video and physiological measurement data by AD Instruments, is well known and widely used. By using this software, video and physiological measurement data can be played back in synchronization. However, when this software is used, the image and the physiological information are drawn in separate windows, so it takes a lot of effort for the user to watch the movement and the physiological information at the same time, and the display is easy to understand. Is hard to say.

Further, conventionally, a method of mixing or synthesizing a computer-generated three-dimensional object and a video feed from a film camera in real time is known (see, for example, Patent Document 1). In the method described in Patent Document 1, the body of the film camera can be moved three-dimensionally. It also provides real-time position data that allows a sensor in or attached to the camera to define a 3D position or orientation of the camera, or to calculate a 3D position. On the other hand, Patent Document 1 does not describe a means for efficiently performing motion analysis and visualization of physiological measurement data.

Special Table 2015-521419

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a human body shape model visualization system, a human body shape model visualization method, and a program capable of presenting motion and time change data related to motion in an easy-to-understand manner. And.

The present invention provides the following means for solving the above problems.

(1) The human body shape model visualization system according to one aspect of the present invention has a data registration unit that acquires human body shape model data, human body motion data, and time change data, and time change data acquired by the data registration unit. Based on the time change data initial placement determination unit that determines the initial placement of time change data in the virtual space, and the initial human body shape model in the virtual space based on the human body shape model data acquired by the data registration unit. Stop motion model generation that determines the generation time of the stop motion model generated based on the human body whole body shape model initial placement determination unit that determines the placement and the human body whole body shape model data and human body motion data acquired by the data registration unit. Initial stage of the human body whole body shape model determined by the human body whole body shape model initial placement determination unit according to the time point determination unit, the reproduction operation command unit that sends the reproduction operation command, and the reproduction operation command sent by the reproduction operation command unit. By applying the human body motion data acquired by the data registration unit to the arrangement, an animated human body whole body shape model is generated, and an animated human body whole body shape model generation that generates a timeline showing the passage of time from the start of playback is generated. A stop motion model group generation unit that generates a stop motion model group based on a generation time determined by the stop motion model generation time determination unit, a reproduction operation command sent by the reproduction operation command unit, and the time. Initial arrangement of change data initial arrangement The initial arrangement of time change data in the virtual space determined by the determination unit, the animated human body shape model and timeline generated by the animation human body shape model generation unit, and the stop motion model group generation. Based on the stop motion model group generated by the unit, the time change data position information indicating the position of the time change data in the virtual space and the human body shape model showing the position of the animated human body shape model in the virtual space. Placement moving unit that generates position information and stop motion model group position information indicating the position of the stop motion model group in the virtual space, time change data position information generated by the placement movement part, and human body whole body shape model position information. It also includes a stop motion model group position information and an image generation unit that generates an image presented to the user by the image presentation unit based on the position and orientation of the image presentation unit that presents the image to the user.

(2) In the human body shape model visualization system according to (1) above, the time change data is time-series physiological measurement data obtained by the physiological measurement device, and the image generation unit has elapsed time from the start of reproduction. The value (graph) of the physiological measurement data at that time and the animated human body shape model at that time are placed on the timeline showing, and the animated human body shape model placed on the timeline is , The posture of the human body at that time may be shown.

(3) In the human body shape model visualization system according to (1) or (2) above, the stop motion model generation time-determining unit interacts with the user of the setting file, the external program, and the video presentation unit. The generation interval of the stop motion model in the virtual space may be determined based on at least one of the above.

(4) In the human body shape model visualization system according to any one of (1) to (3) above, the stop motion model generation time point determination unit determines the time point extracted from the human body motion data based on the rule. It may be determined as the generation time of the stop motion model in the virtual space.

(5) In the human body shape model visualization system according to any one of (1) to (4) above, when the reproduction operation command unit sends a forward reproduction operation command, the arrangement moving unit is subjected to. Move the animated human body shape model and the timeline showing the passage of time from the start of playback to the right, or move the waveform of the time change data and the stop motion model group generated by the stop motion model group generation unit to the left. You may move it to.

(6) In the human body shape model visualization system according to any one of (1) to (5) above, the human body whole body shape model data acquired by the data registration unit is a shape model created by imitating a person. It may be the data or the shape model data obtained by converting the point group data obtained by the shape measuring machine.

(7) In the human body shape model visualization system according to any one of (1) to (6) above, the human body motion data acquired by the data registration unit is time series data obtained by the human body motion measuring device. There may be.

(8) In the human body shape model visualization system according to (1) above, the time change data acquired by the data registration unit may be time series data obtained by an external program.

(9) In the human body shape model visualization system according to (1) above, the reproduction operation command unit reproduces according to the operation of the user of the image presentation unit and / or according to a command from an external program. The reproduction operation command that sends an operation command and is transmitted by the reproduction operation command unit includes a forward reproduction operation command, a reverse reproduction operation command, a forward time-lapse reproduction operation command, and a reverse-direction time-lapse reproduction operation. At least one of an instruction, a forward speed-up playback operation command, a reverse speed-up playback operation command, a pause command, a stop command, and a loop playback command may be included.

(10) The human body shape model visualization method according to one aspect of the present invention includes a data registration step for acquiring human body shape model data, human body motion data, and time change data, and time change data acquired in the data registration step. Based on the time change data initial placement determination step that determines the initial placement of the time change data in the virtual space and the human body whole body shape model data acquired in the data registration step, the human body whole body shape model in the virtual space. The stop that determines the generation time of the human body shape model initial placement determination step that determines the initial placement of the human body, and the stop motion model that is generated based on the human body shape model data and the human body motion data acquired in the data registration step. The human body determined in the human body whole body shape model initial placement determination step according to the motion model generation time point determination step, the reproduction operation command step for transmitting the reproduction operation command, and the reproduction operation command transmitted in the reproduction operation command step. By applying the human body motion data acquired in the data registration step to the initial arrangement of the whole body shape model, an animated human body shape model is generated and a timeline showing the passage of time from the start of reproduction is generated. It is sent in the animation human body whole body shape model generation step, the stop motion model group generation step for generating the stop motion model group based on the generation time determined in the stop motion model generation time determination step, and the reproduction operation command step. The playback operation command, the initial arrangement of the time change data in the virtual space determined in the time change data initial arrangement determination step, and the animated human body whole body shape model generated in the animated human body shape model generation step. Based on the timeline and the stop motion model group generated in the stop motion model group generation step, the time change data position information indicating the position of the time change data in the virtual space and the animated human body in the virtual space. An arrangement movement step that generates a human body whole body shape model position information indicating the position of the whole body shape model and a stop motion model group position information indicating the position of the stop motion model group in the virtual space, and an arrangement movement step generated in the arrangement movement step. Time change data The image presented to the user by the image presenting unit based on the data position information, the human body whole body shape model position information, the stop motion model group position information, and the position and posture of the image presenting unit that presents the image to the user. It is provided with a video generation step to generate.

(11) The program according to one aspect of the present invention includes a data registration step for acquiring human body shape model data, human body motion data, and time change data on a computer, and time change data acquired in the data registration step. Based on the time change data initial placement determination step that determines the initial placement of the time change data in the virtual space and the human body whole body shape model data acquired in the data registration step, the human body whole body shape model in the virtual space Human body whole body shape model that determines the initial placement Stop motion that determines the generation time of the stop motion model generated based on the initial placement decision step and the human body whole body shape model data and human body motion data acquired in the data registration step. The whole human body determined in the human body shape model initial placement determination step according to the model generation time point determination step, the reproduction operation command step for transmitting the reproduction operation command, and the reproduction operation command transmitted in the reproduction operation command step. By applying the human body motion data acquired in the data registration step to the initial arrangement of the shape model, an animated human body whole body shape model is generated, and an animation showing the passage of time from the start of playback is generated. It was sent in the human body shape model generation step, the stop motion model group generation step for generating the stop motion model group based on the generation time determined in the stop motion model generation time determination step, and the reproduction operation command step. The reproduction operation command, the initial arrangement of the time change data in the virtual space determined in the time change data initial arrangement determination step, and the animated human body shape model and time generated in the animated human body shape model generation step. Based on the line and the stop motion model group generated in the stop motion model group generation step, the time change data position information indicating the position of the time change data in the virtual space and the animated whole human body in the virtual space. A placement movement step that generates position information of the human body shape model indicating the position of the shape model and stop motion model group position information indicating the position of the stop motion model group in the virtual space, and a time generated in the placement movement step. change Based on the data position information, the human body whole body shape model position information, the stop motion model group position information, and the position and posture of the image presentation unit that presents the image to the user, the image presented by the image presentation unit to the user is displayed. It is for executing the image generation step to be generated.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to provide a human body shape model visualization system, a human body shape model visualization method and a program capable of presenting motion and time change data related to motion in an easy-to-understand manner.

It is a figure which shows an example such as the structure of the human body shape model visualization system of an embodiment. It is a figure for demonstrating the image generated by the image generation part. It is a figure for demonstrating the image at the time t5 generated by the image generation part. It is a figure for demonstrating the image at the time of completion of reproduction generated by an image generation part. It is a figure for demonstrating the image just before the start of reproduction which the image presenting part presents to a user. It is a figure for demonstrating another video just before the start of reproduction which a video presentation part presents to a user. It is a figure for demonstrating another video which a video presentation part presents to a user. It is a flowchart for demonstrating the process performed by a human body shape model visualization system. It is a figure for demonstrating a series of photographs taken by a Murray photograph gun.

Hereinafter, embodiments of the human body shape model visualization system, the human body shape model visualization method, and the program of the present invention will be described in detail with reference to the figures as appropriate. In the drawings used in the following description, in order to make the features of the present invention easy to understand, the featured portions may be enlarged for convenience, and the dimensional ratios of each component may differ from the actual ones. There is. The materials, dimensions, and the like exemplified in the following description are examples, and the present invention is not limited thereto, and can be appropriately modified and carried out within the range in which the effects of the present invention are exhibited.

FIG. 1 is a diagram showing an example of the configuration of the human body shape model visualization system 1 of the embodiment.
In the example shown in FIG. 1, the human body shape model data 110, the human body motion data 120, and the time change data 130 are configured to be input to the human body shape model visualization system 1 via the AND calculation unit 150.
The human body whole body shape model data 110 input to the human body shape model visualization system 1 is, for example, shape model data created by imitating a person.
In another example, the shape obtained by converting the point cloud data obtained by the shape measuring machine (not shown) such as a digitizer by the human body whole body shape model data 110 input to the human body shape model visualization system 1. It may be model data.

In the example shown in FIG. 1, the human body motion data 120 input to the human body shape model visualization system 1 is, for example, time-series data such as each joint angle obtained by a human body motion measuring device (not shown).
The time change data 130 input to the human body shape model visualization system 1 is, for example, time-series data obtained by an external program. The time-series data obtained by the external program is, for example, the time-series physiology measurement data obtained by the physiology measurement device. As the time-series physiological measurement data obtained by the physiological measurement device, for example, there are time-series data such as myoelectricity, heartbeat, respiratory metabolism, and calorie consumption.
In another example, as the time change data 130, the state of mood during exercise recorded by the person to be measured for the human body movement data and the time change data using a portable terminal (not shown) during the measurement, etc. The data indicating the above may be input to the human body shape model visualization system 1.

In the example shown in FIG. 1, the image presenting unit 200 operates an animated human body shape model generated by the human body shape model visualization system 1 based on the human body shape model data 110 and the human body motion data 120. An image of a moving image of a human body shape model) is presented to a user of the human body shape model visualization system 1 and the image presentation unit 200.
Further, the image presentation unit 200 is a stop motion model generated by the human body shape model visualization system 1 based on the human body shape model data 110 and the human body motion data 120 (a moving human body whole body shape model is displayed at a predetermined timing). Present the image of (still) to the user.
Further, the image presentation unit 200 presents to the user an image of the waveform of the time change data generated by the human body shape model visualization system 1 based on the time change data 130.
The image presentation unit 200 is, for example, a known VR (Virtual Reality) device or AR (Augmented Reality) device. Information on the position and posture of the image presentation unit 200 is input to the human body shape model visualization system 1. When the position or posture of the image presenting unit 200 changes with the change of the position or posture of the user who wears the image presenting unit 200, the angle of the image presented to the user by the image presenting unit 200 changes.
In order to detect the position and orientation of the image presenting unit 200, for example, a technique such as a VR tracking technique "Lighthouse" using an external sensor and SLAM (Simultaneous Localization and Mapping) using a built-in camera is used.

In the example shown in FIG. 1, the human body shape model visualization system 1 has a data registration unit 1A, a time change data initial placement determination unit 1B, a human body whole body shape model initial placement determination unit 1C, and a stop motion model generation time point determination unit 1D. , A reproduction operation command unit 1E, an animation human body whole body shape model generation unit 1F, a stop motion model group generation unit 1G, an arrangement movement unit 1H, and an image generation unit 1I.
The data registration unit 1A acquires the human body shape model data 110, the human body motion data 120, and the time change data 130 input to the human body shape model visualization system 1.

The time change data initial arrangement determination unit 1B is connected to the data registration unit 1A. The time change data initial arrangement determination unit 1B determines the initial arrangement of the time change data in the virtual space based on the time change data 130 acquired by the data registration unit 1A. The virtual space is a space displayed on the display of a VR device or an AR device.
That is, the time change data initial arrangement determination unit 1B is the initial position of the time change data waveform in the space displayed on the display of the VR device or the AR device (specifically, for example, the value of the time change data waveform at time t0). Display position) is determined.

The human body whole body shape model initial placement determination unit 1C is connected to the data registration unit 1A. The human body whole body shape model initial arrangement determination unit 1C determines the initial arrangement of the human body whole body shape model in the virtual space based on the human body whole body shape model data 110 acquired by the data registration unit 1A.
That is, the human body whole body shape model initial placement determination unit 1C is the initial position of the human body whole body shape model in the space displayed on the display of the VR device or the AR device (specifically, for example, the display position of the human body whole body shape model at time t0). To decide.

The stop motion model generation time point determination unit 1D is connected to the data registration unit 1A. The stop motion model generation time determination unit 1D is a stop motion model generation time point (for example, time t0, t1, t2, etc.) generated based on the human body shape model data 110 and the human body motion data 120 acquired by the data registration unit 1A. ) Is determined.
The stop motion model generation time point determination unit 1D determines, for example, the generation interval of the stop motion model (for example, generation interval t1-t0, t2-t1, etc.) based on the setting file.
In another example, the stop motion model generation time determination unit 1D may determine the stop motion model generation interval based on an external program, and is based on an operation in a dialogue with the user (for example, a user's request). The stop motion model generation interval may be determined.
Alternatively, the stop motion model generation time point determination unit 1D may determine the time point extracted from the human body motion data 120 based on the rule as the stop motion model generation time point.
The stop motion model generation time point determination unit 1D may combine them to determine the stop motion model generation interval.

In the example shown in FIG. 1, the reproduction operation command unit 1E sends a reproduction operation command. The reproduction operation command is a command for instructing a reproduction operation such as the above-mentioned animated human body whole body shape model (moving image of a moving human body whole body shape model).
The reproduction operation command unit 1E sends a reproduction operation command according to the operation of the user of the human body shape model visualization system 1 and the image presentation unit 200.
In another example (for example, when the human body shape model visualization system 1 is incorporated in the archive viewing system), the reproduction operation command unit 1E may send a reproduction operation command in response to a command from an external program. The reproduction operation command unit 1E may send a reproduction operation command in response to both the user's operation and the command from the external program.

In the example shown in FIG. 1, the reproduction operation command sent by the reproduction operation command unit 1E includes, for example, a forward reproduction operation command, a reverse reproduction operation instruction, a forward time-lapse reproduction operation command, and a reverse direction. The time-lapse playback operation command, the forward speed-up playback operation command, the reverse speed-speed playback operation command, the pause command, the stop command, and the loop playback command are included.
In another example, the reproduction operation command transmitted by the reproduction operation command unit 1E includes at least one of the above-mentioned instructions.

In the example shown in FIG. 1, the animation human body shape model generation unit 1F is connected to the human body body shape model initial placement determination unit 1C and the reproduction operation command unit 1E.
The animated human body whole body shape model generation unit 1F applies the human body motion data 120 to the initial arrangement of the human body whole body shape model in response to the reproduction operation command sent by the reproduction operation command unit 1E, thereby animating the human body whole body shape model. Generates (moving image of a moving human body shape model). The animated human body shape model is displayed on the display of the VR device or AR device as described later.
Further, the animation human body whole body shape model generation unit 1F generates a timeline showing the passage of time from the start of reproduction. The timeline is displayed on the display of the VR device or AR device as described later. Further, the position of the timeline in the virtual space displayed on the display of the VR device or the AR device changes with the passage of time after the start of reproduction.

The stop motion model group generation unit 1G is connected to the stop motion model generation time determination unit 1D. The stop motion model group generation unit 1G generates a stop motion model group based on the generation time determined by the stop motion model generation time determination unit 1D.
For example, when the stop motion model generation time determination unit 1D determines the time t0, the time t1 and the time t2 as the generation time of the stop motion model, the stop motion model group generation unit 1G is an animated whole human body. A shape model (moving image (statue) of a moving human body whole body shape model) stopped at time t0 (still image of a human body whole body shape model) is generated as a stop motion model at time t0. Further, the stop motion model group generation unit 1G generates an animated human body whole body shape model stopped at time t1 as a stop motion model at time t1, and stops the animated human body whole body shape model at time t2. Is generated as a stop motion model at time t2.
Further, when the stop motion model generation time determination unit 1D determines the time t0, the time t1 and the time t2 as the generation time of the stop motion model, the stop motion generated by the stop motion model group generation unit 1G The model group is composed of a stop motion model at time t0, a stop motion model at time t1, and a stop motion model at time t2.

The arrangement movement unit 1H is connected to a time change data initial arrangement determination unit 1B, a reproduction operation command unit 1E, an animation human body whole body shape model generation unit 1F, and a stop motion model group generation unit 1G.
That is, in the arrangement moving unit 1H, the reproduction operation command sent by the reproduction operation command unit 1E, the initial arrangement of the time change data in the virtual space determined by the time change data initial arrangement determination unit 1B, and the animation human body whole body shape model generation. The animated human body shape model and timeline generated by the unit 1F and the stop motion model group generated by the stop motion model group generation unit 1G are input.
Based on the input, the arrangement moving unit 1H indicates the time change data position information indicating the position of the time change data in the virtual space, and the human body shape model position indicating the position of the animated human body shape model in the virtual space. Information and stop motion model group position information indicating the position of the stop motion model group in the virtual space are generated.
The time change data position information is, for example, information on where in the virtual space displayed on the display of the VR device or the AR device the waveform of the time change data should be placed.
The human body shape model position information is, for example, where in the virtual space displayed on the display of the VR device or AR device, for example, an animated human body shape model (moving image of the human body shape model) at time t3 should be placed. This is information about the model.
The stop motion model group position information is, for example, where in the virtual space displayed on the display of the VR device or AR device, the stop motion model at time t0, the stop motion model at time t1, and the stop motion model at time t2. Information on whether to place.

In the example shown in FIG. 1, when the reproduction operation command unit 1E sends a forward reproduction operation command, the arrangement moving unit 1H has an animated human body whole body shape model and a timeline showing the passage of time from the start of reproduction. To the right.
In another example, when the reproduction operation command unit 1E sends a forward reproduction operation command, the arrangement moving unit 1H has a waveform of time change data and a stop motion model group generated by the stop motion model group generation unit 1G. May be moved to the left.

In the example shown in FIG. 1, the video generation unit 1I is connected to the arrangement movement unit 1H.
That is, the time change data position information generated by the arrangement movement unit 1H, the human body whole body shape model position information, and the stop motion model group position information are input to the image generation unit 1I. Further, information on the position and posture of the video presentation unit 200 is input to the video generation unit 1I.
In the image generation unit 1I, the image presentation unit 200 informs the user based on the time change data position information, the human body whole body shape model position information, the stop motion model group position information, and the position and posture of the image presentation unit 200. Generate the video to be presented.

FIG. 2 is a diagram for explaining an image generated by the image generation unit 1I. In detail, FIG. 2 shows an image at time t3 generated by the image generation unit 1I.
In the example shown in FIG. 2, the video generated by the video generation unit 1I includes a time change data waveform TD1, a time change data waveform TD2, a time change data waveform TD3, and a time change data waveform TD4. It has been. Waveforms of time change data TD1, TD2, TD3, TD4 are waveforms of time-series physiological measurement data obtained by a physiological measurement device. Specifically, the waveforms TD1 to TD4 of the time change data show, for example, an electromyographic waveform. The waveform TD1 of the time change data shows, for example, the myoelectric waveform of the gluteus maximus muscle. The waveform TD2 of the time change data shows, for example, the myoelectric waveform of the quadriceps muscle. The waveform TD3 of the time change data shows, for example, the myoelectric waveform of hamstrings. The waveform TD4 of the time change data shows, for example, the myoelectric waveform of the tibialis anterior muscle.
Further, the video generated by the video generation unit 1I includes a timeline TL indicating the passage of time from the start of playback (time t0). The timeline TL in FIG. 2 indicates that the time (t3-t0) has elapsed from the start of reproduction (time t0).
The intersection of the waveform TD1 of the time change data and the timeline TL indicates the value of the gluteus maximus muscle at the time t3. The intersection of the waveform TD2 of the time change data and the timeline TL indicates the value of the quadriceps muscle at the time t3. The intersection of the waveform TD3 of the time change data and the timeline TL indicates the value of the hamstrings myoelectricity at the time t3. The intersection of the waveform TD4 of the time change data and the timeline TL indicates the value of the myoelectricity of the tibialis anterior muscle at time t3.
Further, on the timeline TL, an animated human body shape model AM at the time point t3 is arranged. The animated human body shape model AM shows the posture of the human body at time t3.

Further, in the example shown in FIG. 2, the video generated by the video generation unit 1I includes a stop motion model SMt0 at time t0, a stop motion model SMt1 at time t1, and a stop motion model SMt2 at time t2. ..
As described above, the stop motion model SMt0 corresponds to an animated human body whole body shape model (moving image of a moving human body whole body shape model) AM stopped at time t0. The stop motion model SMt1 corresponds to the animated human body shape model AM stopped at time t1, and the stop motion model SMt2 corresponds to the animated human body shape model AM stopped at time t2. do.
In the example shown in FIG. 2, the generation intervals (generation intervals t1-t0, t2-t1) of the stop motion models SMt0, SMt1, and SMt2 in the virtual space are fixed time intervals (for example, 0. 3 seconds) has been decided.

FIG. 3 is a diagram for explaining an image at time t5 generated by the image generation unit 1I.
In the example shown in FIG. 3, the position of the image presenting unit 200 is different from that in the example shown in FIG. Further, in the example shown in FIG. 3, the time point when the stop motion model generation time determination unit 1D is extracted from the human body motion data 120 based on the rule (for example, the time point when the heel coordinates are used and only the moment of landing is extracted (for example). The time t2, t3, t4)) is determined as the generation time of the stop motion models SMt2, SMt3, SMt4 in the virtual space.
With such a configuration, for example, a stop motion model based on the needs of users (coaches and athletes) can be (semi) automatically generated, and its observation can be made more efficient.
There are cases where the movement phase that the user wants to pay attention to can change depending on the subject of exercise or depending on the researcher, that is, there are cases where you want to observe the moment of heel contact and the moment of kicking even in the same running subject. Can be done.
The moment when the stop motion model is generated is not only the fixed time interval specified as a parameter, but also the moment specified based on the rule (for example, the moment of heel contact in the running motion based on the coordinates of the vertical axis of the ankle joint). It can be extracted and adopted.
This makes it possible to (semi) automatically generate a stop motion model of the heel contact of a characteristic motion during exercise, for example, a sprinting motion, and the generation interval can be arbitrarily changed after the measurement. ..

FIG. 4 is a diagram for explaining a video generated by the video generation unit 1I at the time of completion of reproduction.
In the example shown in FIG. 4, the position of the image presenting unit 200 is different from that in the examples shown in FIGS. 2 and 3. Further, in the example shown in FIG. 4, since the reproduction of the moving image is completed, the animated human body whole body shape model AM, that is, the moving human body whole body shape model does not exist in the video. As a result, there is no timeline TL in the video.

FIG. 5 is a diagram for explaining an image immediately before the start of reproduction presented to the user by the image presenting unit 200. In detail, FIG. 5 shows the field of view of the head-mounted display wearer.
In the example shown in FIG. 5, the position of the image presenting unit 200 is different from that in the examples shown in FIGS. 2 to 4. Further, in the example shown in FIG. 5, since the reproduction of the moving image has not started yet, the animated human body shape model AM, that is, the moving human body shape model does not exist in the video. As a result, there is no timeline TL in the video.
As described above, the video shown in FIG. 5 is not a video generated by the video generation unit 1I, but a video presented to the user by the video presentation unit 200. Therefore, FIG. 5 includes an image for the left eye of the user (image in the circle on the left side of FIG. 5) and an image for the right eye of the user (image in the circle on the right side of FIG. 5). Is done.

FIG. 6 is a diagram for explaining another video presented to the user by the video presentation unit 200 immediately before the start of reproduction (that is, at the same time as the video shown in FIG. 5).
In the example shown in FIG. 6, the position of the image presenting unit 200 is different from that in the example shown in FIG. Further, in the example shown in FIG. 6, similarly to the example shown in FIG. 5, the animated human body shape model AM does not exist and the timeline TL does not exist in the video.
Further, in FIG. 6, as in FIG. 5, the image for the left eye of the user (the image in the circle on the left side of FIG. 6) and the image for the right eye of the user (inside the circle on the right side of FIG. 6). Video) and is included.

FIG. 7 is a diagram for explaining another video presented to the user by the video presentation unit 200.
The image shown in FIG. 7 includes the stop motion model SMt3 included in the image shown in FIG. In the example shown in FIG. 7, the image presenting unit 200 is approaching the stop motion model SMt3, for example, in order to confirm the landing heel.
In the examples shown in FIGS. 5 to 7, by combining the image presenting unit 200 (HMD) that presents the image according to the position and posture of the user's head, the exercise that is being performed right in front of the user is observed. As is the case, the user can observe while intuitively and freely selecting a viewpoint. As a result, the user can observe the points of interest such as the extension angle of the knee joint and the angle of the ankle at the moment of landing from a free viewpoint in running, for example.

FIG. 8 is a flowchart for explaining the process executed by the human body shape model visualization system 1.
In the example shown in FIG. 8, in step S11, the data registration unit 1A acquires the human body whole body shape model data 110, the human body motion data 120, and the time change data 130.
Next, in step S12, the time change data initial arrangement determination unit 1B initially arranges the time change data waveforms TD1, TD2, TD3, and TD4 in the virtual space based on the time change data 130 (that is, the waveform of the time change data). The leftmost position of TD1, TD2, TD3, and TD4) is determined.
Next, in step S13, the human body whole body shape model initial placement determination unit 1C moves the initial placement of the human body whole body shape model in the virtual space (that is, the animated human body whole body shape model AM) based on the human body whole body shape model data 110. Determine the starting position).
Next, in step S14, the stop motion model generation time determination unit 1D generates stop motion models SMt0, SMt1, SMt2 (time t0, t1) based on the human body shape model data 110 and the human body motion data 120. , T2) is determined.

Next, in step S15-1, the reproduction operation command unit 1E determines whether or not there is a user's operation or a command from an external program. If there is a user operation or a command from an external program, the process proceeds to step S15-2. If there is no user operation or command from the external program, step S15-1 is repeatedly executed until there is a user operation or a command from the external program.
In step S15-2, the reproduction operation command unit 1E sends a reproduction operation command.
Next, in step S16, the animation human body whole body shape model generation unit 1F acquires the data registration unit 1A for the initial arrangement of the human body whole body shape model determined by the human body whole body shape model initial arrangement determination unit 1C in response to the reproduction operation command. By applying the human body motion data 120, an animated human body shape model AM is generated, and a timeline TL showing the passage of time from the start of reproduction is generated.
Next, in step S17, the stop motion model group generation unit 1G determines the stop motion model group (stop motion model SMt0, t2) based on the generation time (time t0, t1, t2) determined by the stop motion model generation time determination unit 1D. A group of SMt1 and SMt2) is generated.

Next, in step S18, the arrangement moving unit 1H receives a reproduction operation command, an initial arrangement of the waveforms TD1, TD2, TD3, and TD4 of the time change data in the virtual space, and an animated human body shape model AM and a timeline TL. , Time change data position information indicating the positions of the waveforms TD1, TD2, TD3, and TD4 of the time change data in the virtual space based on the stop motion model group (group of stop motion models SMt0, SMt1, SMt2), and the virtual space. Human body shape model position information indicating the position of the animated human body shape model AM in, and stop motion model group position information indicating the position of the stop motion model group (stop motion model SMt0, SMt1, SMt2 group) in the virtual space. And generate.
Next, in step S19, the image generation unit 1I presents the time change data position information, the human body whole body shape model position information, the stop motion model group position information generated by the arrangement movement unit 1H, and the image presentation unit to the user. Based on the position and posture of the 200, the image presenting unit 200 generates an image to be presented to the user.

Specifically, for example, as shown in FIG. 2, the video generation unit 1I is on the timeline TL showing the passage of time from the start of reproduction, and the time change data (time t3 in the example shown in FIG. 2) at that time (time t3 in the example shown in FIG. 2). The values of the waveforms TD1, TD2, TD3, and TD4 of the physiological measurement data) and the animated human body shape model AM at that time are arranged. Further, the animated human body shape model AM arranged on the timeline TL shows the posture of the human body at that time point (time t3).
That is, the example shown in FIG. 2 specifies a unique arrangement method of the physiological measurement data in the virtual space, the timeline TL showing the elapsed time from the start of reproduction, and the animated human body shape model AM.
Therefore, the video presentation unit 200 can present the exercise and the time change data related to the exercise to the user in an easy-to-understand manner. The user can easily grasp the exercise and the time change data related to the exercise.
By presenting the physiological measurement data as waveforms TD1, TD2, TD3, and TD4 of the time change data, the user can easily gaze at the human body movement and the physiological information at the same time as long as the user pays attention to the timeline TL. Become.

In the example shown in FIG. 2, the animated human body shape model AM moves from the left side of FIG. 2 to the right side of FIG. Therefore, in the example shown in FIG. 2, when the reproduction operation command unit 1E sends a forward reproduction operation command, the arrangement moving unit 1H shows the animated human body shape model AM and the passage of time from the start of reproduction. The timeline TL is moved to the right in FIG. Further, the arrangement moving unit 1H arranges the stop motion models SMt0, SMt1 and SMt2 at the positions after the animated human body shape model AM has passed. That is, the stop motion models SMt0, SMt1, and SMt2 are not arranged (displayed) immediately after the start of reproduction, and are arranged (displayed) as soon as the animated human body shape model AM passes through.
On the other hand, for example, when the animated human body shape model AM performs a "vertical jump", the arrangement moving portion 1H tentatively shares the animated human body shape model AM and the stop motion models SMt0, SMt1, and SMt2. If they are placed in the position, they will overlap and the image will be difficult to see.
Therefore, for example, when the animated human body shape model AM performs a vertical jump, the arrangement moving unit 1H has waveforms TD1, TD2, TD3, TD4 of time change data, and a stop motion model group (stop motion model SMt0). , SMt1, SMt2 group) and to the left. Therefore, even if the position of the animated human body shape model AM does not change, the animated human body shape model AM and the stop motion models SMt0, SMt1, and SMt2 do not overlap. As a result, the image can be easily viewed.

As described above, the human body shape model visualization system 1 of the embodiment applies the XR technique in order to visualize the motion / physiological measurement data in an easy-to-understand manner.

The human body shape model visualization system, the human body shape model visualization method and the program of the present invention can be widely used in the fields of life / welfare / healthcare, sports / health support and the like.

1 ... Human body shape model visualization system 1A ... Data registration unit 1B ... Time change data initial placement determination unit 1C ... Human body whole body shape model initial placement determination unit 1D ... Stop motion model generation time point determination unit 1E ... Playback operation command unit 1F ... Animation human body Whole body shape model generation unit 1G ... Stop motion model group generation unit 1H ... Arrangement movement unit 1I ... Image generation unit 110 ... Human body whole body shape model data 120 ... Human body motion data 130 ... Time change data 150 ... AND calculation unit 200 ... Video presentation unit

Claims (10)

A data registration unit that acquires human body shape model data, human body motion data, and time change data,
Based on the time change data acquired by the data registration unit, the time change data initial placement determination unit that determines the initial placement of the time change data in the virtual space, and the time change data initial placement determination unit.
Based on the human body whole body shape model data acquired by the data registration unit, the human body whole body shape model initial arrangement determination unit that determines the initial arrangement of the human body whole body shape model in the virtual space, and the human body whole body shape model initial arrangement determination unit.
A stop motion model generation time determination unit that determines the generation time of the stop motion model generated based on the human body shape model data and the human body motion data acquired by the data registration unit, and the stop motion model generation time determination unit.
A playback operation command unit that sends a playback operation command,
Applying the human body motion data acquired by the data registration unit to the initial arrangement of the human body shape model determined by the human body whole body shape model initial arrangement determination unit in response to the reproduction operation command sent by the reproduction operation command unit. An animated human body shape model generator that generates an animated human body shape model and a timeline that shows the passage of time from the start of playback.
A stop motion model group generation unit that generates a stop motion model group based on the generation time determined by the stop motion model generation time determination unit.
The reproduction operation command sent by the reproduction operation command unit, the initial arrangement of the time change data in the virtual space determined by the time change data initial arrangement determination unit, and the animation generated by the animation human body whole body shape model generation unit were performed. Based on the human body whole body shape model and timeline, and the stop motion model group generated by the stop motion model group generation unit, the time change data position information indicating the position of the time change data in the virtual space and the time change data position information in the virtual space. An arrangement moving unit that generates an animated human body whole body shape model position information indicating the position of the human body whole body shape model and a stop motion model group position information indicating the position of the stop motion model group in the virtual space.
The image presentation is based on the time change data position information, the human body shape model position information, the stop motion model group position information generated by the arrangement moving unit, and the position and posture of the image presentation unit that presents the image to the user. It is equipped with a video generation unit that generates the video presented to the user by the department .
The time change data includes time series data of myoelectricity obtained by a physiological measuring device.
The video generation unit arranges the EMG value at that time and the animated human body shape model at that time on the timeline showing the passage of time from the start of reproduction.
An animated human body shape model placed on the timeline shows the posture of the human body at that time.
The image presented to the user by the image presenter includes a moving image of an animated moving human body shape model placed on a timeline whose position changes over time.
Used for sports coaching,
Human body shape model visualization system. The stop motion model generation time determination unit determines the stop motion model generation interval in the virtual space based on at least one of a setting file, an external program, and an interactive operation with the user of the video presentation unit. ,
The human body shape model visualization system according to claim 1. The stop motion model generation time determination unit determines the time point extracted from the human body motion data based on the rule as the generation time point of the stop motion model in the virtual space.
The human body shape model visualization system according to claim 1 or 2. When the reproduction operation command unit sends a forward reproduction operation command, the arrangement movement unit moves the animated human body shape model and the timeline showing the passage of time from the start of reproduction to the right. , The waveform of the time change data and the stop motion model group generated by the stop motion model group generation unit are moved to the left.
The human body shape model visualization system according to any one of claims 1 to 3. The human body whole body shape model data acquired by the data registration unit is shape model data created by imitating a person or shape model data obtained by converting point cloud data obtained by a shape measuring machine. be,
The human body shape model visualization system according to any one of claims 1 to 4. The human body motion data acquired by the data registration unit is time-series data obtained by the human body motion measuring device.
The human body shape model visualization system according to any one of claims 1 to 5. The time change data acquired by the data registration unit is time series data obtained by an external program.
The human body shape model visualization system according to claim 1 . The playback operation command unit sends a playback operation command in response to an operation by a user of the video presentation unit and / or in response to a command from an external program.
The reproduction operation command sent by the reproduction operation command unit includes a forward reproduction operation command, a reverse reproduction operation command, a forward time-lapse reproduction operation command, a reverse time-lapse operation instruction, and a forward increase. Includes at least one of a fast play operation command, a reverse speed increase play operation command, a pause command, a stop command, and a loop play command.
The human body shape model visualization system according to claim 1. A data registration step to acquire human body shape model data, human body motion data, and time change data,
Based on the time change data acquired in the data registration step, the time change data initial placement determination step for determining the initial placement of the time change data in the virtual space, and the time change data initial placement determination step.
Based on the human body whole body shape model data acquired in the data registration step, the human body whole body shape model initial placement determination step for determining the initial placement of the human body whole body shape model in the virtual space, and the human body whole body shape model initial placement determination step.
A stop motion model generation time point determination step for determining the generation time point of the stop motion model generated based on the human body shape model data and the human body motion data acquired in the data registration step, and the stop motion model generation time point determination step.
A playback operation command step that sends a playback operation command, and
In response to the reproduction operation command sent in the reproduction operation command step, the human body motion data acquired in the data registration step is added to the initial arrangement of the human body whole body shape model determined in the human body whole body shape model initial arrangement determination step. By applying, an animated human body shape model generation step that generates an animated human body shape model and a timeline showing the passage of time from the start of playback,
A stop motion model group generation step for generating a stop motion model group based on the generation time determined in the stop motion model generation time point determination step, and a stop motion model group generation step.
The reproduction operation command sent in the reproduction operation command step, the initial arrangement of the time change data in the virtual space determined in the time change data initial arrangement determination step, and the animation human body whole body shape model generation step generated. Based on the animated human body shape model and timeline, and the stop motion model group generated in the stop motion model group generation step, the time change data position information indicating the position of the time change data in the virtual space, and the time change data position information. A placement movement step that generates a human body shape model position information indicating the position of the animated human body shape model in the virtual space and a stop motion model group position information indicating the position of the stop motion model group in the virtual space.
The image is based on the time change data position information, the human body shape model position information, the stop motion model group position information generated in the arrangement movement step, and the position and posture of the image presentation unit that presents the image to the user. It is equipped with a video generation step of generating a video presented to the user by the presentation unit.
The time change data includes time series data of myoelectricity obtained by a physiological measuring device.
In the video generation step, the EMG value at that time and the animated human body shape model at that time are arranged on the timeline showing the passage of time from the start of playback.
An animated human body shape model placed on the timeline shows the posture of the human body at that time.
The image presented to the user by the image presenter includes a moving image of an animated moving human body shape model placed on a timeline whose position changes over time.
Used for sports coaching,
Human body shape model visualization method. On the computer
A data registration step to acquire human body shape model data, human body motion data, and time change data,
Based on the time change data acquired in the data registration step, the time change data initial placement determination step for determining the initial placement of the time change data in the virtual space, and the time change data initial placement determination step.
Based on the human body whole body shape model data acquired in the data registration step, the human body whole body shape model initial placement determination step for determining the initial placement of the human body whole body shape model in the virtual space, and the human body whole body shape model initial placement determination step.
A stop motion model generation time point determination step for determining the generation time point of the stop motion model generated based on the human body shape model data and the human body motion data acquired in the data registration step, and the stop motion model generation time point determination step.
A playback operation command step that sends a playback operation command, and
In response to the reproduction operation command sent in the reproduction operation command step, the human body motion data acquired in the data registration step is added to the initial arrangement of the human body whole body shape model determined in the human body whole body shape model initial arrangement determination step. By applying, an animated human body shape model generation step that generates an animated human body shape model and a timeline showing the passage of time from the start of playback,
A stop motion model group generation step for generating a stop motion model group based on the generation time determined in the stop motion model generation time point determination step, and a stop motion model group generation step.
The reproduction operation command sent in the reproduction operation command step, the initial arrangement of the time change data in the virtual space determined in the time change data initial arrangement determination step, and the animation human body whole body shape model generation step generated. Based on the animated human body shape model and timeline, and the stop motion model group generated in the stop motion model group generation step, the time change data position information indicating the position of the time change data in the virtual space, and the time change data position information. A placement movement step that generates a human body shape model position information indicating the position of the animated human body shape model in the virtual space and a stop motion model group position information indicating the position of the stop motion model group in the virtual space.
The image is based on the time change data position information, the human body shape model position information, the stop motion model group position information generated in the arrangement movement step, and the position and posture of the image presentation unit that presents the image to the user. With the video generation step to generate the video presented to the user by the presentation unit
It is a program to execute
The time change data includes time series data of myoelectricity obtained by a physiological measuring device.
In the video generation step, the EMG value at that time and the animated human body shape model at that time are arranged on the timeline showing the passage of time from the start of playback.
An animated human body shape model placed on the timeline shows the posture of the human body at that time.
The image presented to the user by the image presenter includes a moving image of an animated moving human body shape model placed on a timeline whose position changes over time.
Used for sports coaching,
program.

Images