JP2024154850A - Information processing system, information processing method, and program - Google Patents

Abstract

To utilize data on motion of other users.SOLUTION: An information processing system includes: a storage unit which stores master motion information representing time series of partial or entire motion of a person or a thing; a motion acquisition unit which acquires first motion information on partial or entire motion of a user; a calculation unit which calculates a value of a predetermined parameter regarding similarity between the first motion information acquired by the motion acquisition unit and the master motion information stored in the storage unit; and an extraction processing unit which extracts one or more pieces of master motion information corresponding to the first motion information acquired by the motion acquisition unit, out of the multiple pieces of master motion information stored in the storage unit, on the basis of the value of the predetermined parameter calculated by the calculation unit.SELECTED DRAWING: Figure 9

Description

This disclosure relates to an information processing system, an information processing method, and a program.

A technology is known that analyzes a user's motion based on sensor data of a detection position detected by a sensor device worn by the user, and controls the display of an image of the user based on the results of the motion analysis.

"Shiftall, a small and wireless device that tracks whole-body movements "HaritoraX Wireless" - AV Watch" https://av.watch.impress.co.jp/docs/news/1464641.html posted on December 19, 2022

However, the above-mentioned conventional techniques do not assume that data on the movements of other users will be obtained.
Therefore, in one aspect, the present disclosure aims to utilize movement data of other users.

In one aspect, a storage unit stores master movement information representing a time series of a movement of a part or the whole of a person or an object;
A movement acquisition unit that acquires first movement information related to a movement of a part or an entirety of a user;
a calculation unit that calculates a value of a predetermined parameter related to a similarity between the first motion information acquired by the motion acquisition unit and the master motion information stored in the storage unit;
An information processing system is provided, comprising: an extraction processing unit that extracts one or more master motion information corresponding to the first motion information acquired by the motion acquisition unit from among the multiple master motion information stored in the memory unit based on the value of the specified parameter calculated by the calculation unit.

In one aspect, the present disclosure allows for the use of movement data from other users.

1 is a block diagram of a search support system according to an embodiment of the present invention. FIG. 2 is a schematic diagram illustrating an example of a hardware configuration of a server device. FIG. 2 is a schematic diagram illustrating an example of a hardware configuration of a terminal device. 1 is a flowchart illustrating an example of a process executed by the search support system. FIG. 4 is an explanatory diagram of the process shown in FIG. 3, and is an explanatory diagram of an example of the mounting position of the acceleration sensor. FIG. 4 is an explanatory diagram of the process shown in FIG. 3, illustrating an example of a joint. FIG. 4 is an explanatory diagram of the process shown in FIG. 3, showing the relationship between input and output. 10 is a flowchart illustrating another example of the flow of processes executed by the search support system. FIG. 1 is a block diagram illustrating an example of a search support system according to an embodiment of the present invention. FIG. 2 is an explanatory diagram illustrating an example of data in an avatar information storage unit. 11 is an explanatory diagram showing an example of data in a master movement information storage unit; FIG. FIG. 13 is an explanatory diagram of a profile image. FIG. 1 is an explanatory diagram (part 1) of animation content for automatic playback. FIG. 2 is an explanatory diagram (part 2) of animation content for automatic playback. FIG. 13 is an explanatory diagram relating to the motions of multiple avatars. FIG. 11 is an explanatory diagram of a hierarchical structure of master motion information.

Each embodiment will be described in detail below with reference to the attached drawings. Note that in the attached drawings, for ease of viewing, only some of the reference symbols may be assigned to multiple parts with the same attribute.

An overview of a search support system 1 according to one embodiment will be described with reference to FIG. 1. FIG. 1 is a block diagram of the search support system 1 according to this embodiment.

The search support system 1 includes a server device 10 and one or more terminal devices 20. For simplicity, one terminal device 20 is illustrated in FIG. 1, but the number of terminal devices 20 is arbitrary.

The server device 10 is, for example, an information processing system such as a server managed by one or more operators that provides the search support function according to this embodiment. The terminal device 20 is, for example, a device used by a user, such as a mobile phone, a smartphone, a tablet terminal, a PC (Personal Computer), a head-mounted display, or a game device. The terminal device 20 may include a wearable device. The wearable device may be a head-mounted display or a glasses-type device. The glasses-type device may be so-called AR (Augmented Reality) glasses or MR (Mixed Reality) glasses. In either case, the wearable device may be separate from the main body of the terminal device 20, or may realize some or all of the functions of the terminal device 20.

The terminal device 20 is capable of executing a search support application according to this embodiment. The search support application may be received by the terminal device 20 from the server device 10 or a specified application distribution server via the network 3, or may be stored in advance in a storage device provided in the terminal device 20 or a storage medium such as a memory card readable by the terminal device 20. The server device 10 and the terminal device 20 are connected to each other via the network 3 so as to be able to communicate with each other. For example, the server device 10 and the terminal device 20 cooperate to execute various processes related to the search support function according to this embodiment.

The terminal devices 20 are connected to each other so that they can communicate with each other via the server device 10. In the following, "one terminal device 20 transmits information to another terminal device 20" means "one terminal device 20 transmits information to another terminal device 20 via the server device 10". Similarly, "one terminal device 20 receives information from another terminal device 20" means "one terminal device 20 receives information from another terminal device 20 via the server device 10". However, in a modified example, each terminal device 20 may be connected to each other so that they can communicate with each other without going through the server device 10.

The network 3 may include a wireless communication network, the Internet, a Virtual Private Network (VPN), a Wide Area Network (WAN), a wired network, or any combination of these.

In the following, the search support system 1 realizes an example of an information processing system, but each element of a specific terminal device 20 may realize an example of an information processing system, or multiple terminal devices 20 may cooperate to realize an example of an information processing system. Also, the server device 10 may independently realize an example of an information processing system, or the server device 10 and one or more terminal devices 20 may cooperate to realize an example of an information processing system.

FIG. 2 is a schematic diagram showing an example of the hardware configuration of the server device 10. In FIG. 2, a peripheral device 129 is illustrated in association with the hardware configuration of the server device 10. The peripheral device 129 is optional, and may be a display, etc.

The server device 10 includes a CPU (Central Processing Unit) 111, a RAM (Random Access Memory) 112, a ROM (Read Only Memory) 113, an auxiliary storage device 114, a drive device 115, and a communication interface 117, all connected by a bus 119, as well as a wired transceiver unit 125 and a wireless transceiver unit 126 connected to the communication interface 117.

The auxiliary storage device 114 is, for example, a hard disk drive (HDD) or a solid state drive (SSD), and is a storage device that stores data related to application software, etc.

The wired transceiver 125 includes a transceiver capable of communicating over a wired network. Peripheral devices 129 are connected to the wired transceiver 125. However, some or all of the peripheral devices 129 may be connected to the bus 119 or to the wireless transceiver 126.

The wireless transceiver 126 is a transceiver capable of communicating using a wireless network. The wireless network may include the network 3 described above.

The server device 10 may be connectable to a recording medium 116. The recording medium 116 stores a specific program. The program stored in the recording medium 116 is installed in the auxiliary storage device 114 of the server device 10 via the drive device 115. The installed specific program can be executed by the CPU 111 of the server device 10. For example, the recording medium 116 may be a recording medium that records information optically, electrically or magnetically, such as a CD (Compact Disc)-ROM, a flexible disk, or a magneto-optical disk, or a semiconductor memory that records information electrically, such as a ROM or a flash memory.

Figure 3 is a schematic diagram showing an example of the hardware configuration of the terminal device 20. In Figure 3, a peripheral device 260 is illustrated in association with the hardware configuration of the server device 10.

The terminal device 20 includes a CPU 211, a RAM 212, a ROM 213, an auxiliary storage device 214, a drive device 215, and a communication interface 217, which are connected by a bus 219, as well as a wired transceiver unit 225 and a wireless transceiver unit 226 connected to the communication interface 217.

The auxiliary storage device 214 is, for example, a HDD or SSD, and is a storage device that stores data related to application software, etc.

The wired transceiver 225 includes a transceiver capable of communicating over a wired network. Peripheral devices 260 are connected to the wired transceiver 225. However, some or all of the peripheral devices 260 may be connected to the bus 219 or to the wireless transceiver 226.

The wireless transceiver 226 is a transceiver capable of communicating using a wireless network. The wireless network may include the network 3 described above.

The terminal device 20 may be connectable to a recording medium 216. The recording medium 216 stores a specific program. The program stored in the recording medium 216 is installed in the auxiliary storage device 214 of the terminal device 20 via the drive device 215. The installed specific program can be executed by the CPU 211 of the terminal device 20. For example, the recording medium 216 may be a recording medium that records information optically, electrically or magnetically, such as a CD-ROM, a flexible disk, or a magneto-optical disk, or a semiconductor memory that records information electrically, such as a ROM or a flash memory.

The peripheral device 260 is optional, but in this embodiment includes a display device 2623 and an input device 2624.

The display device 2623 includes a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The display device 2623 is capable of displaying a variety of images. The display device 2623 is configured, for example, with a touch panel, and functions as an interface that detects a variety of user operations. The display device 2623 may also be in a form that is built into a head-mounted display.

The input device 2624 may include physical keys, or may further include any input interface including a pointing device such as a mouse. The input device 2624 may also be capable of accepting non-contact user input such as voice input, gesture input, and gaze input. For gesture input, a sensor (image sensor, acceleration sensor, distance sensor, etc.) for detecting various states of the user, a dedicated motion capture integrating sensor technology and a camera, a controller such as a joypad, etc. may be used. The camera for gaze detection may also be disposed in a head-mounted display. As described above, the various states of the user are, for example, the user's orientation, position, movement, or the like. In this case, the user's orientation, position, and movement are concepts that include not only the orientation, position, and movement of a part or all of the user's body such as the face or hands, but also the orientation, position, movement, or the like of the user's gaze.

Next, the search support system 1 will be further described with reference to FIG. 4 onward.

Figure 4 is a flow chart showing an example of the flow of processing executed by the search support system 1. Figures 5 to 7 are explanatory diagrams of the processing shown in Figure 3, and Figure 5 is an explanatory diagram of an example of the mounting position of an acceleration sensor. Figure 6 is an explanatory diagram of an example of a joint, and Figure 7 is a diagram showing the relationship between input and output.

In FIG. 4, the processing is shown separately for the server device 10 and the terminal device 20. Note that although the database function is shown separately from the server device 10 for convenience, it may be realized by the server device 10. In the following, one terminal device 20 is described, but similar functions can be realized in other terminal devices 20. In the following, a user refers to a user of one terminal device 20 unless otherwise specified.

The user generates input motion information (an example of first movement information) via the terminal device 20 (step S2100) and transmits it to the server device 10 (step S2110). The input motion information is information that represents the movement of a person or an object. The input motion information may be information based on a posture at a single point in time, or may be information (time-series data) based on a series of movements (posture changes).

The input motion information is typically information that represents the user's own movements, but it may also be information that represents the movements of other people (e.g., friends), information that represents the movements of animals such as dogs, information that represents the movements of objects such as robots, etc.

In addition, the input motion information may be image information such as a video or still image, or may be information (secondary information) generated based on image information. For example, information generated based on image information may be information that extracts only the movement based on image processing or image analysis, or information that represents the movement in an abstract form.

In addition, the input motion information may be information based on actual movements, or may be information artificially generated by drawing, etc. For example, the input motion information may be generated based on existing pictures drawn in manga, etc., or may be generated based on hand-drawn pictures by the user.

In addition, the generation of input motion information may utilize sensors for detecting various states of the user (image sensors, acceleration sensors, distance sensors, etc.), dedicated motion capture that integrates sensor technology and cameras, controllers such as joypads, etc. For example, FIG. 5 shows examples of positions at which acceleration sensors may be worn by the user. The acceleration sensor may further have a function for detecting gyro (angular velocity). In the case of FIG. 5, the acceleration sensors are worn at a total of six locations on the person's body: the head, wrist, buttocks, and ankles (see HD, WL, WR, HP, AKL, and AKR in FIG. 5), but they may be worn in other ways.

The input motion information may be generated as the movement of joints based on a distance image or the like. For example, the example shown in FIG. 6 is a model represented by multiple joints and skeletons (links) between the joints. The joint model shown in FIG. 6 is a 16-joint model with one head, three torso (torso), both arms, and both legs, with both ends of the head defined as joints, and the other three ends and midpoints defined as joints. Specifically, the joint model consists of 16 joints A0 to A15 and 15 skeletons B1 to B15 (also called "parts B1 to B15") that connect the joints. As can be understood, for example, joints A4 and A7 are left and right shoulder joints, and joint A2 is a joint related to the cervical vertebrae. Also, joints A14 and A15 are left and right hip joints, and joint A0 is a joint related to the lumbar vertebrae. The input motion information may also include the movement of finer joints, such as finger joints.

The input motion information may represent the entire movement of a moving subject (person or object), or only a part of it. For example, the input motion information may represent the movement of the entire body, or the movement of a specific part of the body. The movement may represent not only the movement of each part via the joints, but also the movement related to facial expressions (movement of muscles, etc.). The movement may represent the movement of the user's gaze. In this case, the camera for gaze detection may be placed inside the head-mounted display.

When the server device 10 acquires input motion information (step S2200), it calculates the value of a similarity parameter related to the input motion information for the motion information data stored in a given master motion information database (step S2210).

The similarity parameter related to the input motion information is a parameter that indicates the degree of similarity between the input motion information and other motion information to be compared. Here, as described above, the input motion information is information that indicates movement. Therefore, the degree of similarity between the input motion information and other motion information to be compared relates to the degree of similarity of each movement. The method of calculating the value of the similarity parameter related to the input motion information is arbitrary, but for example, it may be calculated in such a way that a relatively heavy weight is assigned to characteristic movements (e.g. movements that occur frequently or unique movements).

The degree of similarity between the input motion information and other motion information to be compared may be evaluated after processing the input motion information and/or the other motion information to be compared. For example, the input motion information and the other motion information to be compared may be compared with each other in parts that have been partially cut out or compressed as a digest. The degree of similarity between the input motion information and other motion information to be compared may be evaluated based on the frequency analysis results (e.g., FFT: Fast Fourier Transform) for each time series waveform. In this case, when moving in accordance with a certain rhythm, a peak appears at a frequency corresponding to that rhythm, so that the similarity of the motion information when moving in accordance with the rhythm can be appropriately evaluated. Other parameters that can be suitably used for evaluation may include the total time (time of movement), tempo (e.g., Beats Per Minute), accuracy (degree of synchronization) with respect to the corresponding rhythm, etc.

The degree of similarity between the input motion information and other motion information to be compared may be only the degree of similarity regarding the movement, but may also include similarity of other elements (artistic quality, associated music, and background). In this case, the input motion information may include additional information other than the movement (e.g., music and background). Other elements and additional information may include avatar parts, type of costume, ID, location of the avatar in the virtual space, date, user attributes (country, language, age, gender, etc.), etc.

Note that a large number of pieces of motion information to be compared may typically be stored in the master motion information database that stores other pieces of motion information to be compared. In this case, the value of a similarity parameter related to the input motion information may be calculated for each of the many pieces of motion information to be compared. In this case, the greater the amount of motion information to be compared stored in the master motion information database, the greater the likelihood that the master motion information database contains motion information to be compared that has a high value of a similarity parameter related to the input motion information. Note that there may be multiple master motion information databases. Therefore, when one master motion information database is used, if no other motion information to be compared is obtained that has a similarity parameter related to the input motion information that is equal to or greater than a threshold value, another master motion information database may be used.

The server device 10 calculates the value of a similarity parameter related to the input motion information (step S2210), and extracts one or more master motion information corresponding to the input motion information based on the calculation result (step S2220). The one or more master motion information to be extracted may include, for example, other motion information to be compared (motion information in the master motion information database) for which the value of a similarity parameter related to the input motion information is equal to or greater than a threshold value. Alternatively, the one or more master motion information to be extracted may be only other motion information to be compared that has the highest similarity parameter value among other motion information to be compared that has a similarity parameter value related to the input motion information equal to or greater than a threshold value.

As described above, the server device 10 may be formed by multiple server computers. Therefore, one server computer forming the server device 10 may cooperate with another server computer forming the server device 10 to extract one or more pieces of master movement information corresponding to the input motion information. In this case, the server computers may cooperate with each other via an API (Application Programming Interface).

The server device 10 extracts one or more pieces of master motion information corresponding to the input motion information (step S2220) and outputs the extraction results (step S2230). The destination of the extraction results is arbitrary, and the extraction results may be output, for example, to a user related to the input motion information, or may be output for other processing within the server device 10.

In this way, in this embodiment, when input motion information is input, the search support system 1 extracts and outputs one or more pieces of master movement information corresponding to the input motion information based on the input motion information.

The master motion information here is information that represents a time series of the movement of a part or the whole of a person or object, and may be the same information as the input motion information. For example, the master motion information may include, as a time series of movement, at least one of a time series of movements related to a person's facial expressions and a time series of movements of parts of a person or object that have joints. However, unlike the input motion information, the master motion information is information that is prepared (managed and accumulated) in advance (motion information in a master motion information database). The source of the master motion information is arbitrary, and may be, for example, performance data of a famous performer or actor, animation, video, etc. However, in this case, it may be assumed that a license (for example a copyright-related license) has been obtained, or it may be developed as a paid service.

In the following, one or more pieces of master motion information that may be extracted when one piece of input motion information is input will also be referred to as one or more pieces of master motion information corresponding to one piece of input motion information. In addition, one piece of input motion information in this case will also be referred to as input motion information corresponding to one or more pieces of master motion information that are extracted.

As described above, a single piece of input motion information and one or more pieces of master motion information corresponding to the single piece of input motion information have a relationship in which the value of the similarity parameter related to the single piece of input motion information is equal to or greater than a threshold value. As described above, a single piece of input motion information and one or more pieces of master motion information corresponding to the single piece of input motion information do not necessarily represent motions of similar length (time length). For example, one or more pieces of master motion information corresponding to the single piece of input motion information may represent motions that are significantly longer or shorter than the single piece of input motion information. More specifically, for example, while a single piece of input motion information is based on a video that lasts a relatively long time, one or more pieces of master motion information corresponding to the single piece of input motion information may be a video that lasts a relatively short time (for example, a short video such as TikTok (registered trademark)).

In this embodiment, the extraction conditions for the master motion information for one piece of input motion information may be customizable by the user. In this case, the customizable extraction conditions are arbitrary, and may relate to thresholds, attributes, or performers. For example, extraction conditions may be set such that only master motion information related to female performers is extracted.

The search support system 1 may output the master motion information extracted in this manner as is, or may output it after processing, etc. The output destination is arbitrary, but may be set to the user who input the input motion information, for example. In this case, a user who wants to obtain master motion information for a certain movement can achieve the purpose by generating input motion information representing a movement similar to the movement and inputting the generated input motion information to the search support system 1. In other words, the user may be able to obtain master motion information related to the movement. This eliminates the need for the user to create similar motion information from scratch, improving convenience. For example, an example of input motion information is shown on the left side of FIG. 7. In this case, when input motion information is input, corresponding master motion information is obtained, so that the user can use the master motion information corresponding to the input motion information.

According to the search support system 1 of this embodiment, a data search function based on the similarity of movement information can be realized, improving convenience. For example, a user can obtain desired master movement information based on movements that are difficult to express in words, improving convenience. In addition, the search target can be managed in advance as master movement information, making it possible to limit the search area and reduce the processing load.

When a user searches for a video or other content that includes motion data, conventionally, the only way to extract the data is based on a character string associated with the video or other content (such as the title, a song title using a hash hug, or a celebrity name). However, there may be cases where no character string is attached to the video or other content.

In contrast, according to this embodiment, when a user generates input motion information, for example by moving his or her arms and legs, that input motion information can be used to search for similar motions or videos that include similar motions.
In this embodiment, the master motion information may be searched for as a viewing target. That is, a user may use the search support system 1 for viewing, not for using the master motion information. In this case, the search support system 1 may have different search targets for searches for viewing purposes and searches for production purposes. For example, in the case of a search for viewing purposes, the motion data that can be searched for as the master motion information may be videos produced for viewing (e.g., videos that can be viewed on TikTok (registered trademark), YouTube (registered trademark), or the like). In this case, a user can efficiently watch a desired video by using a new method for searching videos for viewing.
In addition, in this embodiment, the input motion information used to search for the desired master motion information may be information representing an emotional element (emote), and in this case, the information representing the emote may include an image of a corresponding facial expression, text information representing a corresponding emotion, or the like. In this case, the master motion information corresponding to the desired emote can be searched, and based on the information, animations, etc., as described below, can be generated. For example, the user selects the desired master motion information from among a plurality of pieces of master motion information that may be presented, and based on the selected master motion information, animations, etc., as described below, can be generated. In this case, since the input motion information is the user's own motion, it may be information representing, for example, awkward motion. However, the master motion information searched based on the information may be a sharp motion (such as a motion generated by a user who is good at motion, as if the user's own motion has been corrected), and in that case, the user may reflect such a sharp motion in his or her avatar (see the generation of avatar motion information, described below).

Figure 8 is a flowchart that shows an outline of another example of the processing flow executed by the search support system 1.

The example shown in FIG. 8 differs from the example shown in FIG. 4 in that it includes a specific example of a process (step S2230) for outputting the extraction results. The example shown in FIG. 8 includes the processes of steps S22301 and S22302 as step S2230 of the example shown in FIG. 4.

Specifically, the server device 10 transmits the extraction result (an example of the second movement information) to the terminal device 20 (step S22301), and the terminal device 20 outputs the extraction result to the user via the display device 2623 (step S2120). Note that the manner in which the extraction result is output via the display device 2623 is arbitrary, and a video based on one or more pieces of master movement information may be played in a form similar to video playback. Alternatively, a link such as a URL that allows access to the extracted one or more pieces of master movement information may be output. In this case, the extraction result transmitted from the server device 10 to the terminal device 20 may include link information such as a URL that allows access to the extracted one or more pieces of master movement information. With such a configuration, the communication load can be reduced.

When the user wishes to reflect the master movement information on his/her avatar based on the one or more pieces of extracted master movement information, the user may generate a reflection instruction to that effect and send it to the server device 10 (step S2130). Note that the reflection instruction may include an instruction as to which one or more pieces of extracted master movement information to use.

In this case, the server device 10 generates motion information (hereinafter, for the sake of distinction, also referred to as "avatar motion information") (an example of second motion information) in which the master motion information is reflected in a specific avatar (for example, the user's own avatar). Then, the server device 10 transmits the generated avatar motion information to the terminal device 20 (step S22302). In this case, the terminal device 20 may output the received avatar motion information to the user via the display device 2623 (step S2140). Note that the output mode of the avatar motion information via the display device 2623 is arbitrary, and a video based on the avatar motion information may be played in a form similar to video playback. Alternatively, a link such as a URL that enables access to a WEB site that can play the generated avatar motion information may be output. In this case, the avatar motion information transmitted from the server device 10 to the terminal device 20 may include link information such as a URL that enables access to the avatar motion information. According to such a configuration, the communication load can be reduced compared to the case where the avatar motion information is transmitted to the terminal device 20 in a video file format. Alternatively, the avatar motion information transmitted from the server device 10 to the terminal device 20 may be only a part of the avatar motion information. For example, only information (e.g., metadata, etc.) required to reflect the avatar motion information on the user's avatar may be transmitted from the server device 10 to the terminal device 20. For example, in this case, the terminal device 20 may play a video based on the avatar motion information using information (e.g., information on the user's avatar) stored on the terminal device 20 side. In this case, the communication load can be reduced compared to the case where the avatar motion information is transmitted to the terminal device 20 in the form of a video file. In addition, by storing metadata such as information on the user's avatar (parts ID, ID of the item worn), audio data, etc. in association with a timestamp, the terminal device 20 may reproduce these data (client rendering).

Here, the method of reflecting the master movement information in the movement of the user's avatar is arbitrary, but for example, if the master movement information has time-series movement data for each joint or part, similar time-series movement data may be given to each corresponding joint or part of the avatar. In this case, if the person or object that is the model for the master movement information and the user's avatar differ in the number of joints or the length of parts, such differences may be appropriately corrected. For example, if the number of joints associated with time-series movement data among the joints of the person that is the model for the master movement information is fewer than the number of joints of the user's avatar, the movements of the other joints of the avatar may be appropriately corrected using techniques such as inverse kinematics.

The number of pieces of master movement information to be reflected in the movement of the user's avatar is arbitrary, and for example, two or more pieces of master movement information may be combined and reflected in the movement of the user's avatar. In this case, the two or more pieces of master movement information may be combined in a time-division manner, or may be combined in such a manner that different master movement information is reflected for each joint or body part.

Next, further details of the search support system 1 of this embodiment will be described with reference to Figure 9 onwards.

Fig. 9 is a block diagram showing an example of the search support system 1 of this embodiment. Fig. 10 is an explanatory diagram showing an example of data in the avatar information storage unit 142. Fig. 11 is an explanatory diagram showing an example of data in the master movement information storage unit 144. In Figs. 10 and 11, "***" indicates a state in which some information is stored, and "..." indicates a state in which similar information is repeatedly stored. Fig. 12 is an explanatory diagram of a profile image. Figs. 13A and 13B are explanatory diagrams of animation content for automatic playback. Fig. 14 is an explanatory diagram related to the motions of multiple avatars. Fig. 15 is an explanatory diagram of the hierarchical structure of master movement information.

The functions of the search support system 1 described below may be realized by a server device 10, one or more terminal devices 20, or any combination of these.

In the example shown in FIG. 9, the search support system 1 includes a user information storage unit 140, an avatar information storage unit 142, a master movement information storage unit 144, an input motion information acquisition unit 150, a similarity calculation unit 152, an extraction processing unit 154, an output processing unit 156, a display motion generation unit 158, a content generation unit 160, an information management unit 162, a storage processing unit 164, a profile image generation unit 166, an animation generation unit 168, and an animation playback processing unit 170.

Note that each processing unit such as the input motion information acquisition unit 150 can be realized by the CPU 111 shown in FIG. 2 and/or the CPU 211 shown in FIG. 3 using a storage device (e.g., ROM 113, 213, etc.). Also, storage units such as the evaluation information storage unit 190 and the user information storage unit 192 can be realized by the auxiliary storage device 114 shown in FIG. 2 and the auxiliary storage device 214 shown in FIG. 3.

The user information storage unit 140 may store user IDs and associated avatar IDs. When the user information storage unit 140 is realized by a terminal device 20, the user information storage unit 140 may store only the avatar IDs related to the users of the corresponding terminal devices 20. When the user information storage unit 140 is realized by a server device 10, the user information storage unit 140 may store user IDs and avatar IDs related to each user.

The avatar information storage unit 142 stores avatar information related to the user's avatar. In the example shown in FIG. 10, in the avatar information 700, each avatar ID is associated with a face part ID, a hairstyle part ID, a clothing part ID, etc. The part information related to appearance, such as the face part ID, hairstyle part ID, clothing part ID, etc., is a parameter that characterizes the avatar, and may be selected by each corresponding user. For example, multiple types of information related to appearance, such as the face part ID, hairstyle part ID, clothing part ID, etc., related to the avatar are prepared. In addition, for the face part ID, a part ID is prepared for each type, such as face shape, eyes, mouth, nose, etc., and the information related to the face part ID may be managed as a combination of IDs of each part that constitutes the face. In this case, it is possible to draw each avatar not only on the server device 10 but also on the terminal device 20 side based on each ID related to appearance linked to each avatar ID.

In addition, in this embodiment, the avatar information may include joint information and body part information of the corresponding avatar.

The avatar's joint information may include, for example, which joints the avatar has (which joints function), information on the positional relationship of the joints (e.g., the positional relationship in a normal posture), the range of motion of the joints (degrees of freedom), etc. The part information may include information such as the shape of the parts connecting the joints.

When the avatar information storage unit 142 is realized by a terminal device 20, the avatar information storage unit 142 may store only avatar information related to the user of the corresponding terminal device 20. When the avatar information storage unit 142 is realized by a server device 10, the avatar information storage unit 142 may store avatar information related to each user.

The master motion information storage unit 144 corresponds to the master motion information database described above with reference to FIG. 4 and the like. The master motion information storage unit 144 stores the master motion information to be extracted as described above. The storage format of the master motion information is arbitrary, and may be, for example, the format of a video file, or the format of motion information extracted based on the video file (for example, time-series data of the motion of each joint). The storage format may be different for each piece of master motion information. In the example shown in FIG. 11, the master motion information storage unit 144 is associated with various information items such as motion data, audio information, background information, hardware information, uniqueness information, and management information for each master motion information ID. The motion data may be data representing a motion, as described above. The audio information, background information, hardware information, uniqueness information, and the like may be any information item that may not be associated. For example, when the master motion information is a motion that matches music, the audio information may be associated with the master motion information. When the master motion information is a video that is captured against a specific background, the background information may be associated with the master motion information. The hardware information is information that can identify the hardware (e.g., acceleration sensor, depth camera, etc.) used to acquire the master motion information. The hardware information may include frame rate, recording time, position information, etc. The uniqueness information is information that represents the uniqueness (originality) of the corresponding motion data. The hardware information may represent a sub-process executed in the server device 10 or the terminal device 20 or virtual hardware by emulation. The uniqueness information may be automatically evaluated based on artificial intelligence, etc. The uniqueness information may be generated based on the value of a parameter (e.g., a similarity parameter) related to the similarity between other master motion information that has already been stored. The uniqueness related to the uniqueness information may be evaluated by the motion data alone, or may be evaluated in combination with audio information, background information, hardware information, etc. The management information may include character information for tagging, information such as the creator (performer, etc.) of the corresponding master motion information, and the creation date and time. The management information may include information on the usage conditions when using the corresponding master motion information. The usage conditions may include, for example, conditions based on costs in the case of paid use, whether processing is possible, whether deformation is required, etc. The usage conditions may also include conditions regarding the length of the user's limbs (or the length of the avatar's limbs) and the hardware configuration.

The master movement information storage unit 144 may also store additional information that allows the search range (the range of objects to be calculated by the similarity calculation unit 152, described later) to be narrowed down. For example, the additional information may include attributes of the movement related to the master movement information (e.g., dance, sports, daily movements, etc.).

The master movement information storage unit 144 is preferably implemented by a server device 10 since it is suitable for storing a relatively large amount of data, but may also be implemented by multiple terminal devices 20.

The input motion information acquisition unit 150 acquires input motion information. The input motion information is as described above.

The input motion information acquisition unit 150 may be realized by the terminal device 20 alone. In this case, the terminal device 20 acquires input motion information from the corresponding user. On the other hand, the input motion information acquisition unit 150 may be realized by multiple terminal devices 20 and the server device 10. In this case, the server device 10 may acquire input motion information from each user via the terminal device 20.

In this embodiment, the input motion information acquisition unit 150 acquires input motion information in association with a motion information request from a user. The motion information request from a user is a request related to the acquisition of display motion information, which will be described later, and may include filtering information, which will be described later, information specifying a specific avatar (when requesting the generation of avatar motion information, which will be described later), and the like.

The similarity calculation unit 152 calculates the value of a predetermined parameter related to the similarity between the input motion information acquired by the input motion information acquisition unit 150 and the master motion information stored in the master motion information storage unit 144. The predetermined parameter may be the similarity parameter described above.

The similarity calculation unit 152 may calculate the value of the similarity parameter for each of all the master motion information stored in the master motion information storage unit 144 for one input motion information, but preferably, the calculation target may be narrowed down from the viewpoint of reducing the processing load. For example, when the attribute of one input motion information is distinguishable, the calculation target may be narrowed down to only the master motion information having the same attribute. Furthermore, when one input motion information includes auxiliary information other than the motion, the calculation target may be narrowed down based on the auxiliary information. For example, the auxiliary information may be narrowed down information that can be input by the user. For example, the auxiliary information may include text information such as "*** dance". Furthermore, the auxiliary information may include information regarding the hardware resource (e.g., an acceleration sensor, etc.) used to acquire the input motion information. In this case, the calculation target may be narrowed down to only the master motion information acquired using the same or similar series of hardware resources. Furthermore, the auxiliary information may include background information. Furthermore, the auxiliary information may include audio information. In this case, the calculation target may be narrowed down to only the master motion information to which the same series of audio information is linked. The auxiliary information may also include background information. When the input motion information is image information, the background information is information about the background of a moving person or object. In this case, the calculation target may be limited to only master motion information that is linked to background information of the same type.

In this embodiment, the similarity calculation unit 152 includes a movement similarity evaluation unit 1521 and an other similarity evaluation unit 1522.

The movement similarity evaluation unit 1521 calculates the value of a parameter related to the similarity between movements. In this case, the movements related to the input motion information and the movements related to the master movement information are compared. As described above, the method of comparing the movements may be the same as the method of calculating the value of the similarity parameter described above.

For human movements that involve joints, the value of a parameter related to similarity may be calculated based on at least one of the joint angle, the change in the joint angle, the joint position, the change in the joint position, and the length between the joints. Time-series waveforms of the values of these parameters or the results of frequency analysis thereof may be used to compare similarities. The joint angle may be evaluated using other parameters such as the dot product.

The other similarity evaluation unit 1522 calculates parameter values related to similarities other than between movements. In this case, the attributes and additional information described above may be taken into consideration.

The similarity calculation unit 152 may calculate the final value of the predetermined parameter based on the calculated value of the parameter related to the similarity between the movements and the calculated value of the parameter related to the similarity other than between the movements. Note that artificial intelligence may be used to calculate the similarity.

The extraction processing unit 154 extracts one or more pieces of master motion information based on the calculation results by the similarity calculation unit 152. The method of extracting the master motion information based on the calculation results by the similarity calculation unit 152 is arbitrary, but the master motion information with the largest value of a specified parameter may be extracted, or a specified number of pieces of master motion information may be extracted in descending order of the value of the specified parameter. Alternatively, all master motion information with a value of a specified parameter equal to or greater than a threshold value may be extracted.

The output processing unit 156 outputs the display motion information generated based on one or more pieces of master motion information extracted by the extraction processing unit 154 in a manner that is visible to humans. The output processing unit 156 may output to any display device. For example, the information may be displayed on a terminal viewed by specific people, such as the display device 2623 of the terminal device 20, or may be displayed on a device viewed by an unspecified number of people, such as digital signage.

In this embodiment, the output processing unit 156 may output the display motion information if the motion information request satisfies a predetermined condition. The predetermined condition is arbitrary, but may include conditions related to copyright, public order, and morals. The predetermined condition may also include a condition for ensuring the uniqueness of the display motion information. In this case, the uniqueness may be evaluated based on the relationship between the display motion information and the original master motion information corresponding to it.

The display motion generation unit 158 generates display motion information to be output by the output processing unit 156 based on one or more pieces of master motion information extracted by the extraction processing unit 154.

The display motion information may be information that collectively displays two or more pieces of master motion information extracted by the extraction processing unit 154, or may be generated for each piece of master motion information. The display motion information related to a piece of master motion information may be the piece of master motion information itself, or may be new information generated based on the piece of master motion information. In the latter case, the display motion information related to a piece of master motion information may be generated by processing the piece of master motion information. In this case, the processing method is arbitrary, but may include, for example, shortening to a digest version (summary version) or changing the background or rhythm of the music, etc.

The display motion generation unit 158 may also generate the avatar motion information described above. That is, the display motion generation unit 158 may generate display motion information (avatar motion information) by reflecting one or more pieces of master motion information extracted by the extraction processing unit 154 on a specific avatar. In this case, the specific avatar may be designated by a user (e.g., the user who has issued the motion information request), and may be, for example, the avatar of that user. For example, the display motion generation unit 158 may reflect the master motion information searched for in relation to the emote information on the movement of the corresponding user's avatar, based on a reflection instruction from the user.

The display motion generation unit 158 may be provided on the server device 10 side as described above in relation to the avatar motion information, or on the terminal device 20 side, or on both the server device 10 side and the terminal device 20 side in a manner in which the functions are shared between them.

The content generating unit 160 generates video content or animation based on the display motion information generated by the display motion generating unit 158. The video content or animation may be the display motion information itself generated by the display motion generating unit 158, or may be video content or animation different from the display motion information generated by the display motion generating unit 158. In the latter case, for example, the content generating unit 160 may incorporate the display motion information generated by the display motion generating unit 158 into a part of the video content or animation. Note that in this case, the video content or animation to be incorporated is arbitrary, and may be produced based on other display motion information.

In this embodiment, the content generation unit 160 may generate video content or animation by associating background information, audio information, emote information, or text information with the display motion information generated by the display motion generation unit 158. In this case, the associated background information, audio information, emote information, or text information may be supplied by a user (e.g., the user who has made the motion information request), or may be automatically generated based on artificial intelligence or the like. Alternatively, the video content or animation generated by associating the emote information may be registered in a menu or the like that shows a list of emotes available to the corresponding user.

The content generator 160 may also execute an NFT conversion process on the generated video content or animation in response to an application from a user, etc. The NFT conversion process includes issuing the target item as an NFT (Non-Fungible Token). In this case, the content generator 160 may be minted via a smart contract on a blockchain network (not shown). The NFT conversion process may also include listing the issued NFT on a marketplace, etc.

The information management unit 162 sets usage restrictions on the master motion information stored (accumulated) in the master motion information storage unit 144. For example, the information management unit 162 may set usage restrictions on a piece of master motion information based on the above-mentioned management information associated with the piece of master motion information. In this case, if the above-mentioned management information associated with the piece of master motion information indicates provision for a fee, free provision (use) may be restricted. In this case, the information management unit 162 may exclude master motion information that cannot be used due to the usage restrictions from the search range (or from the calculation target of the similarity calculation unit 152).

The storage processing unit 164 stores new master movement information in the master movement information storage unit 144 based on motion information for master movement information (an example of third movement information) supplied (provided) by a specific user. The specific user is optional, but may be a user who has consented to the above-mentioned usage mode (extraction and use for generating avatar motion information, etc.). When storing new master movement information, the storage processing unit 164 may acquire and generate the above-mentioned management information, etc. In order to encourage the provision of motion information for master movement information, an incentive such as a specific benefit may be given to users who provide motion information for master movement information.

The profile image generating unit 166 generates a profile image associated with the user based on the display motion information generated by the display motion generating unit 158. The profile image may be a profile image accompanied by the movement of an avatar associated with the user. In this case, the profile image generated by the profile image generating unit 166 may be used in a social networking service (SNS) or the like. The profile image may be a still image, or may be a relatively short video such as a short video.

The profile image generating unit 166 may function based on a motion information request from a user. For example, it may function when the motion information request includes a request to generate a profile image. Alternatively, it may function in response to an instruction such as the reflection instruction of step S2130 described above with reference to FIG. 8.

The profile image generating unit 166 may also associate background information, audio information, emote information, or text information with the generated profile image. In this case, the associated background information, audio information, emote information, or text information may be supplied by a user (e.g., a user who has made a motion information request), or may be automatically generated based on artificial intelligence or the like. The emote information may include information indicating what emotion the user is expressing, or an animation of a facial expression.

Note that some or all of the functions of the profile image generation unit 166 may be realized by the content generation unit 160 described above.

Figure 12 shows an example of a profile screen including a profile image G12. If the profile image G12 is a video, it may be automatically played when the profile screen is output. In the example shown in Figure 12, the profile image G12 forms part of the profile screen, but it may also form the entirety of the profile screen.

The animation generation unit 168 generates animation content for automatic playback. The animation content for automatic playback may be a part or all of the animation generated by the content generation unit 160 described above. In this case, a part or all of the functions of the animation generation unit 168 may be realized by the content generation unit 160 described above.

The animation content for automatic playback generated by the animation generation unit 168 may be stored in association with a user (e.g., a user who has made a motion information request). The animation content for automatic playback may also be stored in association with conditions for automatic playback (playback conditions) in, for example, a storage device of the terminal device 20 of the corresponding user. The conditions for automatic playback (playback conditions) are arbitrary, but may be set by the user. The conditions for automatic playback may be used for responses in chats, etc. That is, the animation content for automatic playback may be played as a response in chats, etc. For example, in the example shown in FIG. 13A, when the terminal device 20 of user B receives a question A from another user (e.g., an invitation to go somewhere together), multiple stamps STM1 to STM4 are displayed as candidates. In this case, when each of the stamps STM1 to STM4 is selected, the corresponding animation content for automatic playback may be played. In this case, user B may select one of stamps STM1 to STM4 by himself, but in automatic mode, one of stamps STM1 to STM4 may be automatically selected depending on the conditions for automatic playback.

When the conditions for automatic playback are met for a piece of animation content for automatic playback, the animation playback processing unit 170 plays the content. The animation content for automatic playback and the conditions for automatic playback are as described above.

The animation playback processing unit 170 may also function in the metaverse space. For example, when the conditions for automatic playback are satisfied in the metaverse space, the avatar may be moved in a manner corresponding to the animation content for automatic playback. For example, the various emote information in the above-mentioned emote list (and the video content associated therewith, etc.) may be activated when the corresponding user selects (specifies) one of the emotes, thereby satisfying the conditions for automatic playback. When one emote is activated, the corresponding avatar performs a movement that expresses the emote in the virtual space. The movement that expresses this emote may be a video content or animation generated in association with the above-mentioned emote information. By registering the desired emote information in the emote list in this way, the desired emote from among the various emotes can be easily reflected in one's own avatar.
For example, in the example shown in FIG. 13B, the terminal device 20 of the user B calls up an emote list, and a plurality of emote buttons EM1 to EM4 are displayed as candidates. The plurality of emote buttons EM1 to EM4 may be preregistered content (video content or animation generated in association with emote information). In this case, when each of the emote buttons EM1 to EM4 is selected, the corresponding animation content for automatic playback may be played. In this case, the user B may select one of the emote buttons EM1 to EM4 by himself, but in the automatic mode, for example, the emotion (feeling) of the user B may be automatically detected from bioinformation (e.g., heart rate information, etc.) or detected by image recognition, and one of the emote buttons EM1 to EM4 may be automatically selected. In the example shown in FIG. 13B, the emote buttons EM1 to EM4 are given characters representing the corresponding emotes, but instead of or in addition to them, a digest version of the corresponding content may be played. In addition, in the example shown in Fig. 13B, four emote buttons EM1 to EM4 are illustrated, but the number of emote buttons is arbitrary, and the number of emote buttons may be prepared according to the number of registrations. In addition, in the example shown in Fig. 13B, the terminal device 20 in the form of a smartphone is illustrated, but the same applies when another type of terminal device 20, such as a head-mounted display, is used. For example, in the case of a terminal device 20 in the form of a head-mounted display, the buttons corresponding to the emote buttons EM1 to EM4 may be realized by a user interface that can be called up in a virtual space.

Although each embodiment has been described in detail above, it is not limited to a specific embodiment, and various modifications and changes are possible within the scope of the claims. It is also possible to combine all or some of the components of the above-mentioned embodiments.

For example, in the above-described embodiment, one piece of input motion information or one piece of master motion information represents the motion of a corresponding person or object, and thus represents the motion of one person or object, but is not limited to this. For example, one piece of input motion information or one piece of master motion information may represent the motion of two or more people. In this case, for example, as shown in FIG. 14, avatar motion information relating to the coordinated motion of multiple avatars A1 to A4 can be obtained based on the extracted master motion information. In this case, it is possible to obtain master motion information relating to the motion involving contact or coordination between multiple avatars, such as multiple avatars holding hands or one or more avatars lifting up an avatar.

In addition, in the above-described embodiment, in the master motion information storage unit 144, each piece of master motion information may be managed in a hierarchical structure as shown in FIG. 15. In this case, by determining the similarity to a piece of input motion information from a higher hierarchy, it becomes easier to search for master motion information similar to the piece of input motion information.

1 Search support system 3 Network 10 Server device 20 Terminal device 140 User information storage unit 142 Avatar information storage unit 144 Master motion information storage unit 150 Input motion information acquisition unit (an example of a motion acquisition unit)
152 Similarity calculation unit 1521 Movement similarity evaluation unit 1522 Other similarity evaluation unit 154 Extraction processing unit 156 Output processing unit 158 Display motion generation unit (an example of a second movement information generation unit)
160 Content generation unit 162 Information management unit 164 Storage processing unit 166 Profile image generation unit 168 Animation generation unit 170 Animation playback processing unit 190 Evaluation information storage unit 192 User information storage unit

Claims (22)

A storage unit that stores master movement information representing a time series of a movement of a part or the whole of a person or an object;
A movement acquisition unit that acquires first movement information related to a movement of a part or an entirety of a user;
a calculation unit that calculates a value of a predetermined parameter related to a similarity between the first motion information acquired by the motion acquisition unit and the master motion information stored in the storage unit;
An information processing system comprising: an extraction processing unit that extracts one or more master motion information corresponding to the first motion information acquired by the motion acquisition unit from among the multiple master motion information stored in the memory unit based on the value of the specified parameter calculated by the calculation unit. The information processing system according to claim 1, wherein the calculation unit calculates the value of the predetermined parameter based on a rough version or a part of the master movement information. The information processing system according to claim 1, further comprising an output processing unit that outputs second movement information in a manner that is visible to a human based on one or more pieces of master movement information extracted by the extraction processing unit. The information processing system according to claim 3, wherein the second motion information includes at least one of information identical to one or more of the master motion information extracted by the extraction processing unit and other information generated based on one or more of the master motion information extracted by the extraction processing unit. The information processing system according to claim 3, further comprising a second movement information generating unit that generates the second movement information by reflecting one or more of the master movement information extracted by the extraction processing unit to the movement of a specific avatar. The information processing system of claim 5, wherein the specific avatar includes an avatar associated with the user. The information processing system according to claim 6, further comprising a profile image generating unit that generates a profile image associated with the user, the profile image including a movement of an avatar associated with the user, based on the second movement information. The information processing system according to claim 3, further comprising a content generation unit that generates video content or animation based on the second movement information. The information processing system according to claim 8, wherein the content generation unit generates the video content or animation by associating background information, audio information, emote information, or text information with the second movement information. The information processing system according to claim 1, further comprising a storage processing unit that stores new master movement information in the storage unit based on third movement information supplied from a specific user. the movement acquisition unit acquires the first movement information in association with a request from the user;
The information processing system according to claim 3 , wherein the output processing unit outputs the second movement information to a request source related to the request. The information processing system according to claim 11, wherein the output processing unit outputs the second movement information when the request satisfies a predetermined condition. At least a part of the master motion information stored in the storage unit is associated with master-side acoustic information in the time series,
the movement acquisition unit acquires, in addition to the first movement information, acoustic information of a user side associated with the first movement information;
The information processing system of claim 1 , wherein the calculation unit, when the user's acoustic information is acquired by the movement acquisition unit, calculates the value of the specified parameter further related to the similarity between the user's acoustic information and the master's acoustic information stored in the memory unit. The information processing system according to claim 1, wherein the time series of movements includes at least one of a time series of movements related to facial expressions of a person and a time series of movements of parts of a person or object having joints. The time series of movements includes a time series of movements of parts of a person or object having a joint;
The first movement information relates to movements of at least two or more parts of the user connected via a joint,
The information processing system according to claim 1 , wherein the calculation unit calculates the value of the predetermined parameter based on at least one of a joint angle, a change in the joint angle, a joint position, a change in the joint position, and a length between joints. The information processing system according to claim 1, wherein the extraction processing unit extracts one or more pieces of master motion information from among the plurality of pieces of master motion information stored in the storage unit, prioritizing a high degree of similarity to the first motion information acquired by the motion acquisition unit. The information processing system according to claim 1, wherein at least a portion of the master movement information stored in the storage unit is associated with information capable of identifying a hardware configuration used to generate the movement time series. The information processing system according to claim 1, wherein at least a portion of the master movement information stored in the storage unit is associated with information representing the originality of the movement or the originality of the information itself. The information processing system according to claim 1, further comprising an information management unit that sets usage restrictions for the master movement information stored in the storage unit. an animation generation unit that generates animation content for automatic playback based on the second motion information generated by the second motion information generation unit;
The information processing system according to claim 6 , further comprising an animation playback processing unit that plays back the animation content when a playback condition related to the user is satisfied. Obtaining first movement information relating to a movement of a part or an entirety of a user;
calculating a value of a predetermined parameter related to a similarity between master movement information stored in a storage unit that stores master movement information representing a time series of a movement of a part or an entirety of a person or an object and the acquired first movement information;
An information processing method executed by a computer, comprising: extracting one or more pieces of master movement information corresponding to the first movement information from among the plurality of pieces of master movement information stored in the memory unit based on the calculated value of the specified parameter. Obtaining first movement information relating to a movement of a part or an entirety of a user;
calculating a value of a predetermined parameter related to a similarity between master movement information stored in a storage unit that stores master movement information representing a time series of a movement of a part or an entirety of a person or an object and the acquired first movement information;
A program that causes a computer to execute a process of extracting one or more pieces of master movement information corresponding to the first movement information from among the multiple pieces of master movement information stored in the memory unit based on the calculated value of the specified parameter.