[go: up one dir, main page]

WO2018135334A1 - Dispositif et procédé de traitement d'informations, et programme informatique - Google Patents

Dispositif et procédé de traitement d'informations, et programme informatique Download PDF

Info

Publication number
WO2018135334A1
WO2018135334A1 PCT/JP2018/000170 JP2018000170W WO2018135334A1 WO 2018135334 A1 WO2018135334 A1 WO 2018135334A1 JP 2018000170 W JP2018000170 W JP 2018000170W WO 2018135334 A1 WO2018135334 A1 WO 2018135334A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
information
content
profile
matching process
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2018/000170
Other languages
English (en)
Japanese (ja)
Inventor
白井　太三
洋平 川元
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Corp
Original Assignee
Sony Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Corp filed Critical Sony Corp
Publication of WO2018135334A1 publication Critical patent/WO2018135334A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units

Definitions

  • the technology disclosed in this specification relates to an information processing apparatus, an information processing method, and a computer program that execute processing related to matching between users.
  • a free viewpoint video can be viewed using a head mounted display that is worn on the head.
  • a photographing system for photographing a wide-angle image wider than a display image that is actually displayed is provided, and a display image to be viewed by the user is cut out based on the position information of the user's head detected by the rotation angle sensor.
  • Proposals have been made regarding a head-mounted display system for display (see, for example, Patent Document 1).
  • interactive viewing service can be realized by applying bi-directional communication to free viewpoint video distribution service. For example, it is possible to respond to various needs by distributing videos in which the viewpoint position and the line-of-sight direction are switched for each user (see, for example, Patent Document 2).
  • Free viewpoint video can be used as entertainment content such as sports, games, concerts, and theater.
  • instructions, teaching, guidance, and work support can be performed from the content viewer to the photographer through bidirectional communication between the shooting site and the viewer.
  • An object of the technology disclosed in the present specification is to provide an information processing apparatus, an information processing method, and a computer program that execute processing related to matching between users.
  • the technology disclosed in the present specification has been made in consideration of the above-mentioned problems, and the first aspect thereof is At least one first user and a second user accessing content information associated with the first user's space are at least one of the first user profile information and the content profile information.
  • the information processing apparatus includes a control unit that executes a matching process for matching based on first profile information including one and second profile information including the second user's profile information.
  • the second aspect of the technology disclosed in this specification is: An information processing method executed by the information processing apparatus, A control unit of the information processing apparatus; At least one first user and a second user accessing content information associated with the first user's space are at least one of the first user profile information and the content profile information.
  • This is an information processing method for executing matching processing for matching based on first profile information including one and second profile information including the profile information of the second user.
  • the third aspect of the technology disclosed in this specification is: A computer program for causing an information processing apparatus to execute information processing; In the control unit of the information processing apparatus, At least one first user and a second user accessing content information associated with the first user's space are at least one of the first user profile information and the content profile information.
  • the computer program is written in a computer-readable format so as to execute matching processing based on first profile information including one and second profile information including the second user's profile information.
  • FIG. 1 is a diagram schematically illustrating a configuration example of a free viewpoint video distribution system 100.
  • FIG. 2 is a diagram illustrating a configuration example of an apparatus arranged in a space where the imaging apparatus 101 or the first user exists.
  • FIG. 3 is a diagram illustrating a configuration example of a device arranged in a space where the display device 142 or the second user exists.
  • FIG. 4 is a diagram showing a configuration example of profile information.
  • FIG. 5 is a diagram showing a functional configuration for executing the recommendation process.
  • FIG. 6 is a diagram showing an outline of the secure calculation protocol.
  • FIG. 7 is a diagram showing a data flow in the secure calculation protocol.
  • FIG. 8 is a flowchart showing the processing procedure of the secure calculation protocol.
  • FIG. 1 is a diagram schematically illustrating a configuration example of a free viewpoint video distribution system 100.
  • FIG. 2 is a diagram illustrating a configuration example of an apparatus arranged in a space where the imaging apparatus 101 or the first
  • FIG. 9 is a flowchart illustrating a basic procedure performed when a user who desires to view content logs in to the free viewpoint video distribution system 100.
  • FIG. 10 is a flowchart illustrating a basic procedure performed when a user who captures content logs in to the free viewpoint video distribution system 100.
  • FIG. 11 is a diagram illustrating a configuration example of a dedicated UI screen for selecting information used by the user for matching processing.
  • FIG. 12 is a flowchart showing a processing procedure for performing matching processing between users using profile information in a limited manner.
  • FIG. 13 is a flowchart showing a processing procedure for performing matching processing between users by pre-calculation.
  • FIG. 14 is a flowchart illustrating a processing procedure for performing a matching process while limiting a search range based on user history information.
  • FIG. 1 schematically shows a configuration example of the free viewpoint video distribution system 100.
  • Imaged image information such as a free viewpoint video acquired using the imaging device 101 (for example, a multi-view camera or an all-around camera) is transmitted to the content distribution server 120 via the network 110.
  • the imaging device 101 for example, a multi-view camera or an all-around camera
  • Imaged image information is transmitted to the content distribution server 120 via the network 110.
  • the imaging device 101 for example, a multi-view camera or an all-around camera
  • FIG. 1 only one imaging device 101 is drawn for the sake of simplicity, but it is assumed that a large number of imaging devices 101 serving as a supply source of captured image information are scattered in the real world.
  • the imaging apparatus 101 only needs to be able to acquire captured image information in a space where the first user who is a photographer exists, and various apparatus configurations can be employed.
  • the imaging apparatus 101 is a wearable worn by a photographer such as a head-mounted display equipped with a photographing device such as a camera or an imager in addition to a general camera device, a multi-viewpoint camera, and an all-sky camera. It may take the form of a device.
  • the space in which the first user exists is basically a real space, but can be defined as a virtual space instead of the real space.
  • “real space” or “virtual space” may be simply referred to as “space”.
  • the captured image information acquired by the imaging apparatus 101 can also be referred to as content information associated with the first user's space.
  • the captured image information acquired by the imaging apparatus 101 is also referred to as “content”.
  • a large number of photographers as the first user each have a POI (Point). Of Interest: It is assumed that someone goes to a place where he / she finds it useful or interested and where each imaging apparatus 101 is used for shooting.
  • POI Point
  • the viewer side which will be described later, can select any one of a plurality of viewpoint positions with the same POI and view a free viewpoint video at that viewpoint position.
  • examples of POIs include tourist attractions, commercial facilities or stores in commercial facilities, stadiums, halls, concert venues, theaters, etc. where sports competitions such as baseball and soccer are held.
  • the shooting location is not limited to POI.
  • the photographer Since the photographer as the first user is actually at the shooting site and is active with his / her body (that is, the body actually exists at the site), it may be a “Body” user. it can.
  • the photographer is a person (natural person) or a mobile device such as a vehicle (including a vehicle that is manually driven by a person and a vehicle that is automatically or unattended), a ship, an airplane, a robot, and a drone. It is also assumed.
  • the content distribution server 120 accumulates the content transmitted from each imaging device 101 in the content database 121.
  • the content distribution server 120 performs streaming distribution of real-time (live video) content sent from each imaging device 101 to each viewer of the free viewpoint video via the network 130.
  • the content distribution server 120 may read past recorded (archived) content stored in the content database 121 and perform streaming distribution to each viewer of the free viewpoint video via the network 130.
  • the network 130 may be a part of a wide area network together with the network 110, or may be a network independent of the network 110.
  • Each viewer as a second user is equipped with a personal computer (PC) 141 for free viewpoint video reproduction and a display device 142 for viewing the free viewpoint video.
  • the second user does not act with the body at the site, but can be aware of the site by viewing the video viewed from the viewpoint of the photographer who is the first user. Therefore, the viewer who is the second user can also be referred to as a “Ghost” user because there is only consciousness in the field.
  • the display device 142 is configured, for example, as a head-mounted display capable of viewing VR (Virtual Reality) video, and is equipped with a 3D display that allows the viewer who wears to view the free viewpoint video in both eyes.
  • VR Virtual Reality
  • the display device 142 is not limited to a specific device configuration.
  • the display device 142 is equipped with a stereo camera, a 9 DoF (Degrees of Freedom) sensor, and the like, and can perform self-position estimation. Further, the display device 142 detects the line of sight of the second user who is a viewer using a pupil corneal reflection method or the like, and from the rotation center position of the left and right eyeballs and the direction of the visual axis (and head posture), It is assumed that the line-of-sight direction of the second user can be obtained. Alternatively, the forward direction may be handled as the second user's line-of-sight direction based solely on the posture of the head measured or estimated by head tracking. The display device 142 transmits information on the obtained self-position and line-of-sight direction to the PC 141 successively.
  • 9 DoF Degrees of Freedom
  • the PC 141 receives a content stream of free viewpoint video from the content distribution server 120 via the network 130. Then, the PC 141 renders a free viewpoint video with the self-position received from the display device 142 and a specified FoV (Field of View). The rendering result is displayed on the display of the display device 142. The viewer can freely control the viewpoint position and the line-of-sight direction by changing his / her head posture.
  • the display device 142 may be directly connected to the network 130 without the PC 141 interposed.
  • the display device 142 may not be used, and the rendered free viewpoint video may be displayed on a monitor / display provided in the PC 141 so that the viewer can view it.
  • the PC 141 and the display device With respect to 142, various device configurations may be employed.
  • a UI User Interface
  • recommendation information including a list of contents or the like is displayed on the screen of the display device 142, and the second user may select the contents through an operation on the UI screen.
  • the screen configuration of the UI for displaying the recommendation information is arbitrary.
  • the UI may be a list of content titles or thumbnails of representative images, or a free viewpoint video shooting location (where the imaging device 101 is installed, or the first location).
  • the user's whereabouts a list of user names (including nicknames and handle names) of the first user who is the photographer, and thumbnails of face images.
  • the framework of the interaction when the second user views the content acquired on the first user side is also referred to as “JackIn (connection)”.
  • the second user can view the content associated with the connected first user's space.
  • the first user when connected to the second user, delivers content associated with his / her space.
  • the purpose of connecting users is various.
  • the second user is intended to simply view content related to a space in which the user does not exist or content of interest (for example, a sporting event photographed on the first user side),
  • the user is connected to the first user for the purpose of teaching or providing work support to the first user.
  • the first user not only aims to publicize the content acquired by the imaging apparatus 101 (for free or for a fee) but also receives instruction and work support from the second user who views the content. It may be connected to a second user for the purpose.
  • a connection destination candidate is recommended to the first user or the second user using the recommendation system.
  • a recommendation system is installed in the free viewpoint video distribution system 100 or outside the system 100. For example, when a candidate for the first user who recommends connection to the second user (or a candidate for content recommended for viewing) is obtained based on the matching processing result of the recommendation system, the recommendation information including information about the candidate is obtained. It is presented on the UI screen of the display device 142.
  • the display device 142 sends an access request to the first user (or content associated with the first user's space) selected by the second user through the above-described UI screen operation through the PC 141 and the network 130. Transmit to the content distribution server 120. Alternatively, the display device 142 may directly transmit the access request to the corresponding imaging device 101.
  • FIG. 2 shows a configuration example of an apparatus arranged in the space where the imaging apparatus 101 or the first user exists.
  • a voice input device 201 for inputting voice during shooting by the imaging device 101 is provided.
  • the voice input device 201 may be either a monaural microphone or a stereo microphone, but collects the voice of the first user at the time of shooting, the voice generated by the subject being shot by the imaging device 101, and the like.
  • a sensor device 202 that detects a situation in the vicinity of the first user that dynamically changes such as a shooting environment.
  • the sensor device 202 includes various environmental sensors that detect information related to the weather in the space where the first user exists (or at the time of shooting) such as temperature, humidity, atmospheric pressure, and illuminance.
  • the sensor device 202 may include a biosensor that detects a photographer's biometric information such as body temperature, pulse, sweating, exhalation, and brain waves.
  • the sensor device 202 captures the first user himself / her who is the photographer and the accompanying person of the photographer, and acquires the information of the user himself / herself or the accompanying person through processing such as face detection and face recognition.
  • Other imaging devices may be provided.
  • the sensor device 202 may include a position sensor that measures the current position of the imaging device 101 or the first user.
  • the position sensor receives, for example, a GNSS signal (for example, a GPS signal from a Global Positioning System (GPS) satellite) from a GNSS (Global Navigation Satellite System) satellite, performs positioning, and calculates the latitude, longitude, and altitude of the vehicle. Generate positional information including.
  • the position sensor may specify the current position based on the measurement information from the wireless access point using PlaceEngine (registered trademark) or the like.
  • Sensor information detected by the sensor device 202 can be handled as information associated with the space of the first user.
  • sensor information detected by the sensor device 203 during a period in which content is acquired by the sensor information imaging device 101 detected by the sensor device 202 can be handled as information associated with the content acquisition period.
  • an output device 203 capable of presenting information to the first user who is a photographer through video display or audio output is provided. Yes. On the display screen that the output device 203 is equipped with, a UI including recommendation information including a list of content distribution destinations (second users requesting access to the content) is displayed. You may make it select the delivery destination of a content through operation of UI screen.
  • the output device 203 may be equipped with a configuration that performs output such as vibration, light electrical stimulation, and haptic (tactile sense) in addition to video and audio output. Further, the output device 203 may include a device that can support or restrain at least a portion of the first user's limb, such as an exoskeleton device, to teach the first user of the action. . The output device 203 can be used to provide information feedback from the second user who is a content viewer, and provide instructions and work support to the first user from the second user.
  • the communication device 204 is connected to the network 110 and transmits AV content including content acquired by the image capturing device 101 and sound at the time of image capturing collected by the sound input device 201, and reception of information to be output by the output device 203. To do. Further, the communication device 204 may transmit environment information measured by the sensor device 202 or the like. Further, the communication device 204 can receive an access request (or connection request) for content from the second user directly or indirectly through the content distribution server 120.
  • the imaging device 101 may be configured in the form of a wearable device worn by the photographer, such as a head-mounted display.
  • a head-mounted display may be equipped with all the above-described peripheral devices such as the imaging device 101, the voice input device 201, the sensor device 202, the output device 203, and the communication device 204. It is assumed that the imaging device 101 and the devices 201 to 204 are interconnected.
  • the connection form is not limited to a specific interface standard, and may be either a wired connection or a wireless connection.
  • the imaging apparatus 101 and peripheral apparatuses 201 to 204 are directly or indirectly disposed with respect to a predetermined casing 205 that can be carried by the first user.
  • the direct arrangement corresponds to, for example, incorporation in the housing 205
  • the indirect arrangement corresponds to external connection to the housing 205 via, for example, a connector, a cable, or wireless communication.
  • the content attribute information is also referred to as a “content profile”. Also, parameters such as the resolution, frame rate, transmission rate, or decoding rate of the imaging apparatus 101 may be included in the content profile.
  • the content profile may be stored in a memory inside the imaging apparatus 101 that acquired the content (or an apparatus in which the imaging apparatus 101 and peripheral devices 201 to 204 are integrated).
  • the content profile for each photographer or content may be stored in the content database 121 or in the content profile database 122 installed separately from the content database 121. Details of the content profile will be described later.
  • FIG. 3 shows a configuration example of a device arranged in a space where the display device 142 or the second user exists.
  • the display device 142 is basically used for displaying content (or viewing by a second user) acquired on the first user side as a photographer.
  • the display device 142 has a UI function in addition to the content display function, and can display information related to the content recommended by the recommendation system (described above) or perform a content selection operation by the second user. To do.
  • the display device 142 is equipped with an audio output device such as a speaker, and the object collected at the shooting site (or the first user's space) where the content is acquired. And an audio signal integrated with the video stream, such as an audio signal generated by the first user at the time of shooting, or the like.
  • This audio output device may be composed of a multi-channel speaker so that the sound image can be localized.
  • a voice input device such as a microphone, a camera for photographing the second user and a companion, a character input device such as a keyboard, and a coordinate input device such as a mouse and a touch panel
  • a voice input device such as a microphone, a camera for photographing the second user and a companion
  • a character input device such as a keyboard
  • a coordinate input device such as a mouse and a touch panel
  • the input device 301 may include an input device that is used by being worn on the viewer's body, such as gloves or clothes, and that can directly input the movement of the fingertip or torso.
  • the second user who is viewing the real-time content can also input an instruction (work support or the like) to the first user who is the photographer of the content using the input device 301.
  • an instruction from the second user is output from the output device 203 in the first user space.
  • a sensor device 302 that detects a situation in the vicinity of the second user that dynamically changes such as a viewing environment.
  • the sensor device 302 includes various environmental sensors that detect information related to the weather in the space where the second user exists (or when viewing content), such as temperature, humidity, atmospheric pressure, and illuminance.
  • the sensor device 302 may include a biological sensor that detects viewer's biological information such as body temperature, pulse, sweating, expiration, and brain waves.
  • the sensor device 302 includes a photographing device that photographs the second user who is the viewer and the accompanying person, and acquires information on the user and the accompanying person through processing such as face detection and face recognition on the captured image. You may make it do.
  • the senor 302 may include a position sensor that measures the current position of the display device 142 or the second user.
  • the position sensor receives a GNSS signal from a GNSS satellite, performs positioning, and generates position information including the latitude, longitude, and altitude of the vehicle.
  • the position sensor may specify the current position based on the measurement information from the wireless access point using PlaceEngine (registered trademark) or the like.
  • Sensor information detected by the sensor device 302 can be handled as information associated with the space of the second user. Further, the sensor information detected by the sensor device 302 during the period when the received content is displayed on the display device 142 (or during the period when the second user is viewing the content) is associated with the content viewing period. Can also be handled as information.
  • an output device 303 that performs output other than sound may be provided.
  • the output device 303 adjusts temperature and humidity, blows wind (light breeze, headwind, air blast) and splashes (water blast) on the viewer, and touches the viewer's body (effects such as poking back) , A feeling that something touches the neck or feet, etc.), vibrations, light electrical stimulation, scents and scents to control the environment of the second user's space (Or a multimodal interface).
  • the output device 303 is driven based on environmental information measured by the sensor device 202 on the imaging device 101 side, for example, so that the viewer can have a real and realistic experience similar to the shooting location. it can.
  • the output device 303 may be driven based on the analysis result of the content displayed on the display device 142 to add an effect for the second user who views the content.
  • the communication device 304 transmits information via the network 130 via the PC 141.
  • the communication device 304 can be configured to directly transmit information via the network 130 without using the PC 141.
  • the communication device 304 can transmit an access request for the first user or content directly to the imaging device 101 or indirectly via the content distribution server 120.
  • the communication device 304 outputs to the PC 141 the self position estimated by a stereo camera, a 9 DoF sensor, or the like and the viewer's line-of-sight direction. Then, the video rendered by the PC 141 is received by the communication device 304 and displayed on the display device 142.
  • the communication device 304 may be configured to receive the content of the free viewpoint video via the network 130 without using the PC 141 and perform the video rendering process based on the self-position and the line-of-sight direction in the display device 142. .
  • the communication device 304 can transmit the input information input to the input device 301 while the second user is viewing the video to the imaging device 101 side via the PC 141 and the network 130. Further, the communication device 304 can receive the output information via the PC 141 and the network 130 and output the output information from the output device 303 to the second user.
  • the display device 142 may be configured in the form of a wearable device worn by the viewer, such as a head-mounted display.
  • a head-mounted display may be equipped with all peripheral devices such as the display device 142, the input device 301, the sensor device 302, the output device 303, and the communication device 304. It is assumed that the display device 142 and the devices 301 to 304 are interconnected.
  • the connection form is not limited to a specific interface standard, and may be either a wired connection or a wireless connection.
  • the display device 142 and the peripheral devices 301 to 304 are disposed directly or indirectly with respect to a predetermined housing 305 that can be carried by the second user.
  • the direct arrangement corresponds to, for example, being built in the housing 305
  • the indirect arrangement corresponds to, for example, external connection to the housing 305 via a connector, a cable, or wireless communication.
  • the second user views the free viewpoint video on the display device 142 side, that is, as attribute information at the time of using the content, demographic information of the second user as a viewer, behavior and state of the second user History information regarding the second user's biometric information detected by the sensor device 302 while viewing the content, companion information viewing the content together with the second user, and environment information viewing the free viewpoint video. be able to.
  • the attribute information at the time of using these contents is also referred to as “user profile” below.
  • the performance and output format of the display device 142 may be included in the user profile.
  • the user profile for each viewer or each viewing time may be stored in a memory inside the display device 142 (or a device in which the display device 142 and its peripheral devices 301 to 304 are integrated).
  • a user profile for each viewer or each viewing time may be stored in the user profile database 131 constructed on the network 130. Details of the user profile will be described later.
  • ⁇ (2) Profile information> The content profile for each photographer or content is aggregated in the content profile database 122, and the user profile for each viewer or viewing time is aggregated in the user profile database 131.
  • the content profile and the user profile include information similar to each other.
  • profile information having a common data format.
  • some of the information elements (variables) that make up profile information can be used only as a content profile, or conversely, only as a user profile.
  • the variable value should be treated as zero.
  • the matching process is performed by comparing the profile information, that is, calculating the similarity and searching for a user or content having a high similarity. And the recommendation information which listed the candidate with high similarity can be shown to a user.
  • FIG. 4 illustrates information elements constituting the profile information.
  • Profile information includes socioeconomic characteristics such as the sex, age, height / weight, address (residence or work location), hometown, income, occupation or company name, educational background, family composition of the photographer or viewer. Includes demographic information consisting of.
  • the demographic information generally includes a fixed value or static information that changes slowly.
  • demographic information is known in the field of socio-economics that it is strongly linked to consumer behavior such as purchasing and using products, and is also widely used in the field of information technology. In the present embodiment, it is expected that demographic information is strongly linked to JackIn (that is, the content associated with the first user or the first user's space and the second user who views the content). Is done.
  • the profile information may include history information related to the actions and states of the photographer and viewer, such as behavior history, purchase history, viewing history, medical history, and system usage history.
  • the action history includes, for example, information such as a place where the photographer or viewer has visited in the past and a moving route.
  • the viewing history is a viewing history of commercial contents such as movies and TV programs.
  • the system usage history includes a history that the user has used the free viewpoint video distribution system 100 in the past.
  • “use” refers to a history in which a certain user has transmitted content such as a free viewpoint video acquired by the imaging apparatus 101 as the first user to the content distribution server 120 or the second user, and the second user.
  • content such as a free viewpoint video acquired by the imaging apparatus 101 as the first user to the content distribution server 120 or the second user, and the second user.
  • both history of receiving and viewing content such as free viewpoint video by requesting access to the first user.
  • the system usage history as the first user includes the shooting location (or the location from which the content was acquired), date and time, shooting parameters, content transmission history, and the like.
  • the system usage history as the second user includes the access history to the content, the attribute information of the content that has been viewed (content name, content storage location, information for identifying the first user who has taken the image, etc.) , Content playback history (playback section, self-position and line-of-sight direction during playback, displayed angle of view), and the like.
  • the profile information further changes from moment to moment in the imaging device 101 or the photographer, such as environmental information measured by the sensor device 202 around the imaging device 101, the first user's biological information, and accompanying information accompanying the photographing.
  • Dynamic information (that is, information associated with the content information acquisition period) may be included.
  • the information associated with the content information acquisition period changes from time to time in real time.
  • the information associated with the content information acquisition period dynamically changes for each playback section of the content stream.
  • the profile information is a behavior that changes from moment to moment in the display device 142 or the viewer, such as environmental information measured by the sensor device 302 around the display device 142, biological information of the viewer, or companion information accompanying the viewer.
  • Target information that is, information associated with the viewing period of the content information may be included.
  • Demographic information includes information with high personal identification and personal information related to privacy even without personal identification.
  • the history information and the dynamic information also include personal information related to privacy and information that is not related to privacy but that the photographer and viewers do not want to leak out. For this reason, it is necessary to be careful when handling profile information.
  • content meta information which can be referred to as a photographer's shooting log, can be included as a content profile.
  • Matching processing can also be performed by calculating the similarity of content / meta information between contents.
  • the content meta information includes a genre, length, content feature amount, photographer information, shooting or public start time for each content, and is managed in association with an item identifier for identifying the content.
  • the content / meta information can be created based on, for example, the environment information measured by the camera / work or the sensor device 202 in the imaging apparatus 101, the transmission history of the video, and the analysis result of the content.
  • the content meta information may be managed together in the content profile database 122 as a part of a component constituting the content profile, or the content meta information may be managed on the free viewpoint video distribution system 100.
  • -A database (not shown) may be installed.
  • the user profile further includes viewing history information of a free-viewpoint video content stream in the free-viewpoint video distribution system 100, which should be called a viewer's viewing log. Can do.
  • the matching process can also be performed by calculating the similarity of the viewing history information between the second users.
  • the viewing history information includes a viewer identifier (userId) for identifying the viewer, a content identifier (contentId) for identifying the viewed content, a feedback type, a time stamp, and the like.
  • the viewing history information is information generated each time each viewer views the content. Even for the same content, there is viewing history information for each viewer.
  • the viewing history information can be created based on, for example, content viewing history fed back from the display device 142 or device operation history.
  • the viewing history information may be managed together in the user profile database 131 as a part of components constituting the user profile, or the dedicated content viewing status database on the free viewpoint video distribution system 100 (Not shown) may be installed.
  • the first user is a user who uses the free viewpoint video distribution system 100 as a photographer of content such as a free viewpoint video
  • the second user is a user who uses the system 100 by viewing the content.
  • FIG. 1 only one first user and two second users are depicted for simplification, but it is assumed that there are actually a large number of first users and second users.
  • the willingness to use the free viewpoint video distribution system 100 will increase.
  • the content captured by one of the first users is viewed by a number of second users or viewed by a second user who wants to watch the content, the first user continues to shoot and continue to capture the content. Increased willingness to provide. Therefore, it is extremely important to further promote the free viewpoint video distribution system 100 by recommending appropriate content to the second user to encourage viewing.
  • the matching process is performed in a recommendation system installed inside or outside the free viewpoint video distribution system 100 (described above).
  • the content profile that is the attribute information of the content of the free viewpoint video captured by the imaging apparatus 101 and the user profile that is the attribute information when the content on the viewer side is used can be used.
  • the content profile and the user profile are composed of similar information elements (variables) (see FIG. 4).
  • similarity of profile information is calculated, and a user or content whose similarity is higher or exceeds a predetermined value is found as a candidate. Then, recommendation information including information regarding one or more candidates is presented to the user.
  • the “recommended content” mentioned here includes both real-time content currently photographed by the photographer and content archived in the content database 121 in the past.
  • a format for presenting recommendation information to the second user there are a format for presenting a content candidate list and a format for presenting a first user candidate list as a connection destination.
  • the former content list can include both real-time content that is currently being shot and content that has been shot in the past and stored in the content database 121.
  • the latter list of connection destinations can include both the first user who is currently photographing and the first user who has photographed in the past.
  • the candidate list is displayed on the UI screen. Then, the second user can select content or a connection destination that the user wants to view by UI operation.
  • the second user can select the second one from the plurality of first users or contents.
  • the latter is a case where the second user, for example, content related to content that the user has viewed in the past, or other first user related to the first user who captured the content that has been viewed in the past (or This is a use case for selecting a related content by photographing of a first user.
  • CBF Content-based Filtering
  • CF Cold-based Filtering: collaborative filtering
  • CBF is a method of selecting information based on the content of recommended information. For example, the information requested by the user can be selected by comparing the content of the information with the user's request.
  • the content of information corresponds to the content profile on the first user side
  • the user request corresponds to the user profile on the second user side
  • the second The first user (or content obtained by shooting by the first user) that matches the user's preference is selected.
  • a first user or content whose similarity is higher or exceeds a predetermined value by calculating the similarity between the user profile of the second user and the content profile of the first user or content Can be found as a candidate close to the preference of the second user. For example, for a second user who frequently views tourism-related content, the first user who has a profile relating to tourism, or content captured by such a first user can be recommended as a candidate. .
  • another first user similar to an arbitrary first user is calculated by calculating the similarity of content profiles between first users (or between contents). (Or content obtained by shooting by the first user) and other content similar to arbitrary content can be found as candidates. For example, another first user having a profile similar to the first user that the second user has watched in the past, or a first user having a profile similar to the favorite first user. Can be recommended.
  • CBF approach (1-1) A photographer having profile information close to the viewer is recommended. (1-2) A photographer having profile information close to an arbitrary photographer is recommended.
  • CF is a method of selecting information based on user information. For example, information requested by a user can be selected using information of other users with similar preferences.
  • the user information corresponds to the user profile on the second user side, and based on the information of another second user similar to a certain second user, The first user (or content captured by the first user) is selected.
  • a first profile in which the similarity between user profiles is calculated between the second users and content that has been viewed by another second user with a similar preference in the past is captured.
  • a user or content obtained by shooting by the first user
  • the viewer A views each content photographed by the photographers 1, 2, 3, 4 and the viewer B views each content photographed by the photographers 1, 2, 3, and so on. Since viewer A and viewer B have similar preferences (content viewing history), the photographer 4 (or content captured by the photographer 4) is recommended to the viewer B.
  • another first user (or the first user's view) of another second user who has viewed the content photographed by an arbitrary first user. Content by shooting) can be recommended. For example, when the viewer A and the viewer B both watch the content photographed by the photographer 1, the viewer 2 recommends the viewer B for the other content that the viewer A has further viewed.
  • the matching process has been described in the case where the second user who is the viewer selects the first user or content in an initiative.
  • the recommendation information of the viewer can be similarly presented by the CBF approach or the CF approach.
  • the control unit of the content recommendation server 150 installed on the network 110 and the network 130 executes the calculation of the matching process, and the recommended information as the calculation result is sent to each user. Shall be presented.
  • content profiles of a number of first users who are photographers or content are aggregated in the content profile database 122
  • user profiles of a number of second users who are viewers are user profile profiles.
  • the data is collected in the database 131.
  • the content recommendation server 150 uses the profile information stored in the content profile database 122 and the user profile database 131, and executes matching processing.
  • the matching process can be performed at any place on the free viewpoint video distribution system 100 other than the content recommendation server 150.
  • a part or all of the calculation for the matching process can be performed on the imaging device 101 or its peripheral device, or in the control unit such as the display device 142 or the PC 141.
  • the content recommendation server 150 or the display device 142 or the PC 141 is provided with a matching processing unit that executes a matching process
  • the control unit of the content recommendation server 150 or the control unit of the display device 142 or the PC 141 is a matching processing unit. It is good also as a structure which performs a matching process.
  • FIG. 5 illustrates a functional configuration for executing the recommendation process.
  • the content profile of the first user and the content is aggregated in the content profile database 122, and the user profile of the second user is aggregated in the user profile database 131.
  • the recommendation information processing unit 501 generates recommendation information for the second user by using the profile information in the database 131 according to the CBF approach or the CF approach described above and the content profile database 122 and the user profile.
  • the recommendation information processing unit 501 is constructed on the content recommendation server 150. However, in the case 305 including the display device 142 of the second user leading the matching process or the case 205 including the imaging device 101 of the first user leading the matching process, the recommended information processing is performed.
  • a configuration example of a system in which the function of the unit 501 is arranged can be considered.
  • the recommendation information sent from the recommendation information processing unit 501 is displayed on the UI screen of the display device 142.
  • the recommendation information includes a candidate list, and the second user can select content to be viewed or the first user through a UI operation. In response to this selection operation, an access request to the first user is generated, and the content distribution server 120 The distribution of the content stream to the second user is started by the intervention.
  • a playback video of the content is displayed.
  • the second user may perform operations related to content viewing and input feedback information, such as content playback start, stop, pause, fast forward, and rewind, via the UI screen of the display device 142. Can do.
  • the recommended information processing unit 501 receives the viewing history information fed back from the second user side such as the content selected by the second user or the selection result of the first user, the operation history related to the playback of the viewing content, and the like. Save as.
  • the recommended information processing unit 501 learns information fed back from the viewer side and stores it as user preference information 503.
  • the profile creation unit 504 creates profile information such as a content profile and a user profile, and stores the profile information in the content profile database 122 and the user profile database 131.
  • the profile creation unit 504 stores content / meta information created based on environment information measured by the camera / work or the sensor device 202 in the image capturing apparatus 101, a video transmission history, and a content analysis result. They are managed together in the content profile database 122 as a part of components constituting the profile, or stored in a content meta database (not shown).
  • the profile creation unit 504 updates the user profile in the user profile database 131 based on the second user's viewing history information, user preference information 503, and the like stored as the feedback log 502.
  • the recommended information processing unit 501 is realized as an internal process of the content recommendation server 150, but a system configuration method that is realized as an internal process of the display device 142 (or the PC 141) or the imaging device 101 is also assumed.
  • the content profile and user profile used for the matching process are composed of a large number of information elements (variables) (see FIG. 4). Although it is expected that content can be recommended with higher accuracy by matching using more information elements, the amount of calculation for matching processing becomes enormous, increasing the calculation cost, There is a concern that the waiting time until an appropriate candidate is found becomes longer.
  • the content profile and user profile used for the matching process consist of many information elements such as demographic information, history information, and dynamic information. Among these, information related to the privacy of each user is included. Should be included and kept secret. For this reason, there is a concern that confidential information may be leaked from inside a computer (such as the content recommendation server 150) that executes the calculation of matching processing, or the confidential information may be intercepted on a transmission path or the like.
  • demographic information includes highly personally identifiable information and personal information related to privacy even without personally identifiable information.
  • history information and dynamic information include personal information related to privacy and information that is not related to privacy but is not relevant to users, it is also sufficient for handling. You need to be careful.
  • Secure calculation protocol is known as a technique to achieve both prevention and use of confidential information. Therefore, this specification proposes a technique for recommending content with higher accuracy by using a secure calculation protocol to perform matching processing based on profile information including confidential information while preventing leakage.
  • a plurality of entities (computation parties composed of computer devices, etc.) cooperate to perform a calculation without disclosing the input values to each other, thereby calculating a predetermined calculation formula. Is obtained (see, for example, Patent Document 3). Therefore, by treating the confidential information as an input value and executing the secure calculation protocol, it is possible to obtain a calculation result derived from the confidential information as an output value without revealing the confidential information that is the input value to each calculation party. it can.
  • the secure calculation protocol is also referred to as multi-party calculation (MPC).
  • Fig. 6 shows an overview of the secure calculation protocol.
  • the calculation is performed by four calculation parties 60a to 60d.
  • Secret calculation values x1 to x4 that are secret from each other are respectively input to the calculation parties 60a to 60d (see FIG. 6A).
  • the calculation parties 60a to 60d cooperate to execute the secure calculation protocol (see FIG. 6B).
  • the secure calculation protocol information can be communicated between the calculation parties 60a to 60d, but calculation processing is performed so that only the secret input values x1 to x4 can be known.
  • the function f (x1, x2) which is the final output value in a state where the secret input values x1 to x4 are kept secret from the calculation parties 60a to 60d. , X3, x4) is obtained (see FIG. 6C).
  • FIG. 7 shows a data flow in the secure calculation protocol between the two calculation parties 70a and 70b.
  • FIG. 8 shows a schematic processing procedure of the secure calculation protocol in the form of a flowchart.
  • a calculation method called a secret sharing method can be suitably applied to the secure calculation protocol.
  • each calculation party 70a, 70b obtains a function f (x, y) as an output value from two input values (x, y), it divides the secret information into several pieces of shared information and collects the shared information to make a secret. It is assumed that secure calculation protocol calculation processing using the “secret sharing method” for restoring information is performed.
  • the number of calculation parties and the number of input values are not limited to the illustrated example, and a secure calculation protocol based on a different number of input values can be similarly implemented by more calculation parties.
  • secret input values x and y are input to the respective calculation parties 70a and 70b as pre-processing for executing the secure calculation protocol. Throughout the process of the secure calculation protocol, the secret input values x and y are kept secret from each other at the calculation parties 70a and 70b.
  • the secret input values x and y are distributed to the calculation parties 70a and 70b (step S801). For example, as shown in FIG. 7, distributed values x1 and x2 are generated from one secret input value and distributed to the respective calculation parties 70a and 70b. Similarly, distributed values y1 and y2 are generated from the other secret input value and distributed to the respective calculation parties 70a and 70b.
  • the secure calculation for the distributed value means that a calculation protocol corresponding to a predetermined calculation formula is executed using the distributed value as an input value.
  • the predetermined calculation formula is a function f (x, y) having input values x and y as input values. Then, as a result of executing this calculation protocol for the distributed value at each calculation party 70a, 70b, fa (x1, y1) is calculated at the calculation party 70a, and fb (x2, y2) at the calculation party 70b. Is calculated.
  • Step S803 may be performed at one or both of the calculation parties 70a and 70b, and the calculation results fa (x1, y1) and fb (x2, y2) obtained from the calculation parties 70a and 70b It may be implemented at a calculation party (not shown).
  • the secure calculation protocol is configured by combining secret input value distribution, distributed value secure calculation, and output value decryption.
  • Each secret input value is distributed to each calculation party 70a, 70b as a plurality of distributed values, and each calculation party 70a, 70b performs a secure calculation using the distributed value as an input value. Therefore, since the calculation is executed without the secret input values being disclosed to each other at each calculation party 70a, 70b, it is possible to perform a desired calculation on the secret input value while ensuring the confidentiality of the secret input value. is there.
  • each calculation party may be the same party or different parties.
  • a login user is connected to a photographer (first user), which is performed when a user who wants to view content (second user) logs in to the free viewpoint video distribution system 100.
  • the basic procedure is shown in the form of a flowchart.
  • step S901 When a user who desires to view content has logged in to the free viewpoint video distribution system 100 (step S901), the content recommendation server 150 searches for a photographer or content suitable for the login user, that is, viewer-led matching. Processing is executed (step S902).
  • the content recommendation server 150 uses the user profile and the content profile to appropriately perform viewer-driven matching processing according to the CBF approach or the CF approach described above as appropriate.
  • a secure calculation protocol (described above) is applied to the matching process.
  • step S903 when one or more connection destinations (photographers or contents) suitable for recommendation to the login user can be found (Yes in step S903), the UI screen The recommended information including the candidate list is presented to the login user.
  • the login user selects one of the candidates from the candidate list
  • the selected photographer is currently capturing real-time content, recorded content previously captured by the photographer, or login user.
  • the streaming distribution of the content selected by is started (step S904), and this processing routine is completed.
  • the logged-in user can view the received content stream video on the display device 142 owned by the login user.
  • FIG. 10 also connects a viewer (second user) to the logged-in user, which is performed when a user (first user) who captures content logs in to the free viewpoint video distribution system 100.
  • the basic procedure is shown in the form of a flowchart.
  • the content recommendation server 150 views content (including recorded content shot in the past) taken by the login user.
  • the viewer is searched for, that is, a matching process led by the photographer is executed (step S1002).
  • the content recommendation server 150 uses the user profile and the content profile to appropriately perform a photographer-initiated matching processing operation according to either the above-described CBF approach or CF approach.
  • a secure calculation protocol (described above) is applied to the matching process.
  • step S1002 when one or more connection partners suitable for recommendation to the login user, that is, viewer candidates are found (Yes in step S1003), candidates are displayed on the UI screen or the like.
  • Present recommendation information including a list.
  • Step S1004 When the login user selects one of the candidates from the candidate list, streaming distribution of content (including recorded content captured in the past) taken by the login user to the selected viewer starts. (Step S1004), and this processing routine is completed. However, when a connection with the selected viewer is performed, or when streaming of content to the selected viewer is started, processing for obtaining viewing consent from the viewer may be executed. . On the viewer side, the display device 142 receives and displays the content stream via the PC 141, and the viewer can view the video.
  • profile information such as a user profile and a content profile includes a large number of information elements
  • the amount of calculation of the matching processing executed in the above steps S902 and S1002 is enormous.
  • the user profile and the content profile include confidential information, it is preferable to apply a secure calculation protocol to the calculation of matching processing to prevent leakage of confidential information.
  • secure calculations include operations that take time, generally a huge amount of processing time is required compared to normal calculations.
  • step S902 or step S1002 and recommendation information is presented to the user becomes longer, leading to a decrease in service.
  • this specification proposes methods (1) to (3) for improving the efficiency of the matching process.
  • these methods (1) to (3) for users using the free viewpoint video distribution system 100, the waiting time until recommendation information regarding an appropriate connection destination is presented is shortened, and the service is improved. To do.
  • any one of the methods (1) to (3) is not used alternatively, but two or more methods may be combined.
  • the profile information is composed of a number of information elements. By matching using more information elements, it is expected that content can be recommended with higher accuracy, but the amount of calculation for matching processing is enormous. Therefore, by limiting the information elements used for the matching process in the profile information, the information transmission amount for the matching process is reduced, the calculation amount of the matching process is reduced, and the waiting time of the user is reduced accordingly. can do.
  • limited use using only a part of a large number of information elements included in the profile information for matching processing is referred to as “limited use”.
  • the lead user When applying the method of limited use of profile information, as a method of selecting information elements to be used for matching processing, the lead user (viewer or photographer) will be limited to manual operation via a dedicated UI screen.
  • FIG. 11 shows a configuration example 1100 of a dedicated UI screen for selecting information used by the user for matching processing.
  • the profile information is classified into five types: “demographic information”, personal “hobby preference list”, “behavior history information”, “friend list”, and “web browsing history”
  • check boxes for individually selecting each information element are provided. Further, in order to facilitate the selection operation, “select all” and “cancel all” buttons are also provided, and all information elements can be selected at once or can be canceled at once.
  • the user does not want to actively select information elements that may be used or conversely (or do not want to be leaked outside). Can be removed from the selection of high information elements.
  • the user can deselect all confidential information such as privacy information on the dedicated UI screen 1100.
  • the content recommendation server 150 or the like is configured to be able to appropriately switch whether or not to apply the secure calculation protocol, the secure calculation is performed according to the selection result of the information element on the dedicated UI screen 1100 by the user. You may make it control whether a protocol is applied.
  • a “do not apply secure calculation” button may be provided in the dedicated UI screen 1100 so that the user can select whether or not to apply the secure calculation protocol. The user can choose to reduce the computation time at the expense of information concealment.
  • the dedicated UI screen 1100 displays, as options, information elements that are mainly related to privacy and should be concealed among profile information. However, information elements that do not need to be concealed may be included in the options, and the dedicated UI screen 1100 may be configured so that the user can specify whether or not to use the information limitedly.
  • the control part which performs a matching process is good also as a structure which shows the information regarding the processing speed when a matching process is performed based on the information element selected by the user via an output part.
  • an indicator indicated by reference numeral 1101 is provided below the dedicated UI screen 1100 shown in the figure so that the degree of speedup when matching processing is performed using the currently selected information element is displayed as an index. May be.
  • the present invention is not limited to such a UI presentation process. It is good also as a structure which outputs by audio
  • the content recommendation server 150 automatically selects information elements of profile information used for matching processing between users, and performs limited use.
  • profile information used for matching processing between users may be automatically selected in accordance with the purpose of user connection (JackIn).
  • the second user can connect to the first user and view the content for various purposes such as viewing the content photographed by the first user, teaching the first user, and providing work support. It is.
  • the purpose of having the first user connected from the second user and distributing the photographed content to the second user for viewing is to publish the content that he / she has photographed or to teach or work from the viewer of the content. There are various things such as receiving support.
  • a large number of information elements included in the profile information are classified into a plurality of categories, and categories to be used for the matching process are defined in advance according to the purpose of connection of the first user from the second user. Then, when starting the first user-initiated matching process, the matching process is executed only by using the profile information only for the category associated with the connection purpose of the user. Similarly, a category to be used for the matching process is defined in advance according to the purpose for the second user to connect to the first user. Then, when the second user-initiated matching process is started, the matching process is executed by using the profile information only for the category associated with the connection purpose of the user.
  • profile information used for matching processing between users may be automatically selected based on information associated with the first user's space such as sensor information detected by the sensor device 202. Further, the context information in which the first user is currently placed is further read from the sensor information, the connection purpose of the first user is determined using this context information, and used for matching processing according to the connection purpose. Automatically select profile information. By limiting the use of profile information by automatic selection in this way, it is possible to shorten the computation time while suppressing a decrease in the accuracy of matching processing, and to shorten the waiting time until recommendation information is presented. For the first user, there is also an advantage that it is possible to save the trouble of selecting connection purpose and profile information.
  • profile information used for matching processing between users may be automatically selected based on information associated with the second user's space, such as sensor information detected by the sensor device 302. Further, the context information on which the second user is currently placed is further read from the sensor information, and the connection purpose of the second user is determined using this context information, and used for matching processing according to the connection purpose. Automatically select profile information. By limiting the use of profile information by automatic selection in this way, it is possible to shorten the computation time while suppressing a decrease in the accuracy of matching processing, and to shorten the waiting time until recommendation information is presented. For the second user, there is also an advantage that it is possible to save the trouble of selecting connection purpose and profile information.
  • the limited use of profile information by automatic selection makes it possible to shorten the computation time while suppressing a decrease in the accuracy of the matching process, and to shorten the waiting time until recommendation information is presented.
  • the efficiency of the matching process and the waiting time can be shortened while saving the trouble of selecting the profile information by itself through a dedicated UI screen or the like.
  • profile information used for matching processing between users is automatically selected according to information associated with a content capture period in which the first user is shooting in the space. You may make it do.
  • the context information currently placed by the first user is read, and the connection purpose of the first user is determined using this context information, Profile information used for matching processing is automatically selected according to the connection purpose.
  • profile information used for matching processing between users may be automatically selected based on position information from which content is acquired.
  • FIG. 12 shows an example of a processing procedure for performing matching processing between users using the profile information in a limited manner in the free viewpoint video distribution system 100 in the form of a flowchart. It is assumed that the illustrated processing procedure is performed in the content recommendation server 150.
  • the dedicated UI screen 1100 as shown in FIG. 11 is displayed and a selection operation from the user is input (step S1202).
  • the dedicated UI screen 1100 is displayed using the output device 203, for example.
  • a dedicated UI screen 1100 is displayed on the display device 142. Then, the user can select only a specific category from the profile information via the dedicated UI screen 1100.
  • the connection purpose of the target user is set to information such as sensor information associated with the user's space, the user's context information, the first user It is determined based on information associated with the content information acquisition period in step S1204.
  • the target is the first user
  • the purpose of distributing the content is determined.
  • the second user is targeted, the purpose of viewing the content is determined.
  • the profile information only information elements included in the category associated with the determined connection purpose are automatically selected (step S1205).
  • the matching process by the content recommendation server 150 is executed only for the information elements selected by the target user in the profile information and the information elements included in the category automatically selected according to the connection purpose. (Step S1206).
  • a matching process is executed by applying a secure calculation protocol (see FIGS. 6 to 8) as necessary.
  • a secure calculation protocol see FIGS. 6 to 8
  • the secure calculation protocol is applied, there is a concern that the amount of calculation and the calculation time increase, and the waiting time until the recommendation information is presented to the user becomes long. Before starting the matching process, it may be confirmed with the user whether to apply the secure calculation protocol.
  • the recommended information obtained as a process result is presented to the target user (step S1207), and this process routine is terminated.
  • the recommendation information is displayed using, for example, the output device 203.
  • recommendation information is displayed on the display device 142.
  • recommendation information including a candidate list is presented to the user on the UI screen. Further, when no candidate is found, the user may be notified that there is no candidate.
  • connection between the target user and the selected user is started. For example, the distribution of the content to the second user selected by the first user is started, or the distribution of the content of the second user is started from the first user selected by the second user.
  • the limited use of profile information can reduce the amount of calculation compared to a case where all information elements of profile information are used for matching processing while suppressing a decrease in accuracy of matching processing. As a result, it is possible to prevent leakage of confidential information by applying a secure calculation protocol, and to shorten the waiting time until recommendation information is presented to the user.
  • the calculation for the matching process is a similarity calculation of profile information between users (or between users and contents) performed according to the above-described CBF approach or CF approach.
  • “execution in advance” as used herein means that the implementation is performed before the recommendation information is presented to the user, such as when the user logs in. Specifically, such as when there is a surplus in computing resources such as midnight, when the number of users using the free-viewpoint video distribution system 100 is increased, or when profile information of users or content is updated, etc. Prior to the timing at which a request for presenting information is generated, similarity calculation of profile information between users (or between users and content) is executed with a sufficient time margin. In addition, not only when all the matching processes are completed in advance but also that a part of the matching process is completed in advance is included in the “perform in advance”.
  • the first user becomes connectable from the second user when the imaging apparatus 101 starts capturing the space in which the first user exists or when the captured content is sent to the content distribution server 120.
  • the content recommendation server 150 starts the above-described pre-calculation at a timing when the first user is directly or indirectly notified that the first user is in a connectable state. Thereby, for example, when the first user or the second user connects, the waiting time until the recommendation information is presented to the user can be shortened.
  • the context information that was used when the above profile information was automatically selected can be used for pre-calculation.
  • the context information in which the first user is currently placed is read, and this context information is used.
  • the connection purpose of the first user is determined, and the timing at which the first user starts connection according to the connection purpose (timing at which shooting is started by the imaging device 101 or distribution of the shot content is started). Can be expected.
  • the context information in which the second user is currently placed is read, and the connection purpose of the second user is determined using this context information.
  • the timing at which the second user starts the connection (such as the timing at which viewing of the content is started) according to the connection purpose.
  • the POIs mentioned here may include a venue for a fireworks display or a stadium where sports competitions such as soccer are held.
  • the expected arrival time of the user at the POI is the estimated connection time for starting the shooting and distributing the content, that is, as the first user. Accordingly, when the content recommendation server 150 knows the estimated arrival time of the user at the POI, the content recommendation server 150 is in time for the estimated arrival time (at the estimated arrival time, the calculation of matching processing is completed or recommendation information is generated). The above pre-calculation is started. Thereby, the standby time until the recommendation information is presented to the user can be shortened.
  • FIG. 13 shows an example of a processing procedure for performing matching processing between users by pre-calculation in the free viewpoint video distribution system 100 in the form of a flowchart. It is assumed that the illustrated processing procedure is performed in the content recommendation server 150.
  • the calculation time of the matching process is estimated (step S1301).
  • the amount of profile information used for matching processing (whether to use profile information limitedly by user selection or automatic selection), search range (total number of combinations of users performing similarity calculation), etc. To do.
  • the timing at which a request for presenting recommendation information to the user occurs is estimated (step S1302). For example, based on the sensor information of the sensor device 202 on the first user side or the sensor device 302 on the second user side, context information of each user, or history information such as behavior history or viewing history, the recommendation information is obtained. The timing at which a request to be presented can be estimated.
  • the timing for starting the pre-calculation is when the time when the computing resources such as midnight enter the time zone when there is a margin, when the increase in the number of users using the free viewpoint video distribution system 100 is detected, For example, when an update is made. Further, the pre-computation start timing may be set when the remaining time until the timing when the request for presenting recommendation information to the user estimated in step S1302 is generated is short of the calculation time of the matching process estimated in step S1301. .
  • step S1303 the pre-calculation of the matching process by the content recommendation server 150 is executed (step S1304).
  • the matching process is executed by applying a secure calculation protocol (see FIGS. 6 to 8) as necessary.
  • a secure calculation protocol see FIGS. 6 to 8
  • the secure calculation protocol is applied, leakage of confidential information can be prevented, but there is a concern that the amount of calculation and the calculation time increase, and the waiting time until the recommendation information is presented to the user becomes long.
  • the process waits until the timing for presenting recommendation information to the user arrives (No in step S1305). Note that if it is necessary to recalculate during the standby from the end of the precalculation to the presentation of the recommendation information (Yes in step S1306), the process returns to step S1304 and the precalculation is performed again. Good. For example, when an increase in the number of users using the free-viewpoint video distribution system 100 is detected during standby, when the user or content profile information is updated, and when the standby time exceeds a predetermined time, The pre-calculation may be redone.
  • step S1305 When targeting the first user, the recommendation information is displayed using, for example, the output device 203. In addition, when targeting the second user, for example, recommendation information is displayed on the display device 142.
  • recommendation information including a candidate list is presented to the user on the UI screen. Further, when no candidate is found, the user may be notified that there is no candidate.
  • connection between the target user and the selected user is started. For example, the distribution of the content to the second user selected by the first user is started, or the distribution of the content of the second user is started from the first user selected by the second user.
  • the profile information is not used in a limited manner, but all necessary information elements are used, or as many information elements as possible are used, and the accuracy is high. Recommendation information can be obtained.
  • the matching process in the free viewpoint video distribution system 100 is realized by calculating similarity of profile information for all combinations of users, and a user whose similarity is higher or exceeds a predetermined value is recommended information. Can be presented. However, as the number of users increases, the number of combinations of users also increases, so the amount of calculation for similarity calculation becomes enormous. With only limited use of the above profile information and pre-computation alone, highly accurate recommendation information is waiting for a short time. It becomes difficult to present in time. Therefore, the amount of calculation of matching processing is reduced by actively utilizing the history information of each user in the free viewpoint video distribution system 100 and narrowing the search range.
  • the search range is narrowed based on the user's past viewing history of content, similarity calculation between users is performed, and recommendation candidates are extracted. For example, when searching for a recommendation candidate for the user A who wants to view the content, first, the user X that has captured the content that the user A has viewed in the past is searched, and then the content captured by the user X is searched. A set of other users who have a history of viewing is searched, and the similarity calculation is executed only for this set of users to find recommendation candidates.
  • the amount of calculation can be greatly reduced.
  • the matching process is limited to a set of users whose similarity is assumed to be high in advance, it can be expected that recommended information can be obtained with higher accuracy than searching by a random combination.
  • the search range is narrowed based on the history of viewing of the user in the past, the similarity between users is calculated, and the recommendation candidates are extracted. For example, when searching for a recommendation candidate for the user X who is photographing the content, first, the user A searches for the content photographed by the user X in the past, and then, the user A retrieves the content so far. A set of other users who have a history of viewing is searched, and the similarity calculation is executed only for this set of users to find recommendation candidates.
  • the viewing cycle such as annual, monthly, weekly, etc., or the pattern of viewing at a specific event, or the opportunity to view the content next time, or The timing at which the recommendation information should be presented to the second user can be predicted.
  • the content is requested every year, every month, every week such as every week, or every specific event. By analyzing such a pattern, it is possible to predict the next time the content is requested, or the timing when the recommended information should be presented to the first user.
  • FIG. 14 shows an example of a processing procedure for performing the matching process with the search range limited based on the user history information in the free viewpoint video distribution system 100 in the form of a flowchart. It is assumed that the illustrated processing procedure is performed in the content recommendation server 150.
  • the history information of the target user is acquired (step S1401). If it is a 2nd user, the 1st user who image
  • step S1402 the history pattern is analyzed from the acquired history information, and the next opportunity to present recommended information to the user is estimated.
  • step S1404 when the timing for starting the pre-calculation comes based on the opportunity estimated in step S1402 (Yes in step S1403), the content recommendation server 150 narrows the search range based on the history information acquired in step S1401, Pre-calculation of matching processing is executed (step S1404).
  • the matching process is executed by applying a secure calculation protocol (see FIGS. 6 to 8) as necessary.
  • a secure calculation protocol see FIGS. 6 to 8
  • the secure calculation protocol is applied, there is a concern that the amount of calculation and the calculation time increase, and the waiting time until the recommendation information is presented to the user becomes long. Before starting the matching process, it may be confirmed with the user whether to apply the secure calculation protocol.
  • the process waits until the timing for presenting the recommendation information to the user arrives (No in step S1405). If it is necessary to perform recalculation during standby from the end of precalculation until presentation of recommendation information (Yes in step S1406), the process returns to step S1404 to perform the precalculation again.
  • the pre-calculation may be redone when the user or content profile information included in the search range of the matching process is updated during standby or when the standby time exceeds a predetermined time.
  • step S1407 the recommended information is presented to the user (step S1407), and this processing routine ends.
  • the recommendation information is displayed using, for example, the output device 203.
  • recommendation information is displayed on the display device 142.
  • recommendation information listing a candidate list on the UI screen is presented to the user. Further, when no candidate is found, the user may be notified that there is no candidate.
  • connection between the target user and the selected user is started. For example, the distribution of the content to the second user selected by the first user is started, or the distribution of the content of the second user is started from the first user selected by the second user.
  • the search range is narrowed and the similarity calculation is performed. Compared to the case where the search range is a combination of all users. Thus, the amount of calculation can be greatly reduced. Therefore, the waiting time until the recommendation information is presented to the user can be shortened.
  • the search range is narrowed to reduce the amount of calculation, the need for limited use of profile information is reduced, and highly accurate matching processing is possible.
  • the calculation time is shortened by narrowing the search range, there is an effect that the timing for performing the pre-calculation can be delayed. Further, by predicting the user's next viewing opportunity based on the history information, it is possible to start pre-calculation at an appropriate timing, and to shorten the standby time.
  • the technology disclosed in this specification can take the following configurations.
  • Profile information of the first user and profile information of the content of at least one first user and a second user accessing content information associated with the space of the first user.
  • An information processing apparatus comprising a control unit that executes matching processing based on first profile information including at least one of the first profile information and second profile information including the profile information of the second user.
  • the control unit executes the matching process by limiting at least one of the information elements of the first profile information and the information elements of the second profile information to only a part thereof (1 ) Or the information processing apparatus according to (2).
  • the control unit is an information element selected by the first user or the second user as at least one of the information element of the first profile information and the information element of the second profile information.
  • the control unit provides a user interface for selecting at least one of the information element of the first profile information and the information element of the second profile information via the output unit. Or present to the second user, The information processing apparatus according to (4), wherein the control unit executes the matching process based on an information element selected via the user interface.
  • control unit presents information about a processing speed when the matching process is executed based on the selected information element via the output unit.
  • the control unit selects a part from at least one of the information elements of the first profile information and the information elements of the second profile information, and executes the matching process based on the selected information elements
  • the information processing apparatus according to (3).
  • the control unit includes information associated with the first user space or the second user space, the first user context information or the second user context information, The information processing apparatus according to (7), wherein the selection information element is determined based on at least one piece of information associated with the first user in an information acquisition period.
  • the control unit is configured to change the profile information used for the matching process when the user who is the target of the matching process changes when the calculation resource for performing the matching process has sufficient time,
  • the information processing apparatus according to (9), wherein the matching process is executed at any time when the content information associated with the first user's space can be acquired.
  • control unit determines a search range of a user to be referred to in the matching process based on history information of each user regarding viewing content.
  • the control unit determines the search range based on history information related to the viewing content of the second user or history information related to the content of the first user.
  • the information processing apparatus according to (12).
  • a matching processing unit that executes the matching process is further provided.
  • the control unit causes the matching processing unit to execute the matching process.
  • control unit recommends content selected based on a result of the matching process to the second user.
  • a computer program for causing an information processing apparatus to execute information processing In the control unit of the information processing apparatus, At least one first user and a second user accessing content information associated with the first user's space are at least one of the first user profile information and the content profile information.
  • the series of processes described in the specification can be executed by hardware, software, or a combined configuration of both.
  • the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run.
  • the program can be recorded in advance on a recording medium.
  • the program can be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as a built-in hard disk.
  • the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary.
  • the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.
  • an information processing apparatus an information processing method, and a computer program that execute processing related to matching between users are realized.
  • a secure calculation protocol is applied to matching processing between users using profile information including confidential information such as personal information, so that the calculation time becomes long.
  • the information processing apparatus reduces the amount of calculation of matching processing by using any method of limiting use of profile information, pre-calculation, user history information to limit the search range of a user, or combining two or more methods.
  • an information processing apparatus, an information processing method, and a computer program that execute processing related to matching between users are realized.
  • DESCRIPTION OF SYMBOLS 100 Free viewpoint video delivery system 101 ... Camera, 110 ... Network 120 ... Content delivery server, 121 ... Content database 122 ... Content profile database 130 ... Network, 131 ... User profile database 141 ... PC, 142 ... Display Device 150 ... Content recommendation server 201 ... Audio input device 202 ... Sensor device 203 ... Output device 204 ... Communication device 205 ... Case 301 ... Input device 302 ... Sensor device 303 ... Output device 304 ... Communication device 305 ... Case 501 ... Recommended information processing section

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

[Problème] L'invention a pour objet de réaliser un dispositif et un procédé de traitement d'informations, ainsi qu'un programme informatique, qui exécutent des processus liés à une mise en correspondance d'utilisateur à utilisateur. [Solution] Une mise en correspondance d'utilisateur à utilisateur, dans laquelle sont utilisées des informations de profil comprenant des informations confidentielles telles que des informations personnelles, emploie un protocole de calcul sécurisé, ce qui se traduit par une prolongation du temps de calcul. La présente invention concerne un dispositif de traitement d'informations qui obtient des quantités réduites de calcul de processus de mise en correspondance et un temps d'attente raccourci avant que des informations de recommandation soient présentées à l'utilisateur, par un usage limité d'informations de profil, un pré-calcul ou une limitation de l'étendue de recherche d'utilisateurs à l'aide d'informations d'historique d'utilisateur, ces procédés pouvant être utilisés soit isolément, soit en combinaison de deux ou plus.
PCT/JP2018/000170 2017-01-19 2018-01-09 Dispositif et procédé de traitement d'informations, et programme informatique Ceased WO2018135334A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2017-007281 2017-01-19
JP2017007281 2017-01-19

Publications (1)

Publication Number Publication Date
WO2018135334A1 true WO2018135334A1 (fr) 2018-07-26

Family

ID=62909024

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2018/000170 Ceased WO2018135334A1 (fr) 2017-01-19 2018-01-09 Dispositif et procédé de traitement d'informations, et programme informatique

Country Status (1)

Country Link
WO (1) WO2018135334A1 (fr)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2020198474A (ja) * 2019-05-30 2020-12-10 株式会社ｔｏｒａｒｕ 体験共有システム、体験共有方法
JP2022071467A (ja) * 2020-10-28 2022-05-16 住友電気工業株式会社 コミュニケーション支援プログラム、コミュニケーション支援方法及びコミュニケーション支援システム
WO2022267770A1 (fr) * 2021-06-24 2022-12-29 支付宝(杭州)信息技术有限公司 Procédé, appareil et système de publication et de découverte de ressource de données informatiques de confidentialité
JP2023521542A (ja) * 2021-01-10 2023-05-25 グーグル エルエルシー コンテンツ配信における情報に対するアクセスを保護するためのセキュアなｍｐｃおよびベクトル計算の使用
JP2023524356A (ja) * 2021-04-09 2023-06-12 グーグル エルエルシー 分類の正確さを改善するための機械学習モデリングデータの処理
JP2023532833A (ja) * 2021-03-08 2023-08-01 グーグル エルエルシー セキュアマルチパーティ計算を使用したフレキシブルなコンテンツ選択プロセス
US12323506B2 (en) 2021-03-08 2025-06-03 Google Llc Privacy preserving measurements using secure multi-party computation

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008177781A (ja) * 2007-01-17 2008-07-31 Hitachi Ltd 番組表自動生成システム、番組表自動生成方法、および番組表自動生成プログラム
JP2009141952A (ja) * 2007-11-16 2009-06-25 Sony Corp 情報処理装置、情報処理方法、コンテンツ視聴装置、コンテンツ表示方法、プログラム及び情報共有システム
JP2011503717A (ja) * 2007-11-01 2011-01-27 クゥアルコム・インコーポレイテッド ユーザ生成コンテンツ配信のサポートを改善する方法及び装置
WO2011090128A1 (fr) * 2010-01-22 2011-07-28 株式会社ドワンゴ Système de communication et procédé de communication
WO2012127799A1 (fr) * 2011-03-23 2012-09-27 パナソニック株式会社 Serveur de communication, procédé de communication, support de mémoire et circuit intégré
WO2015049810A1 (fr) * 2013-10-01 2015-04-09 株式会社電通 Système d'affichage d'images mobiles à points de vue multiples
JP2015194959A (ja) * 2014-03-31 2015-11-05 ソニー株式会社 情報処理装置、情報処理方法及びプログラム

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008177781A (ja) * 2007-01-17 2008-07-31 Hitachi Ltd 番組表自動生成システム、番組表自動生成方法、および番組表自動生成プログラム
JP2011503717A (ja) * 2007-11-01 2011-01-27 クゥアルコム・インコーポレイテッド ユーザ生成コンテンツ配信のサポートを改善する方法及び装置
JP2009141952A (ja) * 2007-11-16 2009-06-25 Sony Corp 情報処理装置、情報処理方法、コンテンツ視聴装置、コンテンツ表示方法、プログラム及び情報共有システム
WO2011090128A1 (fr) * 2010-01-22 2011-07-28 株式会社ドワンゴ Système de communication et procédé de communication
WO2012127799A1 (fr) * 2011-03-23 2012-09-27 パナソニック株式会社 Serveur de communication, procédé de communication, support de mémoire et circuit intégré
WO2015049810A1 (fr) * 2013-10-01 2015-04-09 株式会社電通 Système d'affichage d'images mobiles à points de vue multiples
JP2015194959A (ja) * 2014-03-31 2015-11-05 ソニー株式会社 情報処理装置、情報処理方法及びプログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
TAKANO, MIKI: "non-official translation (Instantly achieving ''easy'' speeding up with a little expense", NIKKEI PC21, vol. 14, no. 15, 1 August 2009 (2009-08-01), pages 70 - 73 *

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2020198474A (ja) * 2019-05-30 2020-12-10 株式会社ｔｏｒａｒｕ 体験共有システム、体験共有方法
JP2022071467A (ja) * 2020-10-28 2022-05-16 住友電気工業株式会社 コミュニケーション支援プログラム、コミュニケーション支援方法及びコミュニケーション支援システム
JP2023521542A (ja) * 2021-01-10 2023-05-25 グーグル エルエルシー コンテンツ配信における情報に対するアクセスを保護するためのセキュアなｍｐｃおよびベクトル計算の使用
JP7397212B2 (ja) 2021-01-10 2023-12-12 グーグル エルエルシー コンテンツ配信における情報に対するアクセスを保護するためのセキュアなｍｐｃおよびベクトル計算の使用
US12192340B2 (en) 2021-01-10 2025-01-07 Google Llc Using secure MPC and vector computations to protect access to information in content distribution
JP2023532833A (ja) * 2021-03-08 2023-08-01 グーグル エルエルシー セキュアマルチパーティ計算を使用したフレキシブルなコンテンツ選択プロセス
JP7440667B2 (ja) 2021-03-08 2024-02-28 グーグル エルエルシー セキュアマルチパーティ計算を使用したフレキシブルなコンテンツ選択プロセス
US12323506B2 (en) 2021-03-08 2025-06-03 Google Llc Privacy preserving measurements using secure multi-party computation
US12401502B2 (en) 2021-03-08 2025-08-26 Google Llc Flexible content selection processes using secure multi-party computation
JP2023524356A (ja) * 2021-04-09 2023-06-12 グーグル エルエルシー 分類の正確さを改善するための機械学習モデリングデータの処理
JP7422892B2 (ja) 2021-04-09 2024-01-26 グーグル エルエルシー 分類の正確さを改善するための機械学習モデリングデータの処理
WO2022267770A1 (fr) * 2021-06-24 2022-12-29 支付宝(杭州)信息技术有限公司 Procédé, appareil et système de publication et de découverte de ressource de données informatiques de confidentialité

Similar Documents

Publication Publication Date Title
US11449190B2 (en) User tailored of experience feeds
US20250139927A1 (en) Chroma key content management systems and methods
WO2018135334A1 (fr) Dispositif et procédé de traitement d'informations, et programme informatique
US10699482B2 (en) Real-time immersive mediated reality experiences
US10921878B2 (en) Virtual spaces, mixed reality spaces, and combined mixed reality spaces for improved interaction and collaboration
KR102432283B1 (ko) 콘텐츠를 공간 3d 환경에 매칭
JP6558587B2 (ja) 情報処理装置、表示装置、情報処理方法、プログラム、および情報処理システム
US20180279004A1 (en) Information processing apparatus, information processing method, and program
JP2021534474A (ja) 拡張現実環境内でのコンテンツの提案
US9338242B1 (en) Processes for generating content sharing recommendations
CN112639682A (zh) 在增强现实环境中的多设备地图构建和协作
US11024074B2 (en) Virtual spaces, mixed reality spaces, and combined mixed reality spaces for improved interaction and collaboration
US20200210137A1 (en) Virtual spaces, mixed reality spaces, and combined mixed reality spaces for improved interaction and collaboration
KR20150036713A (ko) 검출된 물리적 표시를 통한 사용자 관심 결정
KR20160112898A (ko) 증강현실 기반 동적 서비스 제공 방법 및 장치
US10037077B2 (en) Systems and methods of generating augmented reality experiences
CN108293073B (zh) 沉浸式临场感
JP2018163461A (ja) 情報処理装置、および情報処理方法、並びにプログラム
JP2020513705A (ja) ビデオフレームの複数の部分のフィンガープリントを生成することによって立体ビデオを検出するための方法、システム、および媒体
JP7202935B2 (ja) 注目度算出装置、注目度算出方法、および注目度算出プログラム
CN119790422A (zh) 社交记忆重新体验系统
JP2022521117A (ja) 改善された相互作用および協力のための仮想空間、複合現実空間、および組み合わされた複合現実空間
CN116325765A (zh) 在消息收发系统内针对视频选择广告
CN114341944A (zh) 计算机生成的现实记录器
JP2022532263A (ja) 拡張現実対話を定量化するためのシステム及び方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18741278

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18741278

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP