[go: up one dir, main page]

WO2025192145A1 - Système de traitement d'informations, procédé de traitement d'informations et programme - Google Patents

Système de traitement d'informations, procédé de traitement d'informations et programme

Info

Publication number
WO2025192145A1
WO2025192145A1 PCT/JP2025/004652 JP2025004652W WO2025192145A1 WO 2025192145 A1 WO2025192145 A1 WO 2025192145A1 JP 2025004652 W JP2025004652 W JP 2025004652W WO 2025192145 A1 WO2025192145 A1 WO 2025192145A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
information processing
virtual object
processing system
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/JP2025/004652
Other languages
English (en)
Japanese (ja)
Inventor
元 濱田
新太郎 筒井
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Publication of WO2025192145A1 publication Critical patent/WO2025192145A1/fr
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/131—Protocols for games, networked simulations or virtual reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Definitions

  • the present disclosure provides an information processing method including an information processing device acquiring object data for a display target, displaying a first virtual object based on the object data, accepting a request to change the display mode of the displayed first virtual object, and changing the display mode of the first virtual object based on the change request.
  • the present disclosure provides a program that causes a computer to function as an acquisition unit that acquires object data for a display target, a display control unit that displays a first virtual object based on the object data, and a reception unit that receives a request to change the display mode of the displayed first virtual object, wherein the display control unit changes the display mode of the first virtual object based on the change request.
  • FIG. 1 is a diagram (part 1) for explaining an example of an operation according to the first embodiment of the present disclosure.
  • FIG. 10 is a diagram (part 2) for explaining an example of an operation according to the first embodiment of the present disclosure.
  • FIG. 10 is a diagram (part 3) for explaining an example of an operation according to the first embodiment of the present disclosure.
  • FIG. 10 is a diagram (part 4) for explaining an example of an operation according to the first embodiment of the present disclosure.
  • FIG. 1 is a diagram (part 1) for explaining an example of a method for recognizing the size of a virtual object according to the first embodiment of the present disclosure.
  • FIG. 10 is a diagram (part 2) for explaining an example of a method for recognizing the size of a virtual object according to the first embodiment of the present disclosure.
  • FIG. 10 is a diagram (part 2) for explaining an example of a method for recognizing the size of a virtual object according to the first embodiment of the present disclosure.
  • FIG. 10 is a diagram (part 3) for explaining an example of a method for recognizing the size of a virtual object according to the first embodiment of the present disclosure.
  • FIG. 10 is a diagram for explaining the background of a second embodiment of the present disclosure.
  • 10 is a flowchart of an information processing method according to a second embodiment of the present disclosure.
  • FIG. 10 is a diagram for explaining an example of a flow of information processing according to a second embodiment of the present disclosure.
  • FIG. 10 is a diagram for explaining an example of a configuration of adjustment data according to a second embodiment of the present disclosure.
  • FIG. 1 is a hardware configuration diagram illustrating an example of a computer that realizes functions of a server or the like according to an embodiment of the present disclosure.
  • Fig. 1 is a diagram for explaining the background of an embodiment of the present disclosure, and in detail illustrates an example in which users 1a and 1b participate from two locations (location A and location B) in a 3D remote conference system using holoportation.
  • 6DoF Six Degrees of Freedom
  • 6DoF two-way communication is a technology for delivering 6DoF video (also known as 6DoF content) that allows users to look around in all directions (freely select their line of sight) and move freely in three-dimensional space (freely select their viewpoint).
  • 6DoF content dynamically and virtually recreates three-dimensional space using one or more three-dimensional objects, depending on the user's viewpoint position, line of sight, and field of view (field of view) at each instant of time.
  • users at each location are represented by avatars.
  • avatars For example, using VR technology, users can move around in a virtual space as avatars in 6DoF and communicate with each other.
  • holoportation is a technology that uses a sensor 300 (e.g., a camera) to capture and transmit the realistic image of a person (user 1a, 1b) or object located at a distance, as shown in Figure 1, and then uses VR, AR, and MR technologies to display the person, etc., as a 3D image 2a, 2b, etc.
  • a sensor 300 e.g., a camera
  • VR, AR, and MR technologies to display the person, etc., as a 3D image 2a, 2b, etc.
  • Holoportation allows users to communicate with a person located at a distance as if they were right in front of them, and is expected to dramatically reduce the sense of distance between people and make communication between remote locations smoother.
  • an embodiment of the present disclosure is applied to holoportation, in which the realistic appearance of a user 1 or an object is captured and transmitted by a sensor 300, and a 3D image 2 of the user 1 is displayed.
  • the embodiment of the present disclosure is not limited to application to such holoportation, and may be applied, for example, to displaying the realistic appearance of a user 1 or an object as a 2D image.
  • the embodiment of the present disclosure may be applied to displaying an avatar or icon that resembles the appearance of a user 1 or an object.
  • the information processing system 10 is a system for conducting remote recreation (such as a conference) by holoportation between, for example, point A (first point) where user 1a is located and point B (second point) where user 1b is located. Note that the information processing system 10 is not limited to remote recreation between two points, point A and point B, but may also be carried out between three or more points.
  • the information processing system 10 includes, for example, an information processing device 100a, a sensor 300a, and a display 400a, all located at point A. Furthermore, the information processing system 10 according to this embodiment includes, for example, an information processing device 100b, an input device (User Interface, UI) 200b, a sensor 300b, and a display 400b, all located at point B.
  • the information processing devices 100a and 100b are connected to each other via a communication network 500. Any communication method, whether wired or wireless, can be used in the communication network 500, but it is desirable to use a communication method that can maintain stable operation.
  • the information processing devices 100a and 100b may be separate devices as shown in FIG.
  • the information processing device 100a and the information processing device 100b do not have to be located at point A and point B, and may be located at point C, for example, and are not particularly limited.
  • the input device 200 is not limited to being located only at point B, and may also be located at point A, for example, and is not particularly limited. Below, each device included in the information processing system 10 according to this embodiment will be described in order.
  • the information processing device 100 can transmit data relating to the real appearance of a user 1 or an object captured by a sensor 300 (described later) to another information processing device 100, and can acquire data from the other information processing device 100 and display virtual objects such as a 3D image 2 based on the acquired data on a display 400 (described later).
  • the information processing device 100 can be, for example, a server, a desktop, tablet, laptop, or notebook PC (Personal Computer), a smartphone, or an HMD (Head Mounted Display). The detailed configuration of the information processing device 100 will be described later.
  • the input device 200 may be a camera (not shown) included in the sensor 300 described below. In this case, information such as gestures made by the user 1's hands or arms, line of sight, and blinks can be output to the information processing device 100 as operation information.
  • the input device 200 may be an inertial measurement unit (IMU) (not shown) attached to a part of the body of the user 1. In this case, information such as gestures and movements made by a part of the body of the user 1 can be output to the information processing device 100 as operation information.
  • IMU inertial measurement unit
  • the IMU may include, for example, an acceleration sensor, which is an inertial sensor that acquires acceleration, and a gyro sensor (angular velocity sensor), which is an inertial sensor that acquires angular velocity.
  • the input device 200 may be a microphone (hereinafter referred to as a microphone). In this case, information about the user 1's spoken voice can be output to the information processing device 100 as operation information.
  • the sensor 300 is, for example, an RGB camera (image capture device) installed at each location to capture a realistic image of the user 1 or an object.
  • the RGB camera can capture data necessary to construct 3D data related to, for example, the appearance and posture of the user 1 or an object, and the movements (gestures, etc.) of the user 1.
  • multiple cameras may be installed at each location.
  • the RGB camera can be, for example, a stereo camera or a camera equipped with a wide-angle lens or a fisheye lens.
  • the RGB camera captures images of people, objects, etc. using, for example, a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor as an image sensor.
  • CMOS complementary metal oxide semiconductor
  • CCD charge coupled device
  • the captured image of user 1 output from the RGB camera may be used to estimate user 1's position using SLAM (Simultaneous Localization and Mapping) technology.
  • SLAM Simultaneous Localization and Mapping
  • virtual objects such as 3D video 2 displayed toward user 1 can be adjusted based on user 1's position.
  • Sensor 300 may also include a camera that is attached to the head of user 1 and can detect user 1's line of sight.
  • virtual objects such as 3D video 2 displayed toward user 1 can be adjusted based on user 1's line of sight (viewpoint) detected by the camera.
  • Sensor 300 may also include a depth sensor. The depth sensor can obtain data on the distance to a subject and the subject's outer shape, and, for example, 3D video 2 of user 1 may be adjusted based on the detected outer shape of user 1.
  • the sensor 300 may include an IMU attached to the user 1, a positioning sensor, etc.
  • the posture and movement of the user 1 may be detected using sensing data from the IMU, and the 3D image 2 of the user 1 may be adjusted based on the detected posture and movement of the user 1 (e.g., skeletal posture information, etc., described below).
  • the positioning sensor is a sensor that detects the position of the user 1, and specifically, may be a GNSS (Global Navigation Satellite System) receiver, etc. In this case, the positioning sensor can generate sensing data indicating the latitude and longitude of the user 1's current location based on signals from GNSS satellites.
  • GNSS Global Navigation Satellite System
  • the positioning sensor since it is possible to detect the user's relative position from, for example, information from RFID (Radio Frequency Identification), Wi-Fi access points, wireless base stations, etc., such communication devices may be used as the positioning sensor.
  • the position of user 1 and the surrounding environment e.g., indoors or outdoors
  • the brightness of virtual objects such as user 1's 3D image 2 may be adjusted based on the estimation results.
  • the display 400 is a head-mounted display worn on the head of the user 1 and displays virtual objects such as 3D images to the user 1.
  • the head-mounted display can be either a non-transmissive display or a transmissive display.
  • a non-transmissive display can, for example, superimpose a virtual object (e.g., a 3D image 2 of the user 1 or an object, an avatar, an icon, etc.) (first virtual object) on a virtual space on a display that blocks the user 1's field of view.
  • Transmissive displays can be broadly divided into video see-through displays and optical see-through displays.
  • a video see-through display can capture and display a real space in front of the user 1 and superimpose a virtual object on the displayed real-space image.
  • An optical display can superimpose a virtual object on a real space using a display such as a half mirror that does not block the user 1's field of view.
  • the display 400 may be realized as an LED (Light Emitting Diode) light source or the like that is worn on the eye of the user 1 and projects an image directly onto the retina of the user 1.
  • LED Light Emitting Diode
  • the display 400 may be a projector that projects an image around the user 1.
  • the projector is realized, for example, by a laser projector or an LED projector, and has the function of projecting an image around the user 1 via a lens.
  • the configuration of the information processing system 10 described with reference to FIG. 2 is merely an example, and the configuration of the information processing system 10 according to this embodiment is not limited to this example. In other words, the configuration of the information processing system 10 according to this embodiment can be flexibly modified according to specifications and operations.
  • the information processing device 100 mainly includes a capture unit (acquisition unit) 102, an encoding unit 104, a position and orientation detection unit 106, a transmission unit 108, a reception unit 110, a decoding unit 112, an image generation unit 114, and a rendering unit (display control unit) 116. Furthermore, the information processing device 100 mainly includes an object data management unit 120, an object data sharing unit 122, an object adjustment operation recognition unit (reception unit) 124, and an adjustment data storage unit (storage unit) 126. Below, each functional unit of the information processing device 100 will be described in order.
  • the capture unit 102 can acquire object data necessary for constructing 2D/3D images, etc., relating to the appearance, posture, movement (gestures, etc.) of the captured user 1 or object (display target) from a sensor 300 (e.g., the above-mentioned RGB camera). The capture unit 102 can then output the acquired object data to the encoding unit 104, the position and posture detection unit 106, and the object adjustment operation recognition unit 124, which will be described later. In this embodiment, the capture unit 102 may associate identification information (e.g., an object ID) for identifying the acquired object data with the object data.
  • the object data may include sensing data, etc., from various sensors 300, such as a depth sensor, an IMU, and a positioning sensor.
  • the encoding unit 104 can compress and encode the object data acquired by the capture unit 102 and output the encoded data to a transmission unit 108, which will be described later.
  • the position and orientation detection unit 106 acquires skeletal orientation information of the user 1 based on the object data from the capture unit 102, and can output the acquired skeletal orientation information to the transmission unit 108 (described later) in association with identification information attached to the object data.
  • the skeletal orientation information of the user 1 can be acquired by existing technology using a 3D image of the user 1, a depth image, sensing data from the IMU, etc.
  • the transmitting unit 108 can transmit various types of data to the information processing device 100.
  • the transmitting unit 108 can transmit, together with the object data, information such as the identification information of the object associated with the object, the position information of the user 1, and skeletal posture information.
  • the transmitting unit 108 is a communication interface having the function of transmitting and receiving data, and is realized by, for example, a communication device (not shown) such as a communication antenna, a transmitting and receiving circuit, or a port.
  • the receiving unit 110 can receive various types of data from the information processing device 100.
  • the receiving unit 110 can receive object data as well as associated information such as object identification information, position information of the user 1, and skeletal posture information.
  • the receiving unit 110 can output the received various types of information to a decoding unit 112, an object data management unit 120, and an object data sharing unit 122, which will be described later.
  • the receiving unit 110 is a communication interface having the function of transmitting and receiving data, and is realized by, for example, a communication device (not shown), such as a communication antenna, a transmission/reception circuit, or a port.
  • the image generation unit 114 can perform image processing on the object data to display an image of a virtual object (first virtual object) based on the object data, and output the image to a rendering unit 116 (described later).
  • the rendering unit 116 can adjust the image output from the image generation unit 114 and display the adjusted virtual object on the display 400.
  • the rendering unit 116 can display an image according to the viewpoint of the user 1 to the user 1 by cutting out and displaying an image according to the viewpoint (position, posture, line of sight) of the user 1.
  • the rendering unit 116 can also adjust a virtual object, such as a 3D image 2 of the user 1, based on posture skeleton information of the user 1, for example.
  • the rendering unit 116 can adjust the display mode of a virtual object in response to a change request based on operation information from user 1.
  • changing the display mode means changing at least one of the size, brightness, position, shape, and resolution of an image or virtual object (e.g., 3D image 2 of user 1 or an object, 2D image, avatar, icon, etc.).
  • the object data management unit 120 manages the object data output from the receiving unit 110 and can output the object data to the rendering unit 116. Furthermore, when adjustment data is provided from the object adjustment operation recognition unit 124 (described later), the object data management unit 120 can store the adjustment data in the adjustment data storage unit 126 (described later), and further output the adjustment data to the object data sharing unit 122 (described later). Furthermore, when adjustment data corresponding to the acquired object data is already stored in the adjustment data storage unit 126, the object data management unit 120 can link the adjustment data to the object data and provide the adjustment data to the rendering unit 116.
  • the object data sharing unit 122 can share adjustment data with a communicating party (another information processing device 100). In particular, when the object data sharing unit 122 provides its own adjustment data to the other information processing device 100, it outputs the adjustment data to the transmitting unit 108. Furthermore, the object data sharing unit 122 determines whether or not to reflect the adjustment data sent from the other information processing device 100 in the image display performed on its own information processing device 100 side, and if so, outputs the adjustment data to the object data management unit 120.
  • the object adjustment operation recognition unit 124 recognizes (accepts) an object adjustment operation (change request) by user 1, and can output adjustment data related to adjustment of the display mode (size, brightness, position, shape, resolution, etc.) of a virtual object (e.g., a 3D image 2, 2D image, avatar, icon, etc. of user 1 or an object) to the above-mentioned object data management unit 120 and the later-described adjustment data storage unit 126.
  • the object adjustment operation recognition unit 124 recognizes, from the operation by user 1, which virtual object, which item of the display mode is to be adjusted, and to what extent, and outputs information about each of these as adjustment data to the object data management unit 120 and the adjustment data storage unit 126.
  • the object adjustment operation recognition unit 124 may recognize, for example, a gesture made by the user 1's hand or arm as the object adjustment operation made by the user 1.
  • the gesture made by the user 1 may be an operation performed on a virtual object. More specifically, an operation performed on a virtual object may be, for example, an action of the user 1 touching the virtual object or an action of grabbing the virtual object.
  • the gesture made by the user 1 may be an operation performed on another virtual object (a second virtual object) displayed toward the user 1 for accepting a request to change the display mode of the virtual object. More specifically, an operation performed on another virtual object may be, for example, an action of the user 1 touching the other virtual object.
  • the other virtual object for accepting a request to change the display mode may be a menu display, a button, a switch, a lever, a dial, a scale (measure) icon, etc.
  • the object adjustment operation recognition unit 124 may recognize the spoken voice of user 1 as the object adjustment operation of user 1. Furthermore, the object adjustment operation recognition unit 124 may recognize the operation of user 1 on the input device 200 as the object adjustment operation of user 1.
  • the adjustment data storage unit 126 can store the adjustment data.
  • the adjustment data includes, for example, data related to adjustment of the display mode (size, brightness, position, shape, resolution, etc.) of a virtual object (e.g., a 3D image 2 of a user 1 or an object, a 2D image, an avatar, an icon, etc.), and identification information of the object data corresponding to the virtual object.
  • a virtual object e.g., a 3D image 2 of a user 1 or an object, a 2D image, an avatar, an icon, etc.
  • the configuration of the information processing device 100 described with reference to FIG. 2 is merely an example, and the configuration of the information processing device 100 according to this embodiment is not limited to this example. In other words, the configuration of the information processing device 100 according to this embodiment can be flexibly modified according to specifications and operations.
  • FIG. 3 is a flowchart of the information processing method according to this embodiment.
  • the information processing method according to this embodiment includes a plurality of steps from step S101 to step S111. Each step of the information processing method according to this embodiment will be described below.
  • the information processing system 10 begins recognizing an operation (gesture, menu operation, controller operation) by user 1b to start the virtual object adjustment mode (step S101).
  • the operation by user 1b to start the virtual object adjustment mode may be, for example, a predetermined gesture, an operation on a menu display, or a predetermined operation on the controller.
  • the information processing system 10 determines whether or not it has recognized the operation by user 1b to start the virtual object adjustment mode (step S102). If the information processing system 10 determines that it has recognized the operation by user 1b to start the virtual object adjustment mode (step S102: Yes), it proceeds to the processing of step S103. On the other hand, if the information processing system 10 determines that it has not recognized the operation by user 1b to start the virtual object adjustment mode (step S102: No), it returns to the above-described step S101.
  • the information processing system 10 starts the virtual object adjustment mode (step S103).
  • the information processing system 10 for example, highlights, changes the color of, or blinks the virtual objects that can be adjusted, so that the user 1b can easily recognize the virtual objects that can be adjusted.
  • the information processing system 10 may also, for example, attach an icon to the virtual object indicating that it can be adjusted, so that the user 1b can easily recognize the virtual objects that can be adjusted.
  • the information processing system 10 confirms whether user 1b has selected all of the virtual objects they wish to adjust (step S105).
  • the information processing system 10 may, for example, display a menu and confirm user 1b's intention by having user 1b touch the displayed menu.
  • the information processing system 10 may, for example, confirm user 1b's intention by recognizing a predetermined gesture or speech of user 1b. If the information processing system 10 confirms that all of the virtual objects they wish to adjust have been selected (step S105: Yes), it proceeds to the processing of step S106. On the other hand, if the information processing system 10 confirms that all of the virtual objects they wish to adjust have not been selected (step S105: No), it returns to the above-mentioned step S104.
  • the information processing system 10 recognizes the selection of the item that user 1b wants to adjust (step S106).
  • the information processing system 10 may display a menu of items such as position, size (length), brightness, shape, and cancel adjustment, and recognize the selection of the item that user 1b wants to adjust when user 1b touches the displayed menu.
  • the information processing system 10 may recognize the selection of the item that user 1b wants to adjust by recognizing, for example, a predetermined gesture or speech of user 1b.
  • the information processing system 10 recognizes the adjustment operation of the virtual object by user 1b (step S107).
  • user 1b performs an operation corresponding to the item to be adjusted while looking at the target virtual object, making the adjustment.
  • an adjustment operation for example, user 1b may perform a gesture such as grabbing the target virtual object, moving it, and releasing it. In this case, the position of the virtual object can be adjusted.
  • user 1b may perform a gesture such as clenching both hands and then spreading or narrowing both arms while touching the target virtual object. In this case, the length of the virtual object can be lengthened or shortened.
  • the information processing system 10 recognizes an operation such as deselecting the virtual object to be adjusted by user 1b, it recognizes the completion of the adjustment operation, generates adjustment data based on the adjustment operation by user 1b, and adjusts the virtual object. Furthermore, after performing the adjustment operation, user 1b preferably moves away from the target virtual object and checks the adjustment results. If the adjustment is not as desired, the information processing system 10 will repeat the adjustment operation.
  • the information processing system 10 confirms whether user 1b has any other items that they would like to adjust (step S108).
  • the information processing system 10 may, for example, display a menu and confirm user 1b's intention by having user 1b touch the displayed menu.
  • the information processing system 10 may, for example, confirm user 1b's intention by recognizing a predetermined gesture or spoken voice of user 1b. If the information processing system 10 can confirm that there are no other items that user 1b would like to adjust (step S108: Yes), it proceeds to the processing of step S109. On the other hand, if the information processing system 10 cannot confirm that there are no other items that user 1b would like to adjust (step S108: No), it returns to the above-mentioned step S106.
  • the information processing system 10 confirms whether user 1b has any other virtual objects that he or she wishes to adjust (step S109).
  • the information processing system 10 may, for example, display a menu and confirm user 1b's intention by having user 1b touch the displayed menu.
  • the information processing system 10 may, for example, confirm user 1b's intention by recognizing a predetermined gesture or spoken voice of user 1b. If the information processing system 10 can confirm that there are no other virtual objects that user 1b wishes to adjust (step S109: Yes), it proceeds to the processing of step S110. On the other hand, if the information processing system 10 cannot confirm that there are no other virtual objects that user 1b wishes to adjust (step S109: No), it returns to the above-mentioned step S104.
  • the information processing system 10 begins recognizing user 1b's operation (gesture, menu operation, controller operation) to end the virtual object adjustment mode (step S110).
  • the information processing system 10 may, for example, display a menu and confirm user 1b's intention by having user 1b touch the displayed menu display with his/her hand.
  • the information processing system 10 may, for example, confirm user 1b's intention by recognizing a predetermined gesture or spoken voice from user 1b.
  • the information processing system 10 determines whether or not it has recognized user 1b's operation to end the virtual object adjustment mode (step S111). If it determines that it has recognized user 1b's operation to end the virtual object adjustment mode (step S111: Yes), it terminates the processing. On the other hand, if it determines that it has not recognized user 1b's operation to end the virtual object adjustment mode (step S111: No), it returns to step S101 described above.
  • steps S104 and S105 and step S106 may be executed in reverse order.
  • Fig. 4 to Fig. 7 are diagrams for explaining examples of adjustable items in this embodiment.
  • FIG. 4 shows an example of remote communication via holoportation between point A where user 1a is located and point B where user 1b is located.
  • two users 1a and 1b are standing facing each other and communicating via holoportation.
  • a 3D image 2b reproducing user 1b who is located at a distance, appears smaller than the actual user 1b.
  • user 1a can adjust the size of the 3D image 2b reproducing user 1b by, for example, grasping the 3D image 2b (or, for example, touching the 3D image 2b) and performing a spreading motion while viewing the 3D image 2b.
  • 3D image 2b reproducing user 1b as if they were sitting in the passenger seat.
  • adjustment is not limited to adjusting the position of the 3D image 2b that reproduces user 1b; for example, the position of a 3D image of an object located at point B or an avatar that resembles user 1b may also be adjusted.
  • FIG. 6 shows an example of remote communication between user 1a and user 1b, seated side by side on two seats arranged by holoportation between point A where user 1a is located and point B where user 1b is located.
  • two users 1a and 1b are seated side by side on two seats and communicating.
  • 3D image 2b which reproduces user 1b located at a distance, appears to be sitting unnaturally in relation to the seats.
  • user 1a can adjust the 3D image 2b, which reproduces user 1b, to an appropriate shape by grasping and deforming the 3D image 2b while viewing the 3D image 2b.
  • user 1a can adjust the 3D image 2b, which reproduces user 1b, to appear as if seated in a natural position.
  • This embodiment is not limited to adjusting the shape of the 3D image 2b, which reproduces user 1b.
  • the shape of a 3D image of an object located at point B or an avatar resembling user 1b may also be adjusted.
  • FIG. 7 shows an example of remote communication via holoportation between point A where user 1a is located and point B where user 1b is located.
  • two users 1a and 1b are standing facing each other and communicating via holoportation.
  • 3D image 2b which reproduces user 1b located at a distance
  • user 1a can adjust the luminance (brightness) of 3D image 2b, which reproduces user 1b, by selecting 3D image 2b and operating brightness adjustment UI 510 while viewing the 3D image 2b.
  • This embodiment is not limited to adjusting the luminance of 3D image 2b, which reproduces user 1b; for example, the luminance of a 3D image of an object located at point B or an avatar resembling user 1b may also be adjusted.
  • adjustment of the position, size, shape, and brightness of the 3D image 2b reproducing user 1b is not limited.
  • the resolution of the 3D image 2b reproducing user 1b, the 3D image of the object located at point B, the avatar resembling user 1b, the icon resembling the object located at point B, etc. may be adjusted.
  • various items related to the display mode can be adjusted, which reduces the sense of discomfort felt by the 3D image 2b and allows user 1a to experience realistic communication with user 1b.
  • Fig. 8 to Fig. 11 are diagrams for explaining an example of an operation in this embodiment.
  • Figure 8 shows an example of the operation performed when changing the size of a 3D image 2 that reproduces user 1 located at a distance.
  • the information processing system 10 displays an icon (third virtual object) 512 having a scale shape for comparing the size with the virtual object in the 3D image 2.
  • User 1 then performs an operation to adjust the size of the 3D image 2 while looking at the 3D image 2 and the icon 512.
  • user 1 touches the target 3D image 2 with both hands 3, clenches it as if grabbing it, and then spreads both arms, which causes the 3D image 2 to become longer horizontally.
  • FIG. 9 shows an example of an operation performed when changing the size of 3D image 2 that reproduces user 1 located at a distance.
  • information processing system 10 displays icon 512 with a scale shape for comparing the size with the virtual object in 3D image 2, and dial-shaped icon 514 (second virtual object) for accepting a request to change the display mode of 3D image 2.
  • User 1 then performs an operation to adjust the size of 3D image 2 while looking at 3D image 2 and icon 512.
  • user 1 grasps icon 514 with hand 3 and rotates it, thereby lengthening 3D image 2 horizontally.
  • the length that changes with one scale mark on dial-shaped icon 514 may also be adjusted by user 1's speech, for example.
  • FIG. 10 shows an example of an operation performed when changing the size of a 3D image 2 that depicts user 1 located at a distance.
  • the information processing system 10 displays an icon 512 having a scale (measurement mark) shape for comparing the size with the virtual object in the 3D image 2.
  • User 1 then performs an operation to adjust the size of the 3D image 2 while looking at the 3D image 2 and the icon 512.
  • user 1 can increase the vertical length of the 3D image 2 by speaking.
  • FIG. 11 shows an example of an operation performed when changing the size of a 3D image 4 (virtual object) that reproduces an object located at a distance.
  • user 1 performs an operation to adjust the size of the 3D image 4 while viewing the 3D image 4.
  • user 1 touches the target 3D image 4 between two fingers 7, and then adjusts the vertical length of the 3D image 4 by changing the distance between the fingers 7.
  • the adjustment operations in this embodiment can be performed with a feeling similar to that of operating on an actual object, and can be easily performed by anyone.
  • the information processing system 10 may use a model obtained by such analysis to make adjustments the next time the virtual object is displayed. By using such a model, the information processing system 10 can more appropriately adjust the virtual object.
  • the user 1 while viewing a virtual object such as a 3D image 2 that reproduces the user 1 at a remote location, the user 1 can freely, easily, and in real time adjust the display mode of the virtual object, thereby reducing the sense of discomfort felt by the user 1 with respect to the virtual object. Therefore, according to the present embodiment, the experiential value of the real-life communication experience can be improved without interrupting the experience. Furthermore, according to the present embodiment, the adjustment operation is easy, so the convenience of the system that provides the communication experience is not impaired. Furthermore, according to the embodiment of the present disclosure, data related to the adjustment of the display mode of the virtual object can be shared and used with other locations, or can be used the next time the virtual object is displayed. This allows the display mode of the virtual object to be automatically and appropriately adjusted. As a result, according to the present embodiment, the value of the real-life communication experience can be improved while further enhancing convenience.
  • FIG. 19 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100 and the like according to the embodiments of the present disclosure.
  • the computer 1000 includes a CPU (Central Processing Unit) 1100, a RAM (Random Access Memory) 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input/output interface 1600.
  • the components of the computer 1000 are connected by a bus 1050.
  • the CPU 1100 operates based on programs stored in the ROM 1300 or the HDD 1400 and controls each unit. For example, the CPU 1100 deploys the programs stored in the ROM 1300 or the HDD 1400 to the RAM 1200 and executes processing corresponding to the various programs.
  • the CPU 1100 can operate, for example, as the above-mentioned capture unit (acquisition unit) 102, encoding unit 104, position and orientation detection unit 106, decoding unit 112, image generation unit 114, rendering unit (display control unit) 116, object data management unit 120, object data sharing unit 122, object adjustment operation recognition unit (reception unit) 124, etc.
  • an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array) may be used.
  • BIOS Basic Input Output System
  • HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs.
  • HDD 1400 is a recording medium that records the information processing program related to the present disclosure, which is an example of program data 1450.
  • HDD 1400 can operate as the adjustment data storage unit 126 described above, etc.
  • Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, and semiconductor memories.
  • optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs)
  • magneto-optical recording media such as MOs (Magneto-Optical Disks)
  • tape media magnetic recording media
  • magnetic recording media and semiconductor memories.
  • CPU 1100 of computer 1000 executes programs stored in RAM 1200 to realize each function.
  • processing programs according to the present disclosure are stored in HDD 1400.
  • CPU 1100 reads and executes program data 1450 from HDD 1400, but as another example, these programs may be obtained from another device via external network 1550.
  • the information processing device 100 according to this embodiment may be applied to a system consisting of multiple devices that assumes connection to a network (or communication between devices), such as cloud computing.
  • the information processing device 100 according to this embodiment described above can also be realized as the information processing system 10 according to this embodiment using multiple devices, for example.
  • the above shows an example of the hardware configuration of the information processing device 100, etc.
  • Each of the above components may be configured using general-purpose parts, or may be configured using hardware specialized for the function of each component. This configuration may be modified as appropriate depending on the technical level at the time of implementation.
  • the embodiment of the present disclosure may include, for example, a program that causes the information processing system 10 described above to execute the information processing method according to the embodiment of the present disclosure, and a non-transitory tangible medium on which the program is recorded.
  • the program may also be distributed via a communication line (including wireless communication) such as the Internet.
  • each device shown in the figure are functional concepts and are not limited to being physically configured as shown.
  • the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them may be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.
  • the steps in the information processing method of the above-described embodiment of the present disclosure do not necessarily have to be processed in the order described.
  • the steps may be processed in a different order as appropriate.
  • the steps may be partially processed in parallel or individually.
  • the processing of each step does not necessarily have to be performed in the manner described, and may be performed by other functional units using other methods.
  • the present technology can also be configured as follows. (1) an acquisition unit that acquires object data regarding a display target; a display control unit that displays a first virtual object based on the object data; a receiving unit that receives a request to change a display mode of the displayed first virtual object; Equipped with the display control unit changes the display mode of the first virtual object based on the change request.
  • Information processing system (2) The information processing system according to (1) above, wherein the change request includes a request to change at least one of the size, brightness, position, shape, and resolution of the first virtual object.
  • the information processing system according to any one of (1) to (3), wherein the reception unit recognizes a user's speech and receives a request to change the display mode of the first virtual object.
  • the information processing system according to any one of (1) to (3), wherein the reception unit recognizes an operation performed by a user on an input device and receives a request to change the display mode of the first virtual object.
  • the information processing system according to any one of (1) to (9), wherein the display control unit displays the first virtual object so as to be superimposed on real space.
  • the display control unit displays the first virtual object so as to be superimposed on a virtual space.
  • the display control unit displays the first virtual object according to a viewpoint of the user at the second location.
  • the adjustment data is data based on a change request from the user located at the second location, the display control unit displays the first virtual object, the display mode of which has been changed using the adjustment data, toward another user who is located at a third location different from the first and second locations;
  • the display control unit adjusts the other object data by using the adjustment data stored in the storage unit.
  • the information processing device Obtain object data for the display target; displaying a first virtual object based on the object data; receiving a request to change a display mode of the displayed first virtual object; changing a display mode of the first virtual object based on the change request;
  • An information processing method including: (20) Computer, an acquisition unit that acquires object data regarding a display target; a display control unit that displays a first virtual object based on the object data; a receiving unit that receives a request to change a display mode of the displayed first virtual object; A program that functions as the display control unit changes the display mode of the first virtual object based on the change request. program.
  • Information processing system 100, 100a, 100b, 100c Information processing device 102 Capture unit 104 Encode unit 106 Position and orientation detection unit 108 Transmission unit 110 Reception unit 112 Decode unit 114 Image generation unit 116 Rendering unit 120 Object data management unit 122 Object data sharing unit 124 Object adjustment operation recognition unit 126 Adjustment data storage unit 200, 200b Input device 300, 300a, 300b Sensor 400, 400a, 400b Display 500 Communication network 510 Brightness adjustment UI 512, 514, 516 Icons

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Multimedia (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

L'invention concerne un système de traitement d'informations comprenant une unité d'acquisition qui acquiert des données d'objet relatives à un objet à afficher, une unité de commande d'affichage qui provoque l'affichage d'un premier objet virtuel sur la base des données d'objet à afficher et une unité d'acceptation qui accepte une demande de modification de mode d'affichage pour le premier objet virtuel affiché, l'unité de commande d'affichage modifiant le mode d'affichage du premier objet virtuel sur la base de la demande de modification.
PCT/JP2025/004652 2024-03-11 2025-02-13 Système de traitement d'informations, procédé de traitement d'informations et programme Pending WO2025192145A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2024-036597 2024-03-11
JP2024036597 2024-03-11

Publications (1)

Publication Number Publication Date
WO2025192145A1 true WO2025192145A1 (fr) 2025-09-18

Family

ID=97063379

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2025/004652 Pending WO2025192145A1 (fr) 2024-03-11 2025-02-13 Système de traitement d'informations, procédé de traitement d'informations et programme

Country Status (1)

Country Link
WO (1) WO2025192145A1 (fr)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5821925A (en) * 1996-01-26 1998-10-13 Silicon Graphics, Inc. Collaborative work environment supporting three-dimensional objects and multiple remote participants
JP2018074294A (ja) * 2016-10-26 2018-05-10 学校法人幾徳学園 情報処理システムおよび情報処理方法
JP2018084886A (ja) * 2016-11-22 2018-05-31 セイコーエプソン株式会社 頭部装着型表示装置、頭部装着型表示装置の制御方法、コンピュータープログラム
JP2019220932A (ja) * 2018-06-14 2019-12-26 株式会社バーチャルキャスト コンテンツ配信システム、コンテンツ配信方法、コンピュータプログラム、コンテンツ配信用サーバ、および伝送路
JP2021507355A (ja) * 2017-12-14 2021-02-22 マジック リープ， インコーポレイテッドＭａｇｉｃ Ｌｅａｐ，Ｉｎｃ． 仮想アバタのコンテキストベースのレンダリング
CN113674430A (zh) * 2021-08-24 2021-11-19 上海电气集团股份有限公司 虚拟模型定位配准方法、装置、增强现实设备和存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5821925A (en) * 1996-01-26 1998-10-13 Silicon Graphics, Inc. Collaborative work environment supporting three-dimensional objects and multiple remote participants
JP2018074294A (ja) * 2016-10-26 2018-05-10 学校法人幾徳学園 情報処理システムおよび情報処理方法
JP2018084886A (ja) * 2016-11-22 2018-05-31 セイコーエプソン株式会社 頭部装着型表示装置、頭部装着型表示装置の制御方法、コンピュータープログラム
JP2021507355A (ja) * 2017-12-14 2021-02-22 マジック リープ， インコーポレイテッドＭａｇｉｃ Ｌｅａｐ，Ｉｎｃ． 仮想アバタのコンテキストベースのレンダリング
JP2019220932A (ja) * 2018-06-14 2019-12-26 株式会社バーチャルキャスト コンテンツ配信システム、コンテンツ配信方法、コンピュータプログラム、コンテンツ配信用サーバ、および伝送路
CN113674430A (zh) * 2021-08-24 2021-11-19 上海电气集团股份有限公司 虚拟模型定位配准方法、装置、增强现实设备和存储介质

Similar Documents

Publication Publication Date Title
US11055920B1 (en) Performing operations using a mirror in an artificial reality environment
JP5784818B2 (ja) 拡張現実システムにおける実世界表面への仮想画像のアンカリング
CN110892364A (zh) 扩展现实虚拟助手
JP2009087026A (ja) 映像表示装置
JP2012128779A (ja) 仮想物体表示装置
US11907434B2 (en) Information processing apparatus, information processing system, and information processing method
WO2024253976A1 (fr) Dispositifs, procédés et interfaces utilisateur graphiques pour afficher des vues d'emplacements physiques
WO2025192145A1 (fr) Système de traitement d'informations, procédé de traitement d'informations et programme
US20200410734A1 (en) Spatial reproduction method and spatial reproduction system
JP2024047587A (ja) 情報処理システムおよびプログラム
JP2023134045A (ja) 通信装置、通信システム、表示方法、及びプログラム
JP2024047008A (ja) 情報処理システム
WO2022040729A1 (fr) Système et procédé d'immersion volumétrique

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25770183

Country of ref document: EP

Kind code of ref document: A1